SeaWolf-AI commited on
Commit
2da1014
·
verified ·
1 Parent(s): 38ce261

card: JEV-compatible API, calibration, and what to do on a low score

Browse files
Files changed (1) hide show
  1. README.md +90 -0
README.md CHANGED
@@ -114,6 +114,96 @@ not a calibrated probability — choose a threshold from your own review budget.
114
 
115
  Both read the same input. The curved probe is the one to use when the number matters.
116
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
117
  ## Protocol
118
 
119
  | | |
 
114
 
115
  Both read the same input. The curved probe is the one to use when the number matters.
116
 
117
+ ## API — drop-in for an existing JEV integration
118
+
119
+ The endpoint takes the same request shape and returns the same response shape, so switching an
120
+ existing integration is a URL change.
121
+
122
+ ```bash
123
+ POST /v1/evaluate
124
+ Authorization: Bearer <token>
125
+
126
+ {"model": "vidraft/ztc",
127
+ "state": {"question": "...", "answer": "..."},
128
+ "questions": {"correct": {"type": "boolean",
129
+ "instructions": "Is the ANSWER factually correct?"}}}
130
+ ```
131
+
132
+ ```json
133
+ {"model": "vidraft/ztc-judge-9b",
134
+ "answers": {"correct": {
135
+ "probability": 0.1043,
136
+ "verdict": "review",
137
+ "score": -0.72,
138
+ "position": 0.268,
139
+
140
+ "band": "low",
141
+ "action": "hold_or_escalate",
142
+ "measured": {
143
+ "band_accuracy": 0.485,
144
+ "base_accuracy": 0.748,
145
+ "if_lowest_20pct_dropped": 0.814,
146
+ "escalate_gain_at_20pct_budget": 0.0134,
147
+ "do_not": "resample_same_model",
148
+ "why_not": "measured: fixes 6.7% of wrong answers, breaks 13.1% of right ones"}}},
149
+ "usage": {"generated_tokens": 0}}
150
+ ```
151
+
152
+ `type` accepts `boolean` and `noul`. Existing clients read `answers.<key>.probability` and ignore
153
+ the rest; the additional fields are there for clients that want to act on the score rather than
154
+ merely record it. **0.19 s per call, zero generated tokens.**
155
+
156
+ ### What `probability` means
157
+
158
+ The raw score is unbounded. The shipped calibration maps it to P(answer is correct), fitted
159
+ **leave-one-domain-out** — the mapping never sees the domain it is applied to.
160
+
161
+ | | Expected calibration error |
162
+ |---|---|
163
+ | **ZTC-Judge-27B (after calibration)** | **0.0245** |
164
+ | JEV, as shipped | 0.0381 |
165
+ | JEV, after the same calibration | 0.0261 |
166
+ | Laya-Multilingual, as shipped | 0.4985 |
167
+ | Laya-Typed-Decisions, as shipped | 0.2641 |
168
+
169
+ Measured on the same 2,018 items. **ZTC and JEV are effectively tied on calibration; the difference
170
+ of 0.0016 is not meaningful.** Figures published elsewhere for these systems were measured on other
171
+ test sets and do not reproduce here.
172
+
173
+ 🔴 Calibration is uneven across domains: 0.0225 on biology & medicine, but 0.2941 on scientific
174
+ reasoning and 0.2381 on general reasoning. Treat the probability as reliable in the first case and
175
+ as a ranking signal only in the other two.
176
+
177
+ ## What to do when the score is low
178
+
179
+ The score alone is not actionable, so the response carries the measured consequence of each choice.
180
+
181
+ | Band | Share | **Actual accuracy of answers in this band** | Recommended |
182
+ |---|---|---|---|
183
+ | `low` | 20% | **48.5%** | hold, or escalate to a stronger model |
184
+ | `mid` | 40% | 73.2% | escalate if budget allows |
185
+ | `high` | 40% | **89.6%** | accept |
186
+
187
+ **Three things that work, measured on 2,018 items:**
188
+
189
+ | Action | Effect |
190
+ |---|---|
191
+ | **Drop the lowest-scoring 20%** | accuracy of what remains: **74.8% → 81.4%** |
192
+ | **Escalate the lowest 20% to a stronger model** | **+1.34 pp** end-to-end |
193
+ | **Send the lowest 20% to human review** | catches **47.2%** of all errors — 2.4× random |
194
+
195
+ **One thing that does not work:**
196
+
197
+ 🔴 **Do not resample the same model.** Measured: re-drawing and taking a 3-way majority fixes
198
+ **6.7%** of wrong answers while breaking **13.1%** of right ones. It is a net loss, and it is the
199
+ first thing most pipelines try.
200
+
201
+ *Escalation pays for itself through precision, not recall. Re-answering repairs about 38% of wrong
202
+ answers and damages about 30% of right ones, so a gate is only worth its budget if it mostly calls
203
+ answers that are actually wrong.*
204
+
205
+ ---
206
+
207
  ## Protocol
208
 
209
  | | |