Text Classification
Transformers
Safetensors
English
Korean
qwen3_5
image-text-to-text
ztc
answer-verification
hallucination-detection
zero-token
confidence-estimation
Instructions to use FINAL-Bench/ZTC-Judge-9B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Transformers
How to use FINAL-Bench/ZTC-Judge-9B with Transformers:
# Use a pipeline as a high-level helper from transformers import pipeline pipe = pipeline("text-classification", model="FINAL-Bench/ZTC-Judge-9B")# pip install -U transformers accelerate # Load model directly from transformers import AutoProcessor, AutoModelForMultimodalLM processor = AutoProcessor.from_pretrained("FINAL-Bench/ZTC-Judge-9B") model = AutoModelForMultimodalLM.from_pretrained("FINAL-Bench/ZTC-Judge-9B", device_map="auto") - Notebooks
- Google Colab
- Kaggle
card: JEV-compatible API, calibration, and what to do on a low score
Browse files
README.md
CHANGED
|
@@ -114,6 +114,96 @@ not a calibrated probability — choose a threshold from your own review budget.
|
|
| 114 |
|
| 115 |
Both read the same input. The curved probe is the one to use when the number matters.
|
| 116 |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| 117 |
## Protocol
|
| 118 |
|
| 119 |
| | |
|
|
|
|
| 114 |
|
| 115 |
Both read the same input. The curved probe is the one to use when the number matters.
|
| 116 |
|
| 117 |
+
## API — drop-in for an existing JEV integration
|
| 118 |
+
|
| 119 |
+
The endpoint takes the same request shape and returns the same response shape, so switching an
|
| 120 |
+
existing integration is a URL change.
|
| 121 |
+
|
| 122 |
+
```bash
|
| 123 |
+
POST /v1/evaluate
|
| 124 |
+
Authorization: Bearer <token>
|
| 125 |
+
|
| 126 |
+
{"model": "vidraft/ztc",
|
| 127 |
+
"state": {"question": "...", "answer": "..."},
|
| 128 |
+
"questions": {"correct": {"type": "boolean",
|
| 129 |
+
"instructions": "Is the ANSWER factually correct?"}}}
|
| 130 |
+
```
|
| 131 |
+
|
| 132 |
+
```json
|
| 133 |
+
{"model": "vidraft/ztc-judge-9b",
|
| 134 |
+
"answers": {"correct": {
|
| 135 |
+
"probability": 0.1043,
|
| 136 |
+
"verdict": "review",
|
| 137 |
+
"score": -0.72,
|
| 138 |
+
"position": 0.268,
|
| 139 |
+
|
| 140 |
+
"band": "low",
|
| 141 |
+
"action": "hold_or_escalate",
|
| 142 |
+
"measured": {
|
| 143 |
+
"band_accuracy": 0.485,
|
| 144 |
+
"base_accuracy": 0.748,
|
| 145 |
+
"if_lowest_20pct_dropped": 0.814,
|
| 146 |
+
"escalate_gain_at_20pct_budget": 0.0134,
|
| 147 |
+
"do_not": "resample_same_model",
|
| 148 |
+
"why_not": "measured: fixes 6.7% of wrong answers, breaks 13.1% of right ones"}}},
|
| 149 |
+
"usage": {"generated_tokens": 0}}
|
| 150 |
+
```
|
| 151 |
+
|
| 152 |
+
`type` accepts `boolean` and `noul`. Existing clients read `answers.<key>.probability` and ignore
|
| 153 |
+
the rest; the additional fields are there for clients that want to act on the score rather than
|
| 154 |
+
merely record it. **0.19 s per call, zero generated tokens.**
|
| 155 |
+
|
| 156 |
+
### What `probability` means
|
| 157 |
+
|
| 158 |
+
The raw score is unbounded. The shipped calibration maps it to P(answer is correct), fitted
|
| 159 |
+
**leave-one-domain-out** — the mapping never sees the domain it is applied to.
|
| 160 |
+
|
| 161 |
+
| | Expected calibration error |
|
| 162 |
+
|---|---|
|
| 163 |
+
| **ZTC-Judge-27B (after calibration)** | **0.0245** |
|
| 164 |
+
| JEV, as shipped | 0.0381 |
|
| 165 |
+
| JEV, after the same calibration | 0.0261 |
|
| 166 |
+
| Laya-Multilingual, as shipped | 0.4985 |
|
| 167 |
+
| Laya-Typed-Decisions, as shipped | 0.2641 |
|
| 168 |
+
|
| 169 |
+
Measured on the same 2,018 items. **ZTC and JEV are effectively tied on calibration; the difference
|
| 170 |
+
of 0.0016 is not meaningful.** Figures published elsewhere for these systems were measured on other
|
| 171 |
+
test sets and do not reproduce here.
|
| 172 |
+
|
| 173 |
+
🔴 Calibration is uneven across domains: 0.0225 on biology & medicine, but 0.2941 on scientific
|
| 174 |
+
reasoning and 0.2381 on general reasoning. Treat the probability as reliable in the first case and
|
| 175 |
+
as a ranking signal only in the other two.
|
| 176 |
+
|
| 177 |
+
## What to do when the score is low
|
| 178 |
+
|
| 179 |
+
The score alone is not actionable, so the response carries the measured consequence of each choice.
|
| 180 |
+
|
| 181 |
+
| Band | Share | **Actual accuracy of answers in this band** | Recommended |
|
| 182 |
+
|---|---|---|---|
|
| 183 |
+
| `low` | 20% | **48.5%** | hold, or escalate to a stronger model |
|
| 184 |
+
| `mid` | 40% | 73.2% | escalate if budget allows |
|
| 185 |
+
| `high` | 40% | **89.6%** | accept |
|
| 186 |
+
|
| 187 |
+
**Three things that work, measured on 2,018 items:**
|
| 188 |
+
|
| 189 |
+
| Action | Effect |
|
| 190 |
+
|---|---|
|
| 191 |
+
| **Drop the lowest-scoring 20%** | accuracy of what remains: **74.8% → 81.4%** |
|
| 192 |
+
| **Escalate the lowest 20% to a stronger model** | **+1.34 pp** end-to-end |
|
| 193 |
+
| **Send the lowest 20% to human review** | catches **47.2%** of all errors — 2.4× random |
|
| 194 |
+
|
| 195 |
+
**One thing that does not work:**
|
| 196 |
+
|
| 197 |
+
🔴 **Do not resample the same model.** Measured: re-drawing and taking a 3-way majority fixes
|
| 198 |
+
**6.7%** of wrong answers while breaking **13.1%** of right ones. It is a net loss, and it is the
|
| 199 |
+
first thing most pipelines try.
|
| 200 |
+
|
| 201 |
+
*Escalation pays for itself through precision, not recall. Re-answering repairs about 38% of wrong
|
| 202 |
+
answers and damages about 30% of right ones, so a gate is only worth its budget if it mostly calls
|
| 203 |
+
answers that are actually wrong.*
|
| 204 |
+
|
| 205 |
+
---
|
| 206 |
+
|
| 207 |
## Protocol
|
| 208 |
|
| 209 |
| | |
|