hg38_celltype: violation framework, 33 classes; .ubjx -> .clfx

hg38_celltype is now a violation-framework model with no fitted parameters: it reads each class's expected methylation state off the MRMP binstrings and calls the class the query contradicts least, weighting patterns by sqrt(n_cpg). Confidence is the margin to the runner-up, which collapses when no class fits.

External test 0.842 / macro-F1 0.853 at >=100k covered CpGs. Colorectal cancer single cells call 100% gastrointestinal from liver and lymph metastases alike, and the sink that sent globally demethylated tumours to Acinar.(Pancreas) drops from 30.0% of TCGA calls to 2.3%.

Classifier bundles are renamed .ubjx -> .clfx: .ubjx named a payload format rather than a role and had already stopped being accurate (hg38_sex carried a plain-text linear spec, no UBJSON). Bundles are detected by MSBNDL1 magic, never by extension, so .ubjx stays accepted. The previous filenames remain available at --revision v5.

mm10_celltype.ubjx is unchanged, pending a Liu 2021 rebuild.

.gitattributes

@@ -36,6 +36,6 @@ saved_model/**/* filter=lfs diff=lfs merge=lfs -text
 hg38_wg.updecx filter=lfs diff=lfs merge=lfs -text
 hg38_10k1.updecx filter=lfs diff=lfs merge=lfs -text
 hg38_65celltypes.refx filter=lfs diff=lfs merge=lfs -text
-hg38_celltype.ubjx filter=lfs diff=lfs merge=lfs -text
+hg38_celltype.clfx filter=lfs diff=lfs merge=lfs -text
 mm10_celltype.ubjx filter=lfs diff=lfs merge=lfs -text
 mm10_wg.updecx filter=lfs diff=lfs merge=lfs -text

README.md

@@ -23,14 +23,21 @@ runs directly — no separate annotation files.
 
 ```sh
 # fetch one model (or --local-dir models to grab all)
-hf download zhou-lab/methscope hg38_celltype.ubjx --local-dir .
+hf download zhou-lab/methscope hg38_celltype.clfx --local-dir .
 
-methscope predict query.cg    hg38_celltype.ubjx      # cell type / label
+methscope classify query.cg   hg38_celltype.clfx       # cell type / label
 methscope deconv  mixture.cg  hg38_65celltypes.refx    # cell-type proportions (NNLS)
 methscope upscale -o out.cg   hg38_wg.updecx  query.cg # genome-wide CpG upscaling
-methscope inspect             hg38_sex.ubjx            # framework, labels, features
+methscope inspect             hg38_sex.clfx            # framework, labels, features
 ```
 
+The file extension names the model's **role** — `.clfx` classifier, `.refx`
+deconvolution reference, `.updecx` upscale decoder. All three are the same
+`MSBNDL1` container and are detected by magic, never by name. Classifiers used
+to be `.ubjx`, which described a payload format (a UBJSON booster) rather than a
+role and had already stopped being accurate for the non-booster frameworks;
+`.ubjx` is still accepted everywhere and nothing needs renaming locally.
+
 Small query `.cg` test fixtures (typed cells, a deconvolution mixture, an
 upscale input + truth) live in the GitHub repo
 [`zhou-lab/methscope_data`](https://github.com/zhou-lab/methscope_data) under
@@ -42,16 +49,85 @@ upscale input + truth) live in the GitHub repo
 |------|------|-----------|----------------|
 | `hg38_wg.updecx` | whole-genome CpG upscaling | `UPDEC2` | all 29,401,795 hg38 CpGs; the primary upscaler (2.8 GB) |
 | `hg38_10k1.updecx` | CpG upscaling (single block) | MLP decoder | block 10k1 (10,000 CpGs); small demo model |
-| `hg38_celltype.ubjx` | cell-type annotation | xgboost | 62 human cell types |
-| `mm10_celltype.ubjx` | cell-type annotation | xgboost | 41 mouse-brain cell types |
-| `hg38_sex.ubjx` | sex prediction | logistic | Female, Male (XCI `Xa_hi`/`Xa_lo` markers) |
+| `hg38_celltype.clfx` | cell-type annotation | `violation` | 33 human cell types; **no fitted parameter** — see below |
+| `mm10_celltype.ubjx` | cell-type annotation (brain) | xgboost | 41 mouse-brain cell types; rename to `mm10_celltype_brain.clfx` pending a Liu 2021 rebuild |
+| `hg38_sex.clfx` | sex prediction | logistic | Female, Male (XCI `Xa_hi`/`Xa_lo` markers) |
 | `hg38_65celltypes.refx` | cell-type deconvolution (NNLS) | refx | 65 types = 58 Zhou + 7 Loyfer organ/blood; split MRMP, 15,300 patterns |
 
 `methscope inspect <model>` prints the exact framework, full label list, and (for
 linear models) per-feature weights. Classifiers carry a required framework `kind`
-mark (`xgboost` / `threshold` / `logistic`); `predict` rejects an unmarked
-bundle. Upscale decoders (`.updecx`) run via `upscale`, deconvolution references
-(`.refx`, `kind=refx`) via `deconv`.
+mark (`xgboost` / `violation` / `threshold` / `logistic`); `classify` rejects an
+unmarked bundle. Upscale decoders (`.updecx`) run via `upscale`, deconvolution
+references (`.refx`, `kind=refx`) via `deconv`.
+
+## The `violation` framework
+
+`hg38_celltype.clfx` has **no trained parameters at all**. Every MRMP
+already carries a binstring — the methylation state each reference class takes
+at that pattern — so the reference *is* a per-class prediction. The model just
+asks which class the query contradicts least:
+
+```
+E1 = patterns the class expects methylated;  E0 = expects unmethylated
+w_j = sqrt(n_cpg_j)
+
+v1 = Σ_{j∈E1, observed} w_j·[β_j ≤ 0.5] / Σ_{j∈E1, observed} w_j
+v0 = Σ_{j∈E0, observed} w_j·[β_j >  0.5] / Σ_{j∈E0, observed} w_j
+
+predict argmin_class (v1 + v0), needing ≥20 observed patterns per side
+```
+
+`confidence` is the **margin** to the runner-up, not a probability. It is
+calibrated (accuracy 0.757 / 0.820 / 0.899 at margin ≥ 0.02 / 0.05 / 0.10) and,
+unlike a softmax, it collapses toward zero when *no* class fits — which is how
+the model signals that the query's true type is missing from the reference.
+Records too sparse to clear the minimum are reported `NA` rather than guessed.
+
+Held out **across cohorts** (each class's reference and its test samples come
+from different studies), 113 donor-resolved pseudobulks at ≥100k covered CpGs:
+**accuracy 0.842, macro-F1 0.853** — against 0.873 / 0.868 for an xgboost model
+on the identical MRMP. It trades ~3 points of normal-tissue accuracy for
+behaviour that holds up on tumours.
+
+**Single-cell tumours**, where it earns that trade. Bian is split by anatomical
+site because that is the whole point — a colorectal cancer cell sitting in liver
+must still read as colon:
+
+| cohort | GI fraction | top call |
+|---|---|---|
+| Bian 2018 CRC, **liver metastases** | **100.0%** | `Enterocyte.(Colon)` |
+| Bian 2018 CRC, **lymph metastases** | **100.0%** | `Enterocyte.(Colon)` |
+| Bian 2018 CRC, primaries (L/R colon, rectum) | 97.1–100.0% | `Enterocyte.(Colon)` |
+| Bian 2018 adjacent normal (within-patient control) | 100.0% | `Enterocyte.(Colon)` |
+| Gaiti 2019 CLL | — | `B.Cell` |
+| Chaligne 2021 glioma | — | `Oligodendrocyte` |
+
+Nothing in the reference anticipates the metastatic case, and no bulk cohort can
+separate origin from site the way these cells do. It also resists the
+mitotic-age confound that dominates trained models on this feature space: the
+"most-hypomethylated-class sink" sending globally demethylated tumours to
+`Acinar.(Pancreas)` drops from 30.0% of TCGA calls to 2.3%, with no
+residualisation and no tumour data anywhere in its construction.
+
+Accuracy is coverage-dependent: 0.163 below 8k covered CpGs, 0.719 at 8–32k,
+plateauing near 0.85 above ~130k. **Do not use below ~30k covered CpGs.**
+
+Limitations: `Macrophage` (0.27) and `Monocyte` (0.74) bleed into
+`Dendritic.Cell` — the macrophage reference here is BLUEPRINT's, which is
+monocyte-derived and differentiated *in vitro*, a poor match for tissue
+macrophages. `T.Cell.CD8` leaks to `CD4` at 27%. Every remaining error is a
+within-lineage confusion; none cross lineages. The atlas is also incomplete, and
+a missing origin surfaces as a **low margin** rather than an error — glioma has
+the lowest median margin of any cohort tested, because neither OPC nor astrocyte
+is a class. Margins are not comparable between models with different class
+counts, so recalibrate any threshold per model. For a bulk tumour, deconvolve
+rather than classify: a classifier assigns one label to a mixture.
+
+Rebuild from any MRMP artifact, with no training data, in about 7 seconds:
+
+```sh
+methscope classify-train --framework violation -p 1000 -o out.clfx ref.mrmp
+```
 
 See the [methscope-cli documentation](https://zhou-lab.github.io/methscope-cli/)
 for the full command reference and runnable examples.

SHA256SUMS

@@ -1,7 +1,7 @@
 8f7f2d6d42f64daaba5bf9cede5146fa52c0f62b5c76b3d5c6680c5c7554b1b7  hg38_10k1.updecx
 ce16678dd410c83e3df1c6f1c9b912a796f69c77722b28fc83c5690df69a6c03  hg38_65celltypes.refx
-e1f738290b05f824873b599894228a33b189ffbda0644ed2aad0e82e7768bc7c  hg38_celltype.ubjx
-3d1618f6ca24d9a9d5b0fe1806673540515e5f6e7d7950e7c8873a85bb5c742b  hg38_sex.ubjx
+00216ef333e3fede15b1bce1c820af23f2c0436b1c9d13b97132add311f17e05  hg38_celltype.clfx
+3d1618f6ca24d9a9d5b0fe1806673540515e5f6e7d7950e7c8873a85bb5c742b  hg38_sex.clfx
 60c786aa383c27461e7a5de37d6b17d2e01b33c5f8a6553f3df7083035fac65a  hg38_wg.updecx
 3bbdfeabbc59c2e5810eee06c07d24c8fc9826926dd59ddd167ff46b4cb028de  mm10_celltype.ubjx
 f25cc5317a892105a121300fef009e29e96677f9a43fd02c3b47a59bf64cd482  mm10_wg.updecx

hg38_celltype.ubjx → hg38_celltype.clfx

@@ -1,3 +1,3 @@
 version https://git-lfs.github.com/spec/v1
-oid sha256:e1f738290b05f824873b599894228a33b189ffbda0644ed2aad0e82e7768bc7c
-size 25661483
+oid sha256:00216ef333e3fede15b1bce1c820af23f2c0436b1c9d13b97132add311f17e05
+size 5738519