add README.md

README.md

@@ -0,0 +1,69 @@
+---
+license: other
+library_name: pytorch
+tags:
+  - interpretability
+  - jacobian-lens
+  - j-space
+  - mixture-of-experts
+---
+
+# Inkling J-space lens (n=250, all-layer)
+
+A **Jacobian lens (J-space) fit of `thinkingmachines/Inkling`** (~950B-param MoE: 66 layers,
+d_model 6144, 256 routed experts/layer + 2 shared, hybrid SWA-512/global attention, sconv),
+fitted from the official NVFP4 checkpoint (`thinkingmachines/Inkling-NVFP4`).
+
+**J_ℓ = E[∂h_target,t′ / ∂h_ℓ,t]** — the expected Jacobian of the final residual stream with
+respect to layer ℓ's residual stream, averaged over token positions and prompts.
+
+## Fit configuration
+
+| | |
+|---|---|
+| source layers | **all 65** (0–64), target = layer 65 |
+| prompts | **250** wikitext-103 (train, max_chars 2000), seq_len 128, 2-way prompt-sharded |
+| weights | NVFP4 (block-16 e4m3 scales) resident, bit-exact fused dequant-autograd (bf16 compute, fp32 accumulation) |
+| dim_batch | 64 (96 backward passes/prompt) |
+| rate | ~526 s/prompt/node on 8×H200, 18.6 h/shard, 2 nodes |
+| merge | exact n-weighted: J = (n₀·J₀ + n₁·J₁)/(n₀+n₁), verified ≤3.1e-5 |
+
+## Files
+
+- `inkling_jacobian_lens_n250.pt` — merged lens (`jlens` JacobianLens payload: per-layer fp32
+  `J[6144,6144]` means for layers 0–64, target 65, n=250)
+- `shards/shard0of2.pt`, `shards/shard1of2.pt` — the two disjoint-prompt half-lenses
+  (125 prompts each; used for the split-half robustness checks — same-layer CKA median 0.997)
+- `merge_weights.json`, `convergence_shard{0,1}.csv` — provenance
+- `cka/` — the recombined CKA space: `inkling_cka.npz` (65×65 CKA-vs-ρ matrix),
+  `inkling_cka_blocks.json` (KTS-DP segmentation), `inkling_cka_heatmap.png`, `summary.json`
+
+## Probe convention (frozen before fitting)
+
+`V_ℓ = P · J_ℓ` with `P = W_U[ids] · γ / 24.0` — plain-RMSNorm gain (raw γ, not gemma-style 1+γ),
+muP logit divisor 24 folded in, `ids = sample_probe_ids(vocab_rows=201024, k=4096, seed=0)`
+(22 ids fall in padding rows [200058, 201024); rows are all-zero and kept for convention
+compatibility with the rest of the J-space program).
+
+## Headline geometry (details in the program's E7 report)
+
+- **A third geometry pattern** among big models: weakest blockiness of all five fitted
+  large models (0.106 vs Kimi-K2.5 0.415, DeepSeek-V4 0.499, dense 0.24–0.31); dominant
+  structure is a single sharp cliff at layers 27→28 (adjacent CKA 0.538; ‖V‖ ×3.2 drop);
+  layers 3–27 form a huge-norm PR≈4–7 near-identical spike regime that ends at the cliff.
+- **Closest relative is dense**: matched cross-CKA 0.625 with gemma-3-27b-it (peak 0.836),
+  0.601 with OLMo-32B; **Kimi-K2.5 is the anti-pair at 0.362** — "1T MoE" is not a geometry class.
+- Band (mid-50%) peak PR/d = 0.011; global peak PR 412.6 at ρ 0.985.
+- Scaffold does not track the 6-layer global-attention period (n.s.).
+- Heavy-tail: bulk prompt norms ~853× dense, but tail dense-shaped (max/median 4.8 vs Kimi 29).
+
+## Caveats
+
+- Jacobians carry NVFP4 weight-quantization error (same accepted trade as the program's
+  int4 Kimi-K2.5 fit; forward validated: sane greedy decodes, bit-exact dequant vs reference).
+- Convergence Δmean end states: 4.17e-2 (shard 0) / 1.22e-1 (shard 1; final prompt was a
+  ~2×-median bulk event) — MoE fits do not reach the 5e-3 dense-model bar at n=125/shard.
+- seq_len 128 only; no Δ-resolved (horizon) information in this artifact.
+
+Fitting/analysis code: `open-jlens-data` repo (`code/moe/fit_moe.py`, `nvfp4_experts.py`,
+`analyze_inkling_geometry.py`). Fitted 2026-07-16/17.