Spaces:
Running
Running
Commit History
13 days ago
-
Review model examples and guard dense configuration imports
56acff6 -
Connect TAF results and field guide with contextual reading routes
435410a -
Add protocol-aware measurement lab and reproducible uniform control
b048711 -
Add multilingual research explorer linked to the Transformers Guide
e2715cd
-
feat: Memory Reality + Prediction-vs-Reality + Confidence + UX fixes
New diagnostic modes (browser-only, EN/ES/FR/ZH): - Memory Reality Check: classifies architecture (full/SWA/SSM/RWKV/linear/ TTT/hybrid) from config.json; detection validated vs 13 live HF configs. - Prediction vs Reality + BYOD: compares TAF predictions to MEASURED values (shipped dataset or Diagnose-CLI JSON); server-less contribution to the public dataset. - Confidence score on X-2 / Memory Reality / PvR (checklist of evidence factors). Honesty fix: X-2 discloses the d_horizon == T_eval tautology (D-NEW-1) and routes to measured gamma_obs; documented in docs/LIMITATIONS.md. UX: HF autocomplete on every model-id input (incl. Compare, Diagnose) and fixed "dropdown only opens once" (focus+click reopen, cancellable blur, lastQuery reset). Manual reorganised into static collapsible cards (same as the inventory) with per-topic icons. Added serve.py (no-cache dev server). Tests: +Node (memory_reality 21, confidence 6, prediction_reality 11) and pytest (confidence + X-2 tautology/c…ffe48ab
-
v0.7.7: UX restructure — task tiles + 'Why static?' README FAQ
UX RESTRUCTURE — addresses the 14-tabs-overflow problem before forum-post traffic arrives. NEW <section id='task-tiles'>: 5 tiles grouping the 14 modes by user intent (not feature): - 🔬 Diagnose a model (Profile, Unmask, NIAH→Reason, Quant, Inspector) - ✓ Trust a benchmark score (Contamination, Drift, Arena CI) - ⚙️ Set up an eval correctly (Chat-template, Diagnose CLI) - 🆚 Compare models (Compare, Phase diagram) - 📋 Manual / free-form (Recipe, Ask) Tile buttons emit [data-mode-link='X'] clicks → delegated handler triggers the corresponding mode-btn (no duplicate state, fully reuses the existing mode switcher) and smooth-scrolls to the section so the user immediately sees the form they expected. Mode-tabs strip stays as a power-user backup below the tiles (still wrapped foldable by wrapMainSectionsAsFoldable). Subtitle on tiles points users there if they want full access. Cognitive load drops from 14 buttons → 5 categorical tiles. Adding more modes in v0.8 won't require new top-…b4eccb8 -
v0.7.6: NIAH→reasoning predictor (anti-bullshit #7) + Drift CSS fix
NEW MODE 🔍 NIAH→Reason — pass rate predictor for retrieval vs reasoning at any context - js/niah_reasoning.js: pure logic. Reuses gammaPade/d_horizon machinery. - Inputs: HF model id (auto-fetch config) + target T_eval. - Outputs: NIAH pass rate, multi-hop reasoning rate, gap, "safe context" where reasoning ≥ 65%, verdict (robust / marginal / degraded / retrieval_only / broken). - Logistic on log(T_eval / d_horizon) for NIAH; arch-pressure-modulated penalty for reasoning. Extrapolation penalty when T_eval > T_train (no positional embeddings learned past training). - sweepContextLengths(): scans 1k / 4k / 16k / 64k / T_train so user sees the curve. - Solves: RULER paper (NVIDIA 2024) finding that long-context models pass NIAH but fail reasoning. P6 in HF community pain analysis. VIRTUAL SIMULATION - Llama-3.1-8B → robust at 8k / 64k / 128k (matches RULER findings). - Mistral-7B (SWA 4096) → robust at 4k, broken at 16k+ (SWA limit kicks in). - Pythia-160m at 4× T_train → broken…ab15c91 -
v0.7.5: Cross-framework drift bound (anti-bullshit #6) + autocomplete UX
NEW MODE 🔀 Drift — Cross-framework drift bound (anti-bullshit pack #6) - js/cross_drift.js: pure logic. Given two scores with (framework, dtype, batch, chat_template), predicts the maximum admissible drift from numerical noise alone (additive bound: dtype-pair penalty + framework-pair kernel diff + batch-ratio + 0.3-pt non-determinism floor). - Verdict tiers: noise (≤1× band) / suspicious (1×–2.5× band) / bug (>2.5×) / bug_template (chat-template mismatch — dominates regardless of gap). - Dominant-cause attribution: tells the user WHICH knob explains the gap. - Sample data button loads the canonical lm-eval-harness #1841 chat-template bug (50.2 vs 74.8 — same model, missing --apply_chat_template). - Solves: arxiv 2506.09501 documents this as a major eval reproducibility problem. HF AUTOCOMPLETE UX IMPROVEMENTS (extends v0.7.4) - LRU cache (5 min TTL, 50 entries) so re-typing the same query hits cache, not the API. Halves traffic for indecisive users. - Privacy footer always visible: …edb4038 -
v0.7.4: HF Hub search-as-you-type autocomplete on all model id inputs
NEW js/hf_autocomplete.js — wraps any text input with a dropdown that hits the public huggingface.co/api/models endpoint as the user types. Debounced (300 ms), keyboard-navigable (↑/↓/Enter/Esc), positioned via fixed coords on body so it never gets clipped. Wired to all 5 HF id inputs at startup via attachAllHfAutocompletes(): - #hf-id (Recipe mode) - #profile-hf-id (Profile mode) - #unmask-id (🪟 Unmask) - #template-id (📜 Chat-template) - #quant-id (⚖️ Quant) Each search result row shows: model id (mono) + downloads + likes + library_name. Filtered to text-generation pipeline by default. Sorted by downloads descending so popular official releases surface first. LIVE API VERIFIED — top-5 results for common queries: llama-3 → meta-llama/Llama-3.1-8B-Instruct (9.7M ⬇) mistral → mistralai/Mistral-7B-Instruct-v0.2 (2.6M ⬇) qwen → Qwen/Qwen3-0.6B (19M ⬇) phi-3 → microsoft/Phi-3-mini-4k-instruct (761K ⬇) Network failures degrade gracefully (silent — user can still type full…778145c -
v0.7.2: Arena CI + Contamination Prior + v0.7 Help/Inventory documentation
Closes the v0.7 anti-bullshit pack with two more browser-only modes plus full Help and Inventory documentation across all 4 features in EN/ES/FR/ZH. NEW MODES (research-driven from HF community pain points) 🎯 Arena CI — Arena-Elo CI reconstructor (anti-bullshit #3) - js/arena_ci.js: parseVotesCSV + Bradley-Terry MM-MLE + 200-iteration bootstrap + statistical-tie detection (CI overlap) - Solves: Chatbot Arena strips confidence intervals from its public leaderboard. A 5-Elo gap can be statistically meaningless. Tool reconstructs CIs from raw vote data + flags pairs whose CIs overlap. - Embedded SAMPLE_VOTES_CSV (6 models × 96 votes). One click to demo. - Sim: 39/39 passed. Adversarial 3-model fixture confirms B↔C correctly identified as tied while A clearly distinguishable. 🧪 Contamination Prior — anti-bullshit #4 - js/contamination_prior.js: BENCHMARK_DB of 20 entries (MMLU/HellaSwag/ARC/TruthfulQA/GSM8K/HumanEval/MBPP/BBH/IFEval/MuSR/GPQA/MATH-500/AIME-24/Winogrande/BoolQ/DROP/Tri…d61ea0e -
v0.7.1: Chat-template Sniffer (anti-bullshit #2) + 9th mode
Ships anti-bullshit pack #2: detect when an evaluation framework will silently halve accuracy because the chat template wasn't applied. lm-eval-harness issue #1841: running via vLLM-served auto-applies chat_template, but local hf/vllm mode does not — multi-turn evals drop ~50% with no warning. NEW - 📜 Chat-template mode: paste an HF model id (or raw tokenizer_config.json) → 1-second classification into a known family (Llama-3 / ChatML / Mistral / Gemma / Phi-3 / Alpaca / DeepSeek / custom / none) + exact CLI flags for lm-eval / vLLM / transformers. - js/chat_template_sniffer.js: pure logic module (codes + params, no human strings). Fetches /raw/main/tokenizer_config.json from HF Hub, parses chat_template field, matches distinctive markers per family. - Verdicts: ok (known family) · custom (template present, unrecognized) · missing (no chat_template — base model) · base_model · unknown. - Per-framework command output: lm_eval --apply_chat_template, vllm serve --chat-template <name>, t…f09cd1d -
v0.7.0: SWA Unmasker (anti-bullshit #1) + foldable main panels + preset auto-fill
Ships the first feature of the v0.7 anti-bullshit pack inspired by HF community pain points: detect when a model card claims a max_position_embeddings far larger than its effective context (Mistral-7B-v0.1: declared 32k, attends ~8k via SWA). NEW - 🪟 Unmask mode: paste an HF model id (or raw config.json) → 1-second verdict (HONEST / INFLATED / SEVERELY INFLATED / YARN-EXTENDED). Pure browser arithmetic on config.json: SWA window + RoPE-scaling + GQA/d_head heuristics. No GPU, no inference. - js/swa_unmasker.js: pure logic module, no human strings. Returns warning codes + params; main.js renders via i18n with {placeholder} substitution so EN/ES/FR/ZH all work. - tFmt() i18n helper: t(key) with {placeholder} replacement. UX - All <main> sections now wrapped at runtime in <details open> with foldable header (idempotent wrapMainSectionsAsFoldable). Big ▼ arrow rotates to ▶ when collapsed. Triple-browser marker hide (list-style + ::-webkit-details-marker + ::marker). - Inventory modal: 4…fbec820 -
v0.6.2: landing rework — minimalist hero + 3 modals (manual / quick start / inventory)
- Hero compacted: 1-line value prop ("Diagnose any transformer LLM in 30 seconds. Free. No GPU. No signup.") + plain-language subtitle, no inline arch badges - Three buttons in hero: ⚡ Quick start · 🧰 What it gives you · 📘 Manual & examples — each opens a focused modal - Quick start modal: 3 numbered steps (HF model id → Profile → read TAF Card) - Inventory modal: 4 cards in plain language (8 recipes / diagnostics / formal verification / export) + collapsible arch-supported list with 6 new tooltips (MHA, GQA, ALiBi, AbsPE, SWA, SSM) - modes.tip fix: "Profile = 8 recipes" → "5-recipe TAF Card" (Profile chain runs 5; Help correctly lists 8 total recipes available) - JS generalized: wireModal() factory + Map-based close handlers; one keyboard listener handles Esc + Tab focus-trap on whichever modal is open - CSS: modal styling shared via [role="dialog"] / IDs; hero-buttons flex layout; qs-steps numbered counters; inventory-grid auto-fit - i18n EN/ES/FR/ZH: 369 keys × 4 langs, 0 missing …81fc8a0 -
v0.6.1: 5 audit-driven gap fixes (privacy, exports, a11y, tooling)
- meta tags: corrected to "8 recipes" (Help title already said 8; Profile-context retains "5" = profile chain count) - submit-to-registry: clipboard + title-only URL replaces full-JSON-in-GET (privacy: was leaking body via GH server logs / referer headers) - LaTeX/Markdown export: 6 new download buttons (3 share-bars × 2 formats); profileToLatex / compareToLatex / recipeToLatex serializers - accessibility: ARIA roles on modal/tabs/lang-switcher, skip-to-main link, focus trap + Esc/Tab cycle in Help modal, aria-selected toggle on mode tabs - new: scripts/check_lean_manifest.py — cross-checks lean_status.json against lean-taf source. Detects missing-in-source / line-drift / file-drift (errors) + new-in-source / commit-drift (info). --regenerate rewrites in-place. --strict for CI gate. - i18n: 5 new keys × EN/ES/FR/ZH (parity 0 missing / 0 extra) - 62/62 smoke tests passed locally Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>6e2328e -
v0.6: TAF Card hero+accordion UI + γ_check + Lean+Mathlib badges
- Hero strip (arch class + meta + 3 pills: count breakdown, γ headline, Anti-Ising) + 4 accordions: Recipes, Diagnostics, Verification, Provenance - New js/gamma_check.js: Padé closed-form + classifyRegime (normal/fraud/compressed/overpade/swa), browser-only - New js/lean_badges.js + data/lean_status.json: manifest of 37 theorems in 7 groups + 1 substantive finding (V_derivative_ne_RG_beta factor-2). Sortable theorem table, hero Anti-Ising pill links to Taf/Identities.lean#L188 (beta_chi_closure) - Verdict aggregation: count breakdown (✅ · ⚠ · ❌) replaces worst-of-N confusion when orthogonal recipes disagree - ⓘ tooltips inline: hero pills (3), Profile form params (6), Diagnostics keys (6), γ_check results (5) - Help v0.6 section: TAF Card layout + 5 regime use cases + Cardy ΔH + concrete Lean reverify command + 37/1 count, all 4 langs - i18n EN/ES/FR/ZH parity: 295+ keys per lang, 0 missing - gamma_check classifier verified 5/5 regimes via headless Chrome simulation Co-Authored-By: C…28ac122
-
feat(ui): add Diagnose CLI builder + Phase Diagram viewer modes
Diagnose CLI mode (🩺): - New mode tab + section in index.html with form: model id, theta, context N, --fast/--cpu/--load_in_4bit toggles, --local path. - Generates exact `python cli/diagnose_model.py ...` command, one-click copy to clipboard. - Bridges browser Padé prediction with the GPU CLI measurement script shipped at cli/diagnose_model.py. Phase diagram mode (📊): - Pure-canvas scatter (no Chart.js dep) reading data/master_gamma_results.json. - Plots γ_obs (y) vs log θ (x) for all 23 paper models, colored by phase (A/B/Hagedorn) with corpus alpha (text=opaque, random=40%). - Padé curve overlay at T=2000 + Hagedorn line γ=1. - Hover: full model details. Click: pre-fills Diagnose + Recipe forms. Other: - cli/context_compressor.js: D_f-based runtime token budgeting module (refactored from user proposal, validity guards γ ∈ [0.67, 0.85]). - i18n.js: added ~25 keys × 4 languages (EN/ES/FR/ZH) for the new modes. - style.css: .diag-cmd-box for terminal-style command renderin…b4f7029
-
feat(v0.3): inspector + what-if slider + falsification + community widget + registry bootstrap
Six features in one sprint: 1. 🔍 INSPECTOR mode (new tab): paste raw config.json → full TAF Profile. For private/in-development models not on HF Hub. Auto-detects family (RoPE-MHA/GQA, ALiBi, AbsPE, SWA, SSM) and runs all 5 recipes. 2. 🎚 What-if T_eval slider (in Profile output): Drag T_eval from 256 to 4×T_eval, see γ_Padé / γ_corrected / d_horizon / L_NIAH ceiling / verdict update LIVE in pure JS (no Pyodide call). Click '↻ Recompute full chain' to re-run profile at chosen T_eval. 3. 🔬 FALSIFICATION dashboard (new always-visible section): Live status of all 23 paper predictions (F1-F23): ✅ confirmed: 14 ⚠ partial: 3 ❌ refuted: 4 ⏳ untested: 0 Color-coded table with claim, status, evidence per row. Reads from embedded constant (no network). 4. 🌐 Community submissions widget (new always-visible section): Live feed from GitHub Issues API at karlesmarin/tafagent-registry. Shows last 15 submissions with verdict badge (color), title…c11b76c -
feat(v0.2): A+C community sharing — Download/Upload JSON + Submit to GitHub registry
Implements the community sharing model decided with user: A) Submit results to public GitHub Issues registry (zero infra cost) C) Download/Upload JSON locally (works offline, no auth) Three new buttons in EVERY output section (Profile, Compare, Recipe): - 💾 Download JSON: saves taf-{type}-{name}.json wrapped with metadata ({_taf_export, _taf_type, _taf_version, _taf_timestamp, payload}). - 📤 Submit to registry: opens new GitHub Issue at karlesmarin/tafagent-registry with title pre-filled (e.g. '[TAF X-2] YES') and body pre-rendered as Markdown summary + collapsible JSON. User just clicks Submit on GitHub. - 🔗 Copy share link: existing URL-encoded params (already worked). Plus a top-level 📂 Import section visible at all times: load any TAF JSON file from disk → renders Profile/Compare/Recipe view as if computed locally. Same view, no need to recompute. Storage strategy: - Recommendation: A + C (zero cost to us, GitHub hosts the registry) - Defer HF Dataset registry until tr…449213a -
feat(v0.2): full i18n coverage + round flag buttons + loading progress bar
UI improvements based on self-test feedback: - Round flag buttons (36px circular) with tooltip-on-hover label below. Active language gets glowing border + accent background. - Loading progress bar: shimmering indeterminate + discrete progress (Pyodide load → 50%, TAF load → 95%, ready → hide). Visible during the ~10-30s initial boot so users know something is happening. - Expanded data-i18n coverage to ALL visible text: * Mode tab buttons (Profile/Compare/Ask/Recipe) * Profile section: preset label, HF id label, fetch button, generate button * Compare section: recipe label, T_eval label, models title, compare button * Ask section: title, placeholder, analyze button, example button * Recipe section: title, default option * Form section: input title, preset label, HF label, fetch button, run button * Output sections: verdict title, chain title, chain desc, answer title, share button, TAF Card title, comparison title - New translation keys: profile.hf_placeholder,…ed3d534 -
feat(v0.2): i18n EN/ES/FR/ZH + quickstart banners + humble README + paper §sec:tafagent
- Multi-language UI with flag buttons (top-right): English, Español, Français, 中文. Persists in localStorage, auto-detect browser default. - Quickstart banner in Profile + Compare modes with concrete actionable example (addresses self-test feedback: 'no clear entry point'). - About line in hero: 'Built by an independent researcher. Open source. Not affiliated with any model vendor.' - Footer rewritten with paper repo + cost transparency. - README rewritten with humility + community-care framing: * 'A note before you read on' — independent, no funding, LLMs as research instruments * Architecture coverage table (which families work, which partial) * Languages section with flag list * 'How you can help' (bug reports, translations, falsifications, recipes) * Citation block (paper + tool) * Full Acknowledgements with model commons + infra commons + LLM research-instrument acknowledgement Paper change (separate repo karlesmarin/NeurIPS): - New §sec:tafagent inserted…137cb0a -
feat: Profile + Compare modes, hero rewrite, share URL
Tier 1 features for technicians evaluating models: - HERO: 'Test ANY transformer LLM before you spend GPU/$' + architecture coverage badges (RoPE-MHA, RoPE-GQA, ALiBi, AbsPE, SWA, SSM, Any HuggingFace public model) - 📇 PROFILE mode: paste model id → all 5 recipes run at once → unified TAF Card with verdict per dimension, key numbers (paper §26), falsification status (FALSIFICATION.md F1-F23), architecture class - 🆚 COMPARE mode: 2-3 models side-by-side on same recipe → table with verdicts color-coded + key numbers per row - 🔗 Share URL: encoded params, click to copy, restored on page load - Tooltips on every form field with paper context - Help modal updated with new modes + use cases New Python functions: - profile_model(): runs all 5 recipes + assembles TAF Card - compare_models(): one recipe across N models with key-number extraction Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>0386760 -
feat(ui): info tooltips, help modal, more visible verdict box
- Add (ⓘ) info icon next to every form field with hover tooltip explaining the parameter (theta, T_train, n_kv_heads, etc.) - Add 📘 Help button in header opening modal with full manual: what each recipe does, how to add models, parameter glossary, privacy notes, what verdicts mean. - Make verdict box much more prominent: bigger text (1.6rem), emoji prefix (✅/❌/⚠), thicker border, more padding. - Improve verdict-class detection for all verdict strings (YES/NO/GO/MEMORY-LIMITED/TINY-MODEL/USE SOFT/etc.) - Add console.log debug in renderResult to diagnose if verdict rendering fails (helps users report bugs). - Defensive null-checks on verdict/recipe_id/recipe_name fields. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>d0a945b -
feat: TAF Agent v0.1 — client-side transformer diagnostic
A Pyodide+WebLLM browser app that predicts transformer LLM viability (long-context, training budget, hardware fit, KV compression, custom-vs-API) using the TAF (Thermodynamic Attention Framework) formula chains from Marin 2026. Phase 1: Pyodide loads taf_browser.py (10 formulas, 11 model presets, 11 GPU catalog, deterministic Python, no server) Phase 2: WebLLM loads Llama-3.2-1B in browser → plain-English synthesis Phase 3: Free-form question router (LLM picks recipe + extracts params) Recipes (5): X-1 Custom training vs API X-2 Long Context Viability X-3 Budget Pre-flight X-5 Hardware Selection for serving X-19 KV Compression decision UI: 2 modes (Ask plain-English / Recipe + form), HF Hub config fetch for any public model, audit-trail expandable steps, mobile-responsive. Hosting: GitHub Pages (static); compute: user's browser; cost: \$0/mo. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>6ab0441