Add six SFT runs: audited runtimes, epoch metrics and article citation exports

README.md

@@ -1,5 +1,5 @@
 ---
-title: Data Agent RL Comparison
+title: Data Agent SFT and RL Comparison
 emoji: 📊
 colorFrom: blue
 colorTo: green
@@ -8,10 +8,18 @@ app_file: app.py
 pinned: false
 ---
 
-# Data Agent RL comparison
+# Data Agent SFT and RL comparison
 
 Training and evaluation results in one public Trackio project: **data-agent-rl-comparison**.
 
+## SFT runs
+
+Six SFT runs now share this project: LFM bash (907 and 4,677 tasks), LFM/Qwen OpenCode, and LFM/Qwen four-harness SFT. [Runtime table, epoch scores, caveats and downloadable data](SFT_RESULTS.md).
+
+[SFT overview](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=7e03598b74c5944b42002d1759e0184c%2Cd189e77a4f134099c308ed223b840405%2C2419d3a7bf21850a46903d5df20f3d59%2C60cde9f411f365b5c634ee3de1351505%2Ca9a405588dd8646aea613c02bf479c8b%2C736a2514f326ea6119699671e61bb0ef&smoothing=0&metric_filter=%5E%28eval_observed%2F%28pass_at_1%7Ctool_call_savings_pct%29%7Ctrain_sft%2Floss_mean50%7Ceval_coverage%2Fgraded_cells%29%24) · [SFT training](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=7e03598b74c5944b42002d1759e0184c%2Cd189e77a4f134099c308ed223b840405%2C2419d3a7bf21850a46903d5df20f3d59%2C60cde9f411f365b5c634ee3de1351505%2Ca9a405588dd8646aea613c02bf479c8b%2C736a2514f326ea6119699671e61bb0ef&smoothing=0&metric_filter=%5E%28train_sft%7Ctrain_summary%29%2F) · [SFT evaluations](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=7e03598b74c5944b42002d1759e0184c%2Cd189e77a4f134099c308ed223b840405%2C2419d3a7bf21850a46903d5df20f3d59%2C60cde9f411f365b5c634ee3de1351505%2Ca9a405588dd8646aea613c02bf479c8b%2C736a2514f326ea6119699671e61bb0ef&smoothing=0&metric_filter=%5E%28eval_observed%7Ceval_coverage%7Ceval_usage%7Ceval_efficiency%29%2F)
+
+`train_sft` contains loss, token accuracy, entropy, gradients and token counters; `train_summary` contains runtime and throughput. SFT uses the same `eval_observed`, `eval_coverage`, `eval_efficiency` and `eval_usage` groups as RL. SFT has no explicit efficiency reward; Qwen RL used correctness only, while LFM RL used correctness plus tool efficiency.
+
 ## LFM2.5-2.6B: OpenCode-only vs multi-harness
 
 Both runs reached 1,000 optimizer steps. Both final evaluations have 1,000/1,000 graded cases: **52.3% OpenCode-only**, **54.2% multi-harness**. The evaluation set is 250 fixed tasks × four harnesses, pass@1.

SFT_RESULTS.md

@@ -0,0 +1,35 @@
+# SFT training runtimes and evaluations
+
+Snapshot: 29 September 2026. Six completed two-epoch SFT runs, published alongside the existing RL runs in `data-agent-rl-comparison`.
+
+[Overview](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=7e03598b74c5944b42002d1759e0184c%2Cd189e77a4f134099c308ed223b840405%2C2419d3a7bf21850a46903d5df20f3d59%2C60cde9f411f365b5c634ee3de1351505%2Ca9a405588dd8646aea613c02bf479c8b%2C736a2514f326ea6119699671e61bb0ef&smoothing=0&metric_filter=%5E%28eval_observed%2F%28pass_at_1%7Ctool_call_savings_pct%29%7Ctrain_sft%2Floss_mean50%7Ceval_coverage%2Fgraded_cells%29%24) · [Training](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=7e03598b74c5944b42002d1759e0184c%2Cd189e77a4f134099c308ed223b840405%2C2419d3a7bf21850a46903d5df20f3d59%2C60cde9f411f365b5c634ee3de1351505%2Ca9a405588dd8646aea613c02bf479c8b%2C736a2514f326ea6119699671e61bb0ef&smoothing=0&metric_filter=%5E%28train_sft%7Ctrain_summary%29%2F) · [Evaluations](https://fineenvs-data-agent-training-comparison-trackio.hf.space/?project=data-agent-rl-comparison&run_ids=7e03598b74c5944b42002d1759e0184c%2Cd189e77a4f134099c308ed223b840405%2C2419d3a7bf21850a46903d5df20f3d59%2C60cde9f411f365b5c634ee3de1351505%2Ca9a405588dd8646aea613c02bf479c8b%2C736a2514f326ea6119699671e61bb0ef&smoothing=0&metric_filter=%5E%28eval_observed%7Ceval_coverage%7Ceval_usage%7Ceval_efficiency%29%2F)
+
+| Run | Tasks / examples | Train runtime | Mean loss | Loss: first / last 50 | Epoch 1 pass@1 | Epoch 2 pass@1 |
+|---|---:|---:|---:|---:|---:|---:|
+| LFM 2.6B · SFT bash · 907 tasks | 907 / 907 | 0h 09m 28s | 0.4464 | 0.5437 / 0.3974 | 41.74% (999/1000) | 46.59% (850/1000) |
+| LFM 2.6B · SFT bash · 4677 tasks (eval overlap) | 4,677 / 4,677 | 0h 45m 29s | 0.3533 | 0.5188 / 0.3372 | 41.30% (1000/1000) | 40.80% (1000/1000) |
+| LFM 2.6B · SFT OpenCode · 801 tasks | 801 / 4,825 | 1h 28m 17s | 0.2906 | 0.3997 / 0.2789 | 45.10% (1000/1000) | 47.50% (1000/1000) |
+| Qwen 2B · SFT OpenCode · 801 tasks | 801 / 4,825 | 1h 11m 27s | 0.2270 | 0.2825 / 0.2147 | 28.40% (1000/1000) | 26.50% (1000/1000) |
+| LFM 2.6B · SFT four harnesses · 888 tasks | 888 / 17,929 | 6h 26m 50s | 0.5815 | 0.9076 / 0.5854 | 38.30% (1000/1000) | 43.10% (1000/1000) |
+| Qwen 2B · SFT four harnesses · 888 tasks | 888 / 17,929 | 5h 18m 45s | 0.3844 | 0.5131 / 0.3797 | 30.90% (1000/1000) | 32.70% (1000/1000) |
+
+Runtime is the final Trainer `train_runtime`: training loop only, excluding queue time, preprocessing and evaluation. Loss is supervised cross entropy; absolute losses across different training datasets are not comparable. Training curves include every logged optimizer update; `loss_mean50` is a trailing mean with shorter windows at the start. Keep dashboard smoothing at 0. Historical ingestion timestamps are not original training timestamps.
+
+## Evaluation interpretation
+
+- Pass@1 uses 250 fixed tasks across OpenCode, Claude Code, Codex and Mini-SWE-Agent. Each harness has 33 easy, 118 medium and 99 hard tasks. Scores are fractions on Trackio.
+- Bash 907 epoch 1 has 999/1000 graded; epoch 2 has 850/1000. These are provisional, excluding missing cases. All other SFT epoch evaluations have 1000/1000 graded.
+- Bash 4,677 includes 33 notebook overlaps and four matching test questions. It is an overlap-inclusive experiment, not a clean held-out result.
+- The LFM pretrained baseline has 998/1000 graded. The Qwen baseline has 1000/1000 graded. A baseline repeated at step 0 is the same shared measurement, not another evaluation.
+- The x axis uses each run's optimizer steps; compare epochs as well, since dataset sizes and numbers of updates differ.
+- Tool savings = 100 × (1 − checkpoint calls / baseline calls), on task/harness pairs solved by both. Positive means fewer native ATIF tool calls. Matched cohort sizes are logged; cohorts differ between runs. This is descriptive, not a causal estimate.
+- SFT has no explicit tool-efficiency reward. Qwen RL used correctness only; LFM RL used correctness plus a tool-efficiency bonus.
+- Token means include all captured calls and repeated history; they are not cache-adjusted. Missing counters are omitted, never filled with zero.
+
+## Data and citation
+
+[Training summary CSV](sft_training_summary.csv) · [Training summary JSON](sft_training_summary.json) · [Epoch evaluations, harness breakdowns and efficiency](sft_evaluations.json)
+
+These aggregate exports include SHA-256 hashes of the source metrics and evaluation summaries. The public dashboard contains scalar metrics only. No rollout text, credentials or model weights are included. For a reproducible article citation, use the commit-pinned version of this document and exports rather than relying solely on the live dashboard.
+
+Training datasets: [bash demonstrations](https://huggingface.co/datasets/FineEnvs/SmolDataEnvs-sft), [OpenCode teacher rollouts](https://huggingface.co/datasets/AdithyaSK/qwen38-27b-harbor-rollouts), [multi-harness SFT](https://huggingface.co/datasets/FineEnvs/SmolDataEnvs-multiharness-sft).

app.py

@@ -1,6 +1,6 @@
 import os
 
-os.environ["TRACKIO_PLOT_ORDER"] = 'eval_observed/pass_at_1,eval_observed/combined_reward_normalized,eval_observed/tool_call_savings_pct,train_verified/reward_mean50,eval_coverage/graded_cells,eval_coverage/missing_cells,eval_observed/harness/*,eval_observed/difficulty/*,eval_efficiency/*,eval_usage/overall/*,eval_usage/harness/*,train_verified/reward,train_verified/tools/*,train_verified/completions/*,train_verified/batch/trained_tokens_per_step,train_verified/batch/forwarded_tokens_per_step,eval_observed/harness_difficulty/*,train_verified/*,eval_coverage/*,eval/pass_at_1,eval/provisional_pass_at_1,train/reward_rolling20,eval/harness/*,eval/difficulty/*'
+os.environ["TRACKIO_PLOT_ORDER"] = 'eval_observed/pass_at_1,eval_observed/combined_reward_normalized,eval_observed/tool_call_savings_pct,train_verified/reward_mean50,train_sft/loss_mean50,train_sft/*,train_summary/*,eval_coverage/graded_cells,eval_coverage/missing_cells,eval_observed/harness/*,eval_observed/difficulty/*,eval_efficiency/*,eval_usage/overall/*,eval_usage/harness/*,train_verified/reward,train_verified/tools/*,train_verified/completions/*,train_verified/batch/trained_tokens_per_step,train_verified/batch/forwarded_tokens_per_step,eval_observed/harness_difficulty/*,train_verified/*,eval_coverage/*,eval/pass_at_1,eval/provisional_pass_at_1,train/reward_rolling20,eval/harness/*,eval/difficulty/*'
 
 import trackio
 from trackio import server

sft_training_summary.csv

@@ -0,0 +1,7 @@
+tasks,examples,epoch_steps,runtime_seconds,mean_train_loss,loss_first50,loss_last50,metrics_sha256,run,model,epoch1_pass_at_1,epoch1_graded,epoch2_pass_at_1,epoch2_graded
+907,907,"[114, 228]",567.9285,0.44636775108805876,0.5436920684576034,0.3974331718683243,f4e40c769a103dbe20688c2b237a1effdaca280359dff33452d35a160d15784d,LFM 2.6B · SFT bash · 907 tasks,lfm,0.4174174174174174,999,0.46588235294117647,850
+4677,4677,"[585, 1170]",2729.0271,0.35326611715504247,0.5187726891040803,0.3372010153532028,6cdfaa6803faa4cf730c9c863d7ea80243d4bb4a38bf18c3a3e0b59795101eb0,LFM 2.6B · SFT bash · 4677 tasks (eval overlap),lfm,0.413,1000,0.408,1000
+801,4825,"[604, 1208]",5296.5726,0.29063189937270534,0.3997403588891029,0.2788516191393137,c7fa2e30faeb02f277076b6b12e4a395674973aabac185d1579a43b66d3b4ae5,LFM 2.6B · SFT OpenCode · 801 tasks,lfm,0.451,1000,0.475,1000
+801,4825,"[604, 1208]",4286.5479,0.22704624168155407,0.28249539285898206,0.21466757848858833,282145ba1de34e326eb947f2859473bf30d33dc426685c12910eb141e2c5fb61,Qwen 2B · SFT OpenCode · 801 tasks,qwen,0.284,1000,0.265,1000
+888,17929,"[2242, 4484]",23209.8465,0.5814626613510548,0.9076286560297012,0.5854089191555977,eb94a9046cee795466efba0eade4b14a8e9acac16c12dc967369ec6d2daf5bfc,LFM 2.6B · SFT four harnesses · 888 tasks,lfm,0.383,1000,0.431,1000
+888,17929,"[2242, 4484]",19125.4948,0.384378594270742,0.5131349170207977,0.37968474060297014,e44afd214cef850be4cb07db67999da4c33ba52784ea3e5aeb6c9902c2bb9012,Qwen 2B · SFT four harnesses · 888 tasks,qwen,0.309,1000,0.327,1000

sft_training_summary.json

@@ -0,0 +1,116 @@
+[
+  {
+    "tasks": 907,
+    "examples": 907,
+    "epoch_steps": [
+      114,
+      228
+    ],
+    "runtime_seconds": 567.9285,
+    "mean_train_loss": 0.44636775108805876,
+    "loss_first50": 0.5436920684576034,
+    "loss_last50": 0.3974331718683243,
+    "metrics_sha256": "f4e40c769a103dbe20688c2b237a1effdaca280359dff33452d35a160d15784d",
+    "run": "LFM 2.6B \u00b7 SFT bash \u00b7 907 tasks",
+    "model": "lfm",
+    "epoch1_pass_at_1": 0.4174174174174174,
+    "epoch1_graded": 999,
+    "epoch2_pass_at_1": 0.46588235294117647,
+    "epoch2_graded": 850
+  },
+  {
+    "tasks": 4677,
+    "examples": 4677,
+    "epoch_steps": [
+      585,
+      1170
+    ],
+    "runtime_seconds": 2729.0271,
+    "mean_train_loss": 0.35326611715504247,
+    "loss_first50": 0.5187726891040803,
+    "loss_last50": 0.3372010153532028,
+    "metrics_sha256": "6cdfaa6803faa4cf730c9c863d7ea80243d4bb4a38bf18c3a3e0b59795101eb0",
+    "run": "LFM 2.6B \u00b7 SFT bash \u00b7 4677 tasks (eval overlap)",
+    "model": "lfm",
+    "epoch1_pass_at_1": 0.413,
+    "epoch1_graded": 1000,
+    "epoch2_pass_at_1": 0.408,
+    "epoch2_graded": 1000
+  },
+  {
+    "tasks": 801,
+    "examples": 4825,
+    "epoch_steps": [
+      604,
+      1208
+    ],
+    "runtime_seconds": 5296.5726,
+    "mean_train_loss": 0.29063189937270534,
+    "loss_first50": 0.3997403588891029,
+    "loss_last50": 0.2788516191393137,
+    "metrics_sha256": "c7fa2e30faeb02f277076b6b12e4a395674973aabac185d1579a43b66d3b4ae5",
+    "run": "LFM 2.6B \u00b7 SFT OpenCode \u00b7 801 tasks",
+    "model": "lfm",
+    "epoch1_pass_at_1": 0.451,
+    "epoch1_graded": 1000,
+    "epoch2_pass_at_1": 0.475,
+    "epoch2_graded": 1000
+  },
+  {
+    "tasks": 801,
+    "examples": 4825,
+    "epoch_steps": [
+      604,
+      1208
+    ],
+    "runtime_seconds": 4286.5479,
+    "mean_train_loss": 0.22704624168155407,
+    "loss_first50": 0.28249539285898206,
+    "loss_last50": 0.21466757848858833,
+    "metrics_sha256": "282145ba1de34e326eb947f2859473bf30d33dc426685c12910eb141e2c5fb61",
+    "run": "Qwen 2B \u00b7 SFT OpenCode \u00b7 801 tasks",
+    "model": "qwen",
+    "epoch1_pass_at_1": 0.284,
+    "epoch1_graded": 1000,
+    "epoch2_pass_at_1": 0.265,
+    "epoch2_graded": 1000
+  },
+  {
+    "tasks": 888,
+    "examples": 17929,
+    "epoch_steps": [
+      2242,
+      4484
+    ],
+    "runtime_seconds": 23209.8465,
+    "mean_train_loss": 0.5814626613510548,
+    "loss_first50": 0.9076286560297012,
+    "loss_last50": 0.5854089191555977,
+    "metrics_sha256": "eb94a9046cee795466efba0eade4b14a8e9acac16c12dc967369ec6d2daf5bfc",
+    "run": "LFM 2.6B \u00b7 SFT four harnesses \u00b7 888 tasks",
+    "model": "lfm",
+    "epoch1_pass_at_1": 0.383,
+    "epoch1_graded": 1000,
+    "epoch2_pass_at_1": 0.431,
+    "epoch2_graded": 1000
+  },
+  {
+    "tasks": 888,
+    "examples": 17929,
+    "epoch_steps": [
+      2242,
+      4484
+    ],
+    "runtime_seconds": 19125.4948,
+    "mean_train_loss": 0.384378594270742,
+    "loss_first50": 0.5131349170207977,
+    "loss_last50": 0.37968474060297014,
+    "metrics_sha256": "e44afd214cef850be4cb07db67999da4c33ba52784ea3e5aeb6c9902c2bb9012",
+    "run": "Qwen 2B \u00b7 SFT four harnesses \u00b7 888 tasks",
+    "model": "qwen",
+    "epoch1_pass_at_1": 0.309,
+    "epoch1_graded": 1000,
+    "epoch2_pass_at_1": 0.327,
+    "epoch2_graded": 1000
+  }
+]