review-v3d-clean

Summary: conservative detection of familiar seeded operator mutations, mainly in C and Ruby. The adapter detects only 1 of 58 human-confirmed real bugs; this is not a reliable general bug detector. ARC-Challenge is a complementary science-QA benchmark, not a measure of code-review quality.

A LoRA adapter for Qwen3-8B that reviews one diff hunk at a time (C, Python, Ruby) and answers with a JSON list of concrete defects — or {"findings":[]} when the hunk is clean. It is tuned to stay silent on clean code: false positives on clean hunks are the primary metric.

It is the publishable retrain of the review-v3d reviewer described in the study reinaldosaraiva/review-v3d-study: same recipe, but trained only on open-source code (Django, Flask, requests, FastAPI, Linux, FRR, Open vSwitch, OVN, Rails) with labels from an Apache-2.0 teacher. The training, evaluation and gate data are published in reinaldosaraiva/review-v3d-clean-data.

Results

Measured on publishable data only: a 530-row gold set, 20 seeded A3/paired gate cases and 7 synthetic G5 controls. All four models get the same system prompt (system_prompt.txt) and the same cases. Brackets are Wilson 95 % intervals.

Gate Threshold review-v3d-clean review-v3d base, no adapter teacher
G5 silence on fixed controls 7/7 7/7 [65, 100] 7/7 [65, 100] 0/7 [0, 35] 4/7 [25, 84]
A3 detection on seeded bugs ≥ 8/20 12/20 [39, 78] 14/20 [48, 85] 16/20 [58, 92] 14/20 [48, 85]
FP on the un-mutated twins ≤ 2/20 0/20 [0, 16] 2/20 [3, 30] 15/20 [53, 89] 9/20 [26, 66]
Paired margin ≥ 1 nat (code pairs) ≥ 13/17 16/17 [73, 99] 14/17 [59, 94] 2/17 [3, 34] 6/17 [17, 59]
Clean-hunk FP, full gold ≤ 5.7 % 0.9 % (3/319) [0.3, 2.7] 1.6 % (5/319) [0.7, 3.6] 88.4 % [84.4, 91.5] 32.0 % [27.1, 37.3]
Clean-hunk FP, held-out split ≤ 5.7 % 0.0 % (0/159) [0.0, 2.4] 1.9 % (3/159) [0.6, 5.4] 88.1 % [82.1, 92.2] 34.0 % [27.1, 41.6]
Recall on defective gold hunks — 39.3 % (83/211) [33.0, 46.1] 42.2 % (89/211) [35.7, 48.9] 86.7 % [81.5, 90.7] 79.6 % [73.7, 84.5]
All gates pass yes yes no no

The base is mlx-community/Qwen3-8B-4bit without the adapter (PyTorch, bfloat16). The teacher is Qwen/Qwen3-Coder-30B-A3B-Instruct, run as MLX 4-bit on a Mac, so its backend differs from the other three. Output that does not parse as JSON with findings counts as no finding. This rule was fixed before the baselines ran. The base parsed 483/530 gold outputs, the teacher 522/530, and both adapters 530/530. The base flags almost every hunk, so its recall carries no signal on its own.

Precision across protocols: the code-review results for this adapter and its no-adapter base use bfloat16; the ARC evaluation uses float32 for both. Precision is matched within those comparisons. No controlled bfloat16-versus-float32 ablation was run, so the different tasks cannot isolate a precision effect.

Per language (gold set, Wilson 95 %):

Language Metric review-v3d-clean review-v3d base teacher
C clean FP 0/109 [0.0, 3.4] 3/109 [0.9, 7.8] 98/109 [82.8, 94.3] 50/109 [36.8, 55.2]
C recall, seeded 44/67 [53.7, 75.9] 51/67 [64.7, 84.7] 54/67 [69.6, 88.3] 56/67 [72.9, 90.6]
Python clean FP 2/146 [0.4, 4.9] 1/146 [0.1, 3.8] 128/146 [81.4, 92.1] 33/146 [16.6, 30.0]
Python recall, seeded 6/37 [7.7, 31.1] 15/37 [26.3, 56.5] 29/37 [62.8, 88.6] 30/37 [65.8, 90.5]
Python recall, real defects 1/47 [0.4, 11.1] 1/47 [0.4, 11.1] 45/47 [85.8, 98.8] 35/47 [60.5, 84.7]
Ruby clean FP 1/60 [0.3, 8.9] 1/60 [0.3, 8.9] 54/60 [79.9, 95.3] 18/60 [19.9, 42.5]
Ruby recall, seeded 32/60 [40.9, 65.4] 22/60 [25.6, 49.3] 55/60 [81.9, 96.4] 47/60 [66.4, 86.9]

C and Ruby positives are all seeded mutations, from the same generator family as the training contrastive pairs. The only real defects are the 47 Python hunks rebuilt from Django security fixes. The clean FP rows also include 4 Rust and shell hunks, which is why the language rows sum to 315 of 319.

review-v3d-clean versus review-v3d (exact McNemar test on the same items, two-sided, α = 0.05):

Comparison Only clean flags Only v3d flags Raw p Significant (unadjusted)
Clean-hunk FP, full gold 3 5 0.73 no
Recall, full gold 19 25 0.45 no
Recall, C 1 8 0.039 yes, v3d higher
Recall, Python 2 11 0.022 yes, v3d higher
Recall, Ruby 16 6 0.052 no
A3 detection (20) 1 3 0.63 no
FP on twins (20) 0 2 0.50 no
Paired margin (17) 3 1 0.63 no

The table reports unadjusted p-values. As a sensitivity check across the three language-recall comparisons only, Holm correction gives C: 0.0781, Python: 0.0674, Ruby: 0.0781; none is below 0.05 after that correction. The code-review record preserves the raw values and the exact scope of this adjustment. It does not apply a correction to every comparison in the card.

The gate differences between the two adapters are not significant. The gates have only 20, 17 and 7 cases, so a one-case swing moves a rate by 5 to 14 points, and the intervals above overlap widely. Passing them shows the adapter clears fixed thresholds on these cases. It does not rank it against review-v3d. The thresholds (5.7 %, 13/17 and the others) were committed before any result existed and are reported here unchanged. Against the base and the teacher, review-v3d-clean has significantly lower clean FP everywhere (p < 10⁻⁴ overall) and significantly lower recall in every language except C versus the base (p = 0.08).

The held-out split is fixed in advance (int(sha256(id), 16) % 2 == 1); checkpoint selection used the dev split only. Held-out numbers are reported, never used to choose.

Scope

What the numbers support: silent-by-default detection of the seeded operator mutations it was trained on, in C and Ruby diff hunks (inverted conditions, &&/|| swaps, off-by-one bounds, null checks, unsafe sprintf). Python and real-world defects are experimental. On the 47 real Python defects the adapter finds 1, the same as review-v3d. The teacher finds 35 and the base 45, but they also flag 32 % and 88 % of clean hunks.

What this model does not do (P015 validation, 2026-10-07)

Four pre-registered tests were run after publication; the full write-up is plans/P015-dossier.md in the study.

  • Real bugs from projects outside the training set: 2/93 found. 93 pairs (C 32, Python 30, Ruby 31) from curl, redis, libuv, sqlalchemy, pydantic, rubocop, puma, sidekiq, rack and jekyll, each pair = the fix hunk (control) and the same hunk inverted (bug reintroduced), commits after 2025-07-31, no overlap with training. The adapter flags 2 positives and 3 controls. review-v3d is identical (2/93). The base flags everything (93/93 both sides). The teacher flags 62 positives and 64 controls: no model tells the bug from its fix in an isolated hunk (best pair-level discrimination 18/93). These are the original full-set results. The later human audit confirmed 58 of the 93 pairs; on that subset, this adapter finds 1/58 bugs and flags 2/58 corrected controls (see the audit below).
  • Mutation families it never saw: ~10 % recall. Retrained with the same recipe but without one family (A: eq_swap+and_or; B: boundary_le+boundary_ge+c_boundary), recall on that family drops from 21/50 to 5/50 (A) and from 27/39 to 4/39 (B), p < 10⁻³, while recall on the other families stays equal to this model. Detection is memory of the seen mutation, not generalization.
  • The teacher cannot be made conservative enough to replace it on this gold. With a logit-margin threshold chosen on one half of the gold and evaluated on the other, no teacher operating point reaches clean-hunk FP ≤ 5.7 %; at this adapter's recall (15/35 on that half) the teacher pays 20/86 FP. A "conservative" system prompt reduces the teacher's greedy FP only from 140 to 117 of 319. So the low-FP property is real and specific to the adapter; it just does not come with real-bug detection.

Gold-set provenance

No gold label comes from the teacher that labelled the training data. At dataset construction, no row was confirmed by a person reading it hunk by hunk: human involvement consisted of approving batches prepared by AI agents. A later 100-item human audit is reported separately below; the original labels remain unchanged.

Origin (gold_by) Rows How it was labelled
sown 164 positives (C 67, Python 37, Ruby 60) deterministic mutation generator; the finding and its line come from the mutation
manual 87 Python (47 defects, 40 clean) Django security-fix hunks with the diff reversed; an AI agent drafted each label on a review sheet and the owner approved the sheet in one batch
prelabel-model 119 clean (C 109, Python 6, Rust 3, shell 1) pre-labelled clean by an unrecorded model in an earlier round, then confirmed clean 120/120 by an AI agent and approved by the owner in one batch; one row later proved to be a real bug and was removed
harness-review 160 clean (Python 100, Ruby 60) hunks from commits without a fix, pre-labelled by Qwen3.8-Flash-Next, each read by an AI agent harness and kept only when it saw no plausible defect

Completed human audit (2026-10-14)

One reviewer completed the 100-item gold audit and the 93-pair real-bug sheet. Human authorship was confirmed by the owner; it is a declaration, not independent verification. The initial sheets hid reference labels and model responses. There was no second independent human review. Twelve gold-label disagreements remain without final human adjudication; the technical follow-up below does not change the frozen labels.

Gold-label agreement: 88/100 = 88.0% (Wilson 95% CI 80.19–93.00%), Cohen's kappa 0.67123. The reviewer marked 23 defects and 77 clean hunks; the reference has 25 defects and 75 clean hunks. Five reference-clean items were marked defective and seven reference-defective items were marked clean. This is agreement with the existing labels, not inter-human agreement. The sample is stratified; its unweighted percentages must not be generalized to the entire dataset or production code.

Real-bug pairs: 58 confirmed, 6 rejected and 29 uncertain. The metrics below use only the 58 confirmed pairs, with the existing saved predictions; there was no new model inference or training.

review-v3d-clean metric Count Rate (Wilson 95% CI)
Recall on confirmed bug hunks 1/58 1.72% (0.31–9.14%)
False positives on their corrected controls 2/58 3.45% (0.95–11.73%)
Correct pairs: flag the bug and stay silent on the control 1/58 1.72% (0.31–9.14%)

These results are conditional on the reviewer's pair selection. They preserve the conclusion that real-bug detection is very low. The 3/319 clean-hunk FP result above is from the original gold set and is not replaced by 2/58 on these different controls. The original full 93-pair counts also remain unchanged.

Follow-up AI-assisted inspection found that four disputed seeded mutations change comments or message text, not Boolean operators; two other disputes concern test coverage rather than a direct production-code defect. Commit context also supplied counterevidence to some initial human claims. These are technical observations, not final human adjudications. No full upstream test suites were run for that inspection.

The audit record contains the aggregate results, subset definitions, unresolved disagreement IDs, source-file hashes and verification limits. Raw reviewer sheets are not included in this publication. Kappa within a subset whose reference labels are all clean is degenerate (zero with any disagreement, undefined with perfect agreement); it should not be interpreted as no observed agreement.

AI-assisted technical follow-up (2026-10-14)

The technical record gives a disposition for all 12 disputes: 4 proposed quarantines of textual mutations, 2 retained test-coverage cases, 3 retained clean labels, 1 retained defect and 2 proposed defects. These are AI-assisted technical proposals, not new human labels. The Windows performance issue was not reproduced on Windows; the Rails check used real methods with test doubles, without database integration. Complete pinned upstream revisions were tested, not every isolated gold hunk. Ten Python checks, two optional-dependency controls and ten Ruby combinations passed their stated expectations. Three selected upstream modules yielded 88 passed, 6 skipped; these were not full suites. A separate AI reviewer reran the focused checks and independently recomputed the metrics. The record preserves counterevidence: Django's default legacy-cookie fallback still accepts old collisions, and Requests already had an incorrectly targeted alias before the missing-import regression.

The following is post hoc label/scope sensitivity on the same saved predictions, not a new benchmark. Original metrics and metadata remain unchanged. The proposed scenarios do not estimate a model improvement.

Scenario Rows FP on clean hunks Recall
Original frozen labels 530 3/319 (0.940%) 83/211 (39.336%)
Quarantine 4 textual mutants; propose L034 and L060 as defects 526 3/317 (0.946%) 83/209 (39.713%)
Same proposals, but exclude L034 pending Windows reproduction 525 3/317 (0.946%) 83/208 (39.904%)
Proposed rubric, also excluding all 16 existing test-only rows 510 3/317 (0.946%) 83/193 (43.005%)

The final row applies a mechanical category filter, not 16 new adjudications or a validated real-bug benchmark. The model still detects the same 83 gold cases. Human-confirmed real-bug recall remains 1/58, with 2/58 false positives on corrected controls. The JSON contains Wilson intervals, per-language and original held-out-split slices, changed IDs, source revisions, fixture results and limits. Those post hoc slices are not a new independent held-out estimate. The human agreement remains 88/100.

Limitations

  • Python recall is low (7/84 vs 16/84 for review-v3d; raw p = 0.022, Holm-adjusted p = 0.067 across the three language-recall comparisons). The model is more conservative on Python than on C.
  • It mostly catches the seeded operator-level defects it was trained on. Real defects are rarely found (1/47 Django security fixes; 2/93 fix/pre-fix pairs from ten projects outside the training set), and unseen mutation families are mostly missed (see above).
  • The gate sets are small (20, 17 and 7 cases). Read their rates with the intervals above.
  • The gold labels were originally produced or confirmed by AI agents, with only batch approval by a person. The completed 100-item human audit agrees on 88 labels; 12 remain without final human adjudication. The original clean-hunk FP rate still depends on those reference labels. Only 1/58 human-confirmed real bugs was detected, with 2/58 false positives on their corrected controls.
  • One hunk per call, no repository context. line refers to the new side of the hunk as numbered in the diff.
  • claim and why are written in Portuguese; the JSON keys and CWE ids are language-neutral.
  • Treat findings as review hints, not verdicts.

Complementary ARC-Challenge evaluation (2026-10-14)

This is an author-run, complementary evaluation of multiple-choice science questions. It does not measure code-review quality, resolve the real-bug detection limitations above, or establish performance on other benchmarks. It is AI2 ARC-Challenge, not ARC-AGI. No third-party verification is claimed.

On the primary metric, the adapter lost 47 answers that the base got right and corrected 100 answers that the base got wrong: a net gain of 53/1,172. Both models were evaluated on the full test split with the same zero-shot likelihood protocol.

Metric Exact training base, no adapter review-v3d-clean Adapter minus base
acc_norm (primary) 643/1,172 (54.86%) 696/1,172 (59.39%) +4.52 percentage points
acc (secondary) 628/1,172 (53.58%) 679/1,172 (57.94%) +4.35 percentage points

For the primary metric, the paired 95% bootstrap interval for the gain is +2.56 to +6.57 percentage points; the exact two-sided McNemar test gives p = 0.0000146943. The bootstrap uses 20,000 resamples and seed 20261014. This supports an improvement on this benchmark under this protocol; it is not a claim that every answer improved or that general capabilities improved across tasks.

Mechanism remains untested. A shift in answer priors or likelihood calibration after tuning could contribute to the gain. Zhao et al. (2021) show that answer biases and contextual calibration can affect accuracy in other language-model settings; that paper does not establish the cause of this adapter's ARC result. We have not measured calibration error or run a causal ablation, so calibration is a hypothesis, not a demonstrated or ranked explanation. Acquisition of new science knowledge is not established.

Protocol and provenance:

  • Official lm-eval 0.4.13 arc_challenge task, dataset allenai/ai2_arc, configuration ARC-Challenge, test split, all 1,172 items.
  • Zero-shot, batch size 1, maximum context 2,048 tokens; no chat template, system instruction, or generated chain of thought. The standard prompt is Question: {question}\nAnswer: and the answer choices are scored as continuations. acc selects the highest log likelihood; acc_norm selects the highest log likelihood divided by answer length in characters.
  • The exact MLX 4-bit training base, mlx-community/Qwen3-8B-4bit at 545dc4251c05, was reconstructed in float32 with the published mlx_to_hf.py. The same model was evaluated first without the adapter, then with the unmerged PEFT adapter from review-v3d-clean at 7c8f7d059446.
  • One NVIDIA L40 (48 GB), PyTorch 2.14.0+cu130, Transformers 5.17.0, PEFT 0.21.0, datasets 5.1.0 and NumPy 2.5.3. Model loading and both evaluation passes took 788 seconds (about 13 minutes); environment setup is excluded.

The evaluation record contains exact revisions, hashes, statistics and settings. The per-item score trace contains dataset IDs, gold indices, answer lengths and both models' log likelihoods, allowing the reported scores to be recalculated. All 1,172 item scores were checked from the saved likelihoods. Pretraining contamination was not audited. Comparisons with external ARC scores require matching the few-shot count, prompt, base weights, loader, precision and scoring rule.

Output format

{"findings":[{"axis":"correctness","severity":"P1","line":3080,"cwe":"CWE-476",
  "claim":"teste de ponteiro nulo invertido, seguindo pelo ramo de dereferencia",
  "why":"o guard de NULL passa a executar justamente quando o ponteiro e nulo"}]}

axis ∈ correctness, security, regression, failure_handling, performance, tests, maintainability; severity ∈ P0–P3. The verdict is derived: any P0/P1 → block, only P2/P3 → warn, none → pass.

Quick start

The hosted inference widget is disabled (inference: false): the Hub would load the base through transformers, which cannot read these MLX 4-bit weights. Use the CLI below.

Both backends need the base weights mlx-community/Qwen3-8B-4bit at revision 545dc4251c05 (model.safetensors sha256 f2d29621aab30033…). The adapter was trained on those 4-bit weights dequantized to float, so do not substitute the bf16 Qwen/Qwen3-8B checkpoint.

The pinned base snapshot is 4.62 GB (4.31 GiB) to download in total, including 4,607,835,174 bytes for model.safetensors; the adapters and Python dependencies are additional. Download size differs from runtime memory because the loader dequantizes the weights. These sizes come from the Hub file metadata at the pinned revision; the evaluation record records the exact byte counts.

huggingface-cli download reinaldosaraiva/review-v3d-clean --local-dir review-v3d-clean
huggingface-cli download mlx-community/Qwen3-8B-4bit --revision 545dc4251c05 --local-dir qwen3-8b-4bit
cd review-v3d-clean

macOS (Apple Silicon, ≥ 32 GB unified memory for float32):

python3.12 -m venv .venv && .venv/bin/pip install -r requirements-macos.txt
.venv/bin/python review_v3d_clean.py cases --backend mlx --base ../qwen3-8b-4bit      # expects 47/47 identical
.venv/bin/python review_v3d_clean.py review --backend mlx --base ../qwen3-8b-4bit \
    --diff my.diff --file src/app.py --language python --subject "fix: guard empty list"

Linux x86_64 + NVIDIA GPU (≥ 32 GB VRAM for float32, CUDA 13 driver):

python3.12 -m venv .venv && .venv/bin/pip install -r requirements-linux.txt
.venv/bin/python review_v3d_clean.py cases --backend torch --base ../qwen3-8b-4bit    # expects 47/47 identical

--dtype bfloat16 halves memory (~16 GB); findings stay the same in practice, but line numbers can differ in the last digit, so cases is only byte-exact in float32. Intel Macs are not supported.

Files

Path Content
peft/ PEFT LoRA adapter (r 8, alpha 16, 36 layers, 252 modules) for PyTorch
mlx/ the same adapter converted for mlx_lm (float32 parity 47/47 with peft/)
review_v3d_clean.py inference CLI (review, cases) for both backends
mlx_to_hf.py numpy loader that dequantizes the MLX 4-bit base for PyTorch
system_prompt.txt the exact system prompt used in training (rubric + CWE catalogue)
cases.jsonl 47 test prompts with the expected float32 output
training/ effective training config, seed and checkpoint history
evaluation/ Code-review counts, ARC-Challenge scores, human audit and AI-assisted technical sensitivity
MANIFEST.sha256 sha256 of every file

Training

LoRA r 8 / alpha 16 on all attention and MLP projections of the dequantized Qwen3-8B-4bit, lr 2e-4, 2 epochs, batch of unit pairs, loss on the assistant turn with an extra decision-token term (see the study). Data: 749 teacher-labelled hunks (62 with a finding after a weak-finding filter, capped at 8.4 % positives) plus 800 seeded contrastive pairs (one mutation → one finding, the original → empty), boolean-operator family oversampled. Teacher: Qwen/Qwen3-Coder-30B-A3B-Instruct (MLX 4-bit). Checkpoint 5360 of 5360 was chosen by the dev-split rule (training/selection.json: recall on dev ≥ 39.4 %, then lowest clean FP on dev); training/summary.json lists the trainer's own validation-loss pick (4500), which was not used.

License and provenance

Adapter and code: Apache-2.0. The training data contains GPL-2.0 code (Linux, FRR, OVS datapath) and permissively licensed code; every row's origin and license are listed in the dataset. No code from private repositories was used — every training and evaluation row was checked against the private corpus by id, commit and normalized diff before release.

Downloads last month
-
MLX
Hardware compatibility
Log In to add your hardware

Quantized

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for reinaldosaraiva/review-v3d-clean

Finetuned
Qwen/Qwen3-8B
Adapter
(7)
this model

Dataset used to train reinaldosaraiva/review-v3d-clean

Paper for reinaldosaraiva/review-v3d-clean

Evaluation results