Instructions to use reinaldosaraiva/review-v3d-clean with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- PEFT
How to use reinaldosaraiva/review-v3d-clean with PEFT:
Task type is invalid.
- MLX
How to use reinaldosaraiva/review-v3d-clean with MLX:
# Make sure mlx-lm is installed # pip install --upgrade mlx-lm # if on a CUDA device, also pip install mlx[cuda] # Generate text with mlx-lm from mlx_lm import load, generate model, tokenizer = load("reinaldosaraiva/review-v3d-clean") prompt = "Once upon a time in" text = generate(model, tokenizer, prompt=prompt, verbose=True) - Notebooks
- Google Colab
- Kaggle
- Local Apps Settings
- LM Studio
- MLX LM
How to use reinaldosaraiva/review-v3d-clean with MLX LM:
Generate or start a chat session
# Install MLX LM uv tool install mlx-lm # Generate some text mlx_lm.generate --model "reinaldosaraiva/review-v3d-clean" --prompt "Once upon a time"
- Atomic Chat
review-v3d-clean
Summary: conservative detection of familiar seeded operator mutations, mainly in C and Ruby. The adapter detects only 1 of 58 human-confirmed real bugs; this is not a reliable general bug detector. ARC-Challenge is a complementary science-QA benchmark, not a measure of code-review quality.
A LoRA adapter for Qwen3-8B that reviews one diff hunk at a time (C, Python, Ruby) and answers with a JSON list
of concrete defects — or {"findings":[]} when the hunk is clean. It is tuned to stay silent on clean code:
false positives on clean hunks are the primary metric.
It is the publishable retrain of the review-v3d reviewer described in the study
reinaldosaraiva/review-v3d-study: same recipe, but trained
only on open-source code (Django, Flask, requests, FastAPI, Linux, FRR, Open vSwitch, OVN, Rails) with labels from
an Apache-2.0 teacher. The training, evaluation and gate data are published in
reinaldosaraiva/review-v3d-clean-data.
Results
Measured on publishable data only: a 530-row gold set, 20 seeded A3/paired gate cases and 7 synthetic G5 controls.
All four models get the same system prompt (system_prompt.txt) and the same cases. Brackets are Wilson 95 %
intervals.
| Gate | Threshold | review-v3d-clean | review-v3d | base, no adapter | teacher |
|---|---|---|---|---|---|
| G5 silence on fixed controls | 7/7 | 7/7 [65, 100] | 7/7 [65, 100] | 0/7 [0, 35] | 4/7 [25, 84] |
| A3 detection on seeded bugs | ≥ 8/20 | 12/20 [39, 78] | 14/20 [48, 85] | 16/20 [58, 92] | 14/20 [48, 85] |
| FP on the un-mutated twins | ≤ 2/20 | 0/20 [0, 16] | 2/20 [3, 30] | 15/20 [53, 89] | 9/20 [26, 66] |
| Paired margin ≥ 1 nat (code pairs) | ≥ 13/17 | 16/17 [73, 99] | 14/17 [59, 94] | 2/17 [3, 34] | 6/17 [17, 59] |
| Clean-hunk FP, full gold | ≤ 5.7 % | 0.9 % (3/319) [0.3, 2.7] | 1.6 % (5/319) [0.7, 3.6] | 88.4 % [84.4, 91.5] | 32.0 % [27.1, 37.3] |
| Clean-hunk FP, held-out split | ≤ 5.7 % | 0.0 % (0/159) [0.0, 2.4] | 1.9 % (3/159) [0.6, 5.4] | 88.1 % [82.1, 92.2] | 34.0 % [27.1, 41.6] |
| Recall on defective gold hunks | — | 39.3 % (83/211) [33.0, 46.1] | 42.2 % (89/211) [35.7, 48.9] | 86.7 % [81.5, 90.7] | 79.6 % [73.7, 84.5] |
| All gates pass | yes | yes | no | no |
The base is mlx-community/Qwen3-8B-4bit without the adapter (PyTorch, bfloat16). The teacher is
Qwen/Qwen3-Coder-30B-A3B-Instruct, run as MLX 4-bit on a Mac, so its backend differs from the other three. Output
that does not parse as JSON with findings counts as no finding. This rule was fixed before the baselines ran. The
base parsed 483/530 gold outputs, the teacher 522/530, and both adapters 530/530. The base flags almost every
hunk, so its recall carries no signal on its own.
Precision across protocols: the code-review results for this adapter and its no-adapter base use bfloat16; the ARC evaluation uses float32 for both. Precision is matched within those comparisons. No controlled bfloat16-versus-float32 ablation was run, so the different tasks cannot isolate a precision effect.
Per language (gold set, Wilson 95 %):
| Language | Metric | review-v3d-clean | review-v3d | base | teacher |
|---|---|---|---|---|---|
| C | clean FP | 0/109 [0.0, 3.4] | 3/109 [0.9, 7.8] | 98/109 [82.8, 94.3] | 50/109 [36.8, 55.2] |
| C | recall, seeded | 44/67 [53.7, 75.9] | 51/67 [64.7, 84.7] | 54/67 [69.6, 88.3] | 56/67 [72.9, 90.6] |
| Python | clean FP | 2/146 [0.4, 4.9] | 1/146 [0.1, 3.8] | 128/146 [81.4, 92.1] | 33/146 [16.6, 30.0] |
| Python | recall, seeded | 6/37 [7.7, 31.1] | 15/37 [26.3, 56.5] | 29/37 [62.8, 88.6] | 30/37 [65.8, 90.5] |
| Python | recall, real defects | 1/47 [0.4, 11.1] | 1/47 [0.4, 11.1] | 45/47 [85.8, 98.8] | 35/47 [60.5, 84.7] |
| Ruby | clean FP | 1/60 [0.3, 8.9] | 1/60 [0.3, 8.9] | 54/60 [79.9, 95.3] | 18/60 [19.9, 42.5] |
| Ruby | recall, seeded | 32/60 [40.9, 65.4] | 22/60 [25.6, 49.3] | 55/60 [81.9, 96.4] | 47/60 [66.4, 86.9] |
C and Ruby positives are all seeded mutations, from the same generator family as the training contrastive pairs. The only real defects are the 47 Python hunks rebuilt from Django security fixes. The clean FP rows also include 4 Rust and shell hunks, which is why the language rows sum to 315 of 319.
review-v3d-clean versus review-v3d (exact McNemar test on the same items, two-sided, α = 0.05):
| Comparison | Only clean flags | Only v3d flags | Raw p | Significant (unadjusted) |
|---|---|---|---|---|
| Clean-hunk FP, full gold | 3 | 5 | 0.73 | no |
| Recall, full gold | 19 | 25 | 0.45 | no |
| Recall, C | 1 | 8 | 0.039 | yes, v3d higher |
| Recall, Python | 2 | 11 | 0.022 | yes, v3d higher |
| Recall, Ruby | 16 | 6 | 0.052 | no |
| A3 detection (20) | 1 | 3 | 0.63 | no |
| FP on twins (20) | 0 | 2 | 0.50 | no |
| Paired margin (17) | 3 | 1 | 0.63 | no |
The table reports unadjusted p-values. As a sensitivity check across the three language-recall comparisons only, Holm correction gives C: 0.0781, Python: 0.0674, Ruby: 0.0781; none is below 0.05 after that correction. The code-review record preserves the raw values and the exact scope of this adjustment. It does not apply a correction to every comparison in the card.
The gate differences between the two adapters are not significant. The gates have only 20, 17 and 7 cases, so a
one-case swing moves a rate by 5 to 14 points, and the intervals above overlap widely. Passing them shows the
adapter clears fixed thresholds on these cases. It does not rank it against review-v3d. The thresholds (5.7 %,
13/17 and the others) were committed before any result existed and are reported here unchanged. Against the base
and the teacher, review-v3d-clean has significantly lower clean FP everywhere (p < 10⁻⁴ overall) and significantly
lower recall in every language except C versus the base (p = 0.08).
The held-out split is fixed in advance (int(sha256(id), 16) % 2 == 1); checkpoint selection used the dev split
only. Held-out numbers are reported, never used to choose.
Scope
What the numbers support: silent-by-default detection of the seeded operator mutations it was trained on, in C
and Ruby diff hunks (inverted conditions, &&/|| swaps, off-by-one bounds, null checks, unsafe sprintf).
Python and real-world defects are experimental. On the 47 real Python defects the adapter finds 1, the same as
review-v3d. The teacher finds 35 and the base 45, but they also flag 32 % and 88 % of clean hunks.
What this model does not do (P015 validation, 2026-10-07)
Four pre-registered tests were run after publication; the full write-up is plans/P015-dossier.md in the study.
- Real bugs from projects outside the training set: 2/93 found. 93 pairs (C 32, Python 30, Ruby 31) from
curl, redis, libuv, sqlalchemy, pydantic, rubocop, puma, sidekiq, rack and jekyll, each pair = the fix hunk
(control) and the same hunk inverted (bug reintroduced), commits after 2025-07-31, no overlap with training.
The adapter flags 2 positives and 3 controls.
review-v3dis identical (2/93). The base flags everything (93/93 both sides). The teacher flags 62 positives and 64 controls: no model tells the bug from its fix in an isolated hunk (best pair-level discrimination 18/93). These are the original full-set results. The later human audit confirmed 58 of the 93 pairs; on that subset, this adapter finds 1/58 bugs and flags 2/58 corrected controls (see the audit below). - Mutation families it never saw: ~10 % recall. Retrained with the same recipe but without one family
(A:
eq_swap+and_or; B:boundary_le+boundary_ge+c_boundary), recall on that family drops from 21/50 to 5/50 (A) and from 27/39 to 4/39 (B), p < 10⁻³, while recall on the other families stays equal to this model. Detection is memory of the seen mutation, not generalization. - The teacher cannot be made conservative enough to replace it on this gold. With a logit-margin threshold chosen on one half of the gold and evaluated on the other, no teacher operating point reaches clean-hunk FP ≤ 5.7 %; at this adapter's recall (15/35 on that half) the teacher pays 20/86 FP. A "conservative" system prompt reduces the teacher's greedy FP only from 140 to 117 of 319. So the low-FP property is real and specific to the adapter; it just does not come with real-bug detection.
Gold-set provenance
No gold label comes from the teacher that labelled the training data. At dataset construction, no row was confirmed by a person reading it hunk by hunk: human involvement consisted of approving batches prepared by AI agents. A later 100-item human audit is reported separately below; the original labels remain unchanged.
Origin (gold_by) |
Rows | How it was labelled |
|---|---|---|
sown |
164 positives (C 67, Python 37, Ruby 60) | deterministic mutation generator; the finding and its line come from the mutation |
manual |
87 Python (47 defects, 40 clean) | Django security-fix hunks with the diff reversed; an AI agent drafted each label on a review sheet and the owner approved the sheet in one batch |
prelabel-model |
119 clean (C 109, Python 6, Rust 3, shell 1) | pre-labelled clean by an unrecorded model in an earlier round, then confirmed clean 120/120 by an AI agent and approved by the owner in one batch; one row later proved to be a real bug and was removed |
harness-review |
160 clean (Python 100, Ruby 60) | hunks from commits without a fix, pre-labelled by Qwen3.8-Flash-Next, each read by an AI agent harness and kept only when it saw no plausible defect |
Completed human audit (2026-10-14)
One reviewer completed the 100-item gold audit and the 93-pair real-bug sheet. Human authorship was confirmed by the owner; it is a declaration, not independent verification. The initial sheets hid reference labels and model responses. There was no second independent human review. Twelve gold-label disagreements remain without final human adjudication; the technical follow-up below does not change the frozen labels.
Gold-label agreement: 88/100 = 88.0% (Wilson 95% CI 80.19–93.00%), Cohen's kappa 0.67123. The reviewer marked 23 defects and 77 clean hunks; the reference has 25 defects and 75 clean hunks. Five reference-clean items were marked defective and seven reference-defective items were marked clean. This is agreement with the existing labels, not inter-human agreement. The sample is stratified; its unweighted percentages must not be generalized to the entire dataset or production code.
Real-bug pairs: 58 confirmed, 6 rejected and 29 uncertain. The metrics below use only the 58 confirmed pairs, with the existing saved predictions; there was no new model inference or training.
| review-v3d-clean metric | Count | Rate (Wilson 95% CI) |
|---|---|---|
| Recall on confirmed bug hunks | 1/58 | 1.72% (0.31–9.14%) |
| False positives on their corrected controls | 2/58 | 3.45% (0.95–11.73%) |
| Correct pairs: flag the bug and stay silent on the control | 1/58 | 1.72% (0.31–9.14%) |
These results are conditional on the reviewer's pair selection. They preserve the conclusion that real-bug detection is very low. The 3/319 clean-hunk FP result above is from the original gold set and is not replaced by 2/58 on these different controls. The original full 93-pair counts also remain unchanged.
Follow-up AI-assisted inspection found that four disputed seeded mutations change comments or message text, not Boolean operators; two other disputes concern test coverage rather than a direct production-code defect. Commit context also supplied counterevidence to some initial human claims. These are technical observations, not final human adjudications. No full upstream test suites were run for that inspection.
The audit record contains the aggregate results, subset definitions, unresolved disagreement IDs, source-file hashes and verification limits. Raw reviewer sheets are not included in this publication. Kappa within a subset whose reference labels are all clean is degenerate (zero with any disagreement, undefined with perfect agreement); it should not be interpreted as no observed agreement.
AI-assisted technical follow-up (2026-10-14)
The technical record gives a disposition for all 12 disputes: 4 proposed quarantines of textual mutations, 2 retained test-coverage cases, 3 retained clean labels, 1 retained defect and 2 proposed defects. These are AI-assisted technical proposals, not new human labels. The Windows performance issue was not reproduced on Windows; the Rails check used real methods with test doubles, without database integration. Complete pinned upstream revisions were tested, not every isolated gold hunk. Ten Python checks, two optional-dependency controls and ten Ruby combinations passed their stated expectations. Three selected upstream modules yielded 88 passed, 6 skipped; these were not full suites. A separate AI reviewer reran the focused checks and independently recomputed the metrics. The record preserves counterevidence: Django's default legacy-cookie fallback still accepts old collisions, and Requests already had an incorrectly targeted alias before the missing-import regression.
The following is post hoc label/scope sensitivity on the same saved predictions, not a new benchmark. Original metrics and metadata remain unchanged. The proposed scenarios do not estimate a model improvement.
| Scenario | Rows | FP on clean hunks | Recall |
|---|---|---|---|
| Original frozen labels | 530 | 3/319 (0.940%) | 83/211 (39.336%) |
| Quarantine 4 textual mutants; propose L034 and L060 as defects | 526 | 3/317 (0.946%) | 83/209 (39.713%) |
| Same proposals, but exclude L034 pending Windows reproduction | 525 | 3/317 (0.946%) | 83/208 (39.904%) |
| Proposed rubric, also excluding all 16 existing test-only rows | 510 | 3/317 (0.946%) | 83/193 (43.005%) |
The final row applies a mechanical category filter, not 16 new adjudications or a validated real-bug benchmark. The model still detects the same 83 gold cases. Human-confirmed real-bug recall remains 1/58, with 2/58 false positives on corrected controls. The JSON contains Wilson intervals, per-language and original held-out-split slices, changed IDs, source revisions, fixture results and limits. Those post hoc slices are not a new independent held-out estimate. The human agreement remains 88/100.
Limitations
- Python recall is low (7/84 vs 16/84 for
review-v3d; raw p = 0.022, Holm-adjusted p = 0.067 across the three language-recall comparisons). The model is more conservative on Python than on C. - It mostly catches the seeded operator-level defects it was trained on. Real defects are rarely found (1/47 Django security fixes; 2/93 fix/pre-fix pairs from ten projects outside the training set), and unseen mutation families are mostly missed (see above).
- The gate sets are small (20, 17 and 7 cases). Read their rates with the intervals above.
- The gold labels were originally produced or confirmed by AI agents, with only batch approval by a person. The completed 100-item human audit agrees on 88 labels; 12 remain without final human adjudication. The original clean-hunk FP rate still depends on those reference labels. Only 1/58 human-confirmed real bugs was detected, with 2/58 false positives on their corrected controls.
- One hunk per call, no repository context.
linerefers to the new side of the hunk as numbered in the diff. claimandwhyare written in Portuguese; the JSON keys and CWE ids are language-neutral.- Treat findings as review hints, not verdicts.
Complementary ARC-Challenge evaluation (2026-10-14)
This is an author-run, complementary evaluation of multiple-choice science questions. It does not measure code-review quality, resolve the real-bug detection limitations above, or establish performance on other benchmarks. It is AI2 ARC-Challenge, not ARC-AGI. No third-party verification is claimed.
On the primary metric, the adapter lost 47 answers that the base got right and corrected 100 answers that the base got wrong: a net gain of 53/1,172. Both models were evaluated on the full test split with the same zero-shot likelihood protocol.
| Metric | Exact training base, no adapter | review-v3d-clean | Adapter minus base |
|---|---|---|---|
acc_norm (primary) |
643/1,172 (54.86%) | 696/1,172 (59.39%) | +4.52 percentage points |
acc (secondary) |
628/1,172 (53.58%) | 679/1,172 (57.94%) | +4.35 percentage points |
For the primary metric, the paired 95% bootstrap interval for the gain is +2.56 to +6.57 percentage points; the exact two-sided McNemar test gives p = 0.0000146943. The bootstrap uses 20,000 resamples and seed 20261014. This supports an improvement on this benchmark under this protocol; it is not a claim that every answer improved or that general capabilities improved across tasks.
Mechanism remains untested. A shift in answer priors or likelihood calibration after tuning could contribute to the gain. Zhao et al. (2021) show that answer biases and contextual calibration can affect accuracy in other language-model settings; that paper does not establish the cause of this adapter's ARC result. We have not measured calibration error or run a causal ablation, so calibration is a hypothesis, not a demonstrated or ranked explanation. Acquisition of new science knowledge is not established.
Protocol and provenance:
- Official
lm-eval0.4.13arc_challengetask, datasetallenai/ai2_arc, configurationARC-Challenge, test split, all 1,172 items. - Zero-shot, batch size 1, maximum context 2,048 tokens; no chat template, system instruction, or generated
chain of thought. The standard prompt is
Question: {question}\nAnswer:and the answer choices are scored as continuations.accselects the highest log likelihood;acc_normselects the highest log likelihood divided by answer length in characters. - The exact MLX 4-bit training base,
mlx-community/Qwen3-8B-4bitat545dc4251c05, was reconstructed in float32 with the publishedmlx_to_hf.py. The same model was evaluated first without the adapter, then with the unmerged PEFT adapter fromreview-v3d-cleanat7c8f7d059446. - One NVIDIA L40 (48 GB), PyTorch 2.14.0+cu130, Transformers 5.17.0, PEFT 0.21.0, datasets 5.1.0 and NumPy 2.5.3. Model loading and both evaluation passes took 788 seconds (about 13 minutes); environment setup is excluded.
The evaluation record contains exact revisions, hashes, statistics and settings. The per-item score trace contains dataset IDs, gold indices, answer lengths and both models' log likelihoods, allowing the reported scores to be recalculated. All 1,172 item scores were checked from the saved likelihoods. Pretraining contamination was not audited. Comparisons with external ARC scores require matching the few-shot count, prompt, base weights, loader, precision and scoring rule.
Output format
{"findings":[{"axis":"correctness","severity":"P1","line":3080,"cwe":"CWE-476",
"claim":"teste de ponteiro nulo invertido, seguindo pelo ramo de dereferencia",
"why":"o guard de NULL passa a executar justamente quando o ponteiro e nulo"}]}
axis ∈ correctness, security, regression, failure_handling, performance, tests, maintainability; severity ∈
P0–P3. The verdict is derived: any P0/P1 → block, only P2/P3 → warn, none → pass.
Quick start
The hosted inference widget is disabled (inference: false): the Hub would load the base through
transformers, which cannot read these MLX 4-bit weights. Use the CLI below.
Both backends need the base weights mlx-community/Qwen3-8B-4bit at revision 545dc4251c05
(model.safetensors sha256 f2d29621aab30033…). The adapter was trained on those 4-bit weights dequantized to
float, so do not substitute the bf16 Qwen/Qwen3-8B checkpoint.
The pinned base snapshot is 4.62 GB (4.31 GiB) to download in total, including 4,607,835,174 bytes for
model.safetensors; the adapters and Python dependencies are additional. Download size differs from runtime
memory because the loader dequantizes the weights. These sizes come from the Hub file metadata at the pinned
revision; the evaluation record records the exact byte counts.
huggingface-cli download reinaldosaraiva/review-v3d-clean --local-dir review-v3d-clean
huggingface-cli download mlx-community/Qwen3-8B-4bit --revision 545dc4251c05 --local-dir qwen3-8b-4bit
cd review-v3d-clean
macOS (Apple Silicon, ≥ 32 GB unified memory for float32):
python3.12 -m venv .venv && .venv/bin/pip install -r requirements-macos.txt
.venv/bin/python review_v3d_clean.py cases --backend mlx --base ../qwen3-8b-4bit # expects 47/47 identical
.venv/bin/python review_v3d_clean.py review --backend mlx --base ../qwen3-8b-4bit \
--diff my.diff --file src/app.py --language python --subject "fix: guard empty list"
Linux x86_64 + NVIDIA GPU (≥ 32 GB VRAM for float32, CUDA 13 driver):
python3.12 -m venv .venv && .venv/bin/pip install -r requirements-linux.txt
.venv/bin/python review_v3d_clean.py cases --backend torch --base ../qwen3-8b-4bit # expects 47/47 identical
--dtype bfloat16 halves memory (~16 GB); findings stay the same in practice, but line numbers can differ in
the last digit, so cases is only byte-exact in float32. Intel Macs are not supported.
Files
| Path | Content |
|---|---|
peft/ |
PEFT LoRA adapter (r 8, alpha 16, 36 layers, 252 modules) for PyTorch |
mlx/ |
the same adapter converted for mlx_lm (float32 parity 47/47 with peft/) |
review_v3d_clean.py |
inference CLI (review, cases) for both backends |
mlx_to_hf.py |
numpy loader that dequantizes the MLX 4-bit base for PyTorch |
system_prompt.txt |
the exact system prompt used in training (rubric + CWE catalogue) |
cases.jsonl |
47 test prompts with the expected float32 output |
training/ |
effective training config, seed and checkpoint history |
evaluation/ |
Code-review counts, ARC-Challenge scores, human audit and AI-assisted technical sensitivity |
MANIFEST.sha256 |
sha256 of every file |
Training
LoRA r 8 / alpha 16 on all attention and MLP projections of the dequantized Qwen3-8B-4bit, lr 2e-4, 2 epochs,
batch of unit pairs, loss on the assistant turn with an extra decision-token term (see the study). Data:
749 teacher-labelled hunks (62 with a finding after a weak-finding filter, capped at 8.4 % positives) plus 800
seeded contrastive pairs (one mutation → one finding, the original → empty), boolean-operator family oversampled.
Teacher: Qwen/Qwen3-Coder-30B-A3B-Instruct (MLX 4-bit). Checkpoint 5360 of 5360 was chosen by the dev-split rule
(training/selection.json: recall on dev ≥ 39.4 %, then lowest clean FP on dev); training/summary.json lists the
trainer's own validation-loss pick (4500), which was not used.
License and provenance
Adapter and code: Apache-2.0. The training data contains GPL-2.0 code (Linux, FRR, OVS datapath) and permissively licensed code; every row's origin and license are listed in the dataset. No code from private repositories was used — every training and evaluation row was checked against the private corpus by id, commit and normalized diff before release.
- Downloads last month
- -
Quantized
Model tree for reinaldosaraiva/review-v3d-clean
Dataset used to train reinaldosaraiva/review-v3d-clean
Paper for reinaldosaraiva/review-v3d-clean
Evaluation results
- Clean-hunk false positives (3/319; %, lower is better) on Code-review gold (530 hunks; original labels, audit disputes unresolved)Author-run evaluation on frozen reference labels; see human audit caveats0.940
- Defective-hunk recall (83/211; %, mixed seeded and real defects) on Code-review gold (530 hunks; original labels, audit disputes unresolved)Author-run evaluation on frozen reference labels; see human audit caveats39.336
- ARC-Challenge accuracy normalized (0-shot, %) on ARC-Challenge (complementary, 0-shot, full test)test set Author-run lm-eval 0.4.13 (2026-10-14)59.386
- ARC-Challenge accuracy (0-shot, %) on ARC-Challenge (complementary, 0-shot, full test)test set Author-run lm-eval 0.4.13 (2026-10-14)57.935