Case file · GenLayer · Studionet
Matched SWE-bench Verified
Opus 4.1 scores 72.4 vs 54.6 for GPT-4.1 under identical harness, temperature 0, one retry.
The score is not backed by enough methodology or artifacts.
01 · What was claimed
The published sentence
- Model A
- Claude Opus 4.1 · 2026-02-19
- Model B / baseline
- GPT-4.1 · gpt-4.1-2025-04-14
- Benchmark
- SWE-bench Verified · verified-500-2025-03-15
- Evaluation date
- 2026-03-02
- Metric
- resolved %
- Reported result
- 72.4 vs 54.6
- Claimant
- 0x427cd9730F79901A3A30cb89cCC2D6B9480DF076
- Published
- 2026-09-12T01:32:07.000Z
Methodology
Identical OpenHands harness, temperature 0, retries 1, matched prompts, failed runs unresolved.
Retries
1 vs 1
Temperature
— vs —
Tools
— vs —
Prompts
matched
System prompts
undisclosed
Sample
undisclosed
02 · What evidence was provided
Off-chain artifacts, on-chain hashes
Large files are not stored in the Intelligent Contract. The contract holds canonical hashes and references. The index below is the case file.
OpenHands harness used for both models.
sha256:demo
03 · What was challenged
Public review
Challenge 01 · Reproducibility problem
Need raw trajectories
The claim is otherwise matched; requesting the raw CSV for independent audit.
0x6a5e0671B32b612B73CeF368DADD2780eE5B4D69
04 · What was decided
Verdict
Scores and matched-conditions claim are stated but raw results, trajectories or full logs are absent; only harness reference is supplied.
Key findings
- Benchmark, versions and stated parameters (temp 0, 1 retry, matched prompts) are disclosed
- Single evidence item is harness URI only; no raw CSV, per-instance outcomes or logs
- Open challenge explicitly requests raw trajectories for reproducibility audit
Material issues
- Absence of raw results prevents independent verification of reported 72.4 vs 54.6
Limitations
- Evaluation cannot confirm exact numerical match or exclusion policy without artifacts
05 · Verification
On-chain identifiers
- Claim id
- onchain-1
- On-chain id
- 1
- Network
- studionet · chain 61999
- Tx state
- finalized
- Transaction
- —
Off-chain rows never override a finalized on-chain verdict. Seeded case files illustrate the rubric; live GenLayer consensus is marked explicitly. Open contract in Studio · Live evaluation tx