Independently verify 12 AI coding-benchmark test findings
Project brief
US-based independent Python engineer needed to check whether 12 reported benchmark flaws are real. Disagreeing with a finding is a valid result.
For each item, run the supplied benchmark tests and reference/candidate check commands, inspect the code and task brief, then complete five short answers and a note in the CSV sheet. This is verification, not creating new benchmark problems.
Use Docker and pytest/unittest; allow roughly 100GB of disk space. The source workload estimate is 4-6 hours. Relevant ML-evaluation or SWE-bench experience helps.
Authorized findings, code, commands and answer-sheet prompts must be agreed before work. A confidentiality agreement and disclosure timing are required; choose named or anonymous permission for describing the review. No confidential findings should appear in an application.
Briefly describe a test suite you found that accepted incorrect code.
Deliverables & acceptance
What you'll deliver
- Completed 12-item CSV with five answers and a note per finding, supported by execution evidence.
What the result must meet
- Evidence compares the supplied reference and candidate checks with benchmark-test results; conclusions identify confirmed, disputed or inconclusive findings.
- All 12 records are complete; uncertainty is explained rather than forcing confirmation, and confidentiality/reporting permissions are documented.