Method
How a claim gets onto this site.
The rules were written before any training. The first 17 tasks now run under them.
The repair tasks
Each task is a small app on Supabase with one thing wrong, or nothing wrong. 14 are vulnerable or broken and 3 are correct controls. The vulnerable ones cover 9 kinds of mistake.
- The model sees the task instruction and the database schema. It answers with SQL.
- Every SQL block in the answer is applied as its own transaction. A block that errors is skipped and counted.
- A hidden test then attacks the final database: it signs in as another user, as a visitor, and as the owner, and checks what each can read and change.
- We grade the final state and the test results, not what the model says it did.
- On a control task, the right answer is to change nothing that breaks the app.
The task list with results is Fig. 1 on the home page. Prompts, schemas and hidden tests are not published, so the tasks stay usable as a test.
Trials and settings
- Every task is run 3 times: once greedy, twice sampled. We report counts, not only percentages.
- Every model gets the same prompts, token limit and settings. The run page records the model, precision, thinking mode, token limit and seed.
- Each task runs against a throwaway Postgres container with a stand-in for Supabase’s auth and roles. It is not a full Supabase project.
- Eval tasks never go into training data. Tasks are split by template and repository family before any variants are made.
What a sample size can show
| Tasks | What it can tell apart from noise |
|---|---|
| about 20 | Only a gain of about 30 points or more. |
| about 250, paired | A gain of 10 points. |
We have 17 tasks. A gain smaller than that count can detect is reported here as “no measurable difference”.
What we compare against
- The untuned parent model. measured
- A linter that only detects. measured
- A stronger model with lint tools on the same tasks. not measured yet
Beating only the untuned parent says little about whether a model is useful.
Training data
- Original or explicitly licensed material only.
- No outputs from closed commercial models in the training data.
- If a teacher model generates data, it is an open-weight model under an MIT or Apache-2.0 licence.
- No customer or third-party code unless training use is authorized.
- Every item has a row in a manifest: source, licence, rights status, language, hash and split.
Arabic
- Any claim about Arabic names the variety tested: standard Arabic, Levantine, Egyptian, Gulf.
- A dialect is only claimed after a native speaker has judged the output.
- The Arabic versions of the task prompts are written and wait for that review. No Arabic eval has been run.