Run log · Repair eval ·
Qwen3.5-4B, untuned, 10 tasks
2026-11-22_eval_Qwen3.5-4B_en
Each task is a small Supabase app. The model sees the schema and proposes SQL. Every SQL block is applied as its own transaction, blocks that error are skipped, and a hidden test then attacks the final database. Trial 1 is greedy, trials 2 and 3 are sampled. Prompts, answers and tests are not published, so the tasks stay usable.
Recorded fields
- Model
- unsloth/Qwen3.5-4B
- Adapter
- none, untuned parent
- Loaded in 4-bit
- no, 16-bit
- Layers offloaded to system RAM
- no
- Prompt language
- en
- Trials per task
- 3
- Token limit per answer
- 1024
- Thinking mode
- off (enable_thinking=False)
- Seed
- 3407
- PyTorch
- 2.14.1+cu130
- Tasks
- 10
- Trials
- 30
- Trials passed
- 20
- Tasks passed in every trial
- 6
- Tasks passed at least once
- 8
- Wall time
- 713 s
- Peak VRAM
- 8.66 GiB
- Times re-graded from stored answers
- 1
- Grading policy
- per-block transactions, errored blocks skipped, final state graded (22 Nov 2026)
- Hardware
- RTX 4070 Ti, 12 GB
Per task
| Task | What is wrong in the app | Trials | Passed |
|---|---|---|---|
| clean-001 | Correct per-user bookmarks table (control: nothing to fix) | 3 of 3 | |
| clean-002 | Blog posts readable by everyone on purpose, writable by the author only (control) | 3 of 3 | |
| rls-001 | Personal notes table with RLS never enabled | 3 of 3 | |
| rls-002 | RLS enabled but the select policy lets everyone read private orders | 3 of 3 | |
| rls-003 | Profiles: anyone, including anonymous, can update any profile | 3 of 3 | |
| rls-004 | Shared team board: update policy USING (true) lets any member take over or rewrite another member's task | 1 of 3 | |
| rls-005 | Comments: insert policy accepts any user_id, so a user can post as someone else | 0 of 3 | |
| rls-006 | RLS is correct, but a SECURITY DEFINER function exposed to the API returns every invoice | 3 of 3 | |
| rls-007 | Multi-tenant documents: the policy is right but users can insert their own membership into any org | 0 of 3 | |
| rls-008 | Public profiles expose email and phone to every signed-in user | 1 of 3 |