Run log · Repair eval ·
Qwen3.5-4B, untuned, 7 tasks
2026-11-22_eval_Qwen3.5-4B_batch2_en
Each task is a small Supabase app. The model sees the schema and proposes SQL. Every SQL block is applied as its own transaction, blocks that error are skipped, and a hidden test then attacks the final database. Trial 1 is greedy, trials 2 and 3 are sampled. Prompts, answers and tests are not published, so the tasks stay usable.
Recorded fields
- Model
- unsloth/Qwen3.5-4B
- Adapter
- none, untuned parent
- Loaded in 4-bit
- no, 16-bit
- Layers offloaded to system RAM
- no
- Prompt language
- en
- Trials per task
- 3
- Token limit per answer
- 1024
- Thinking mode
- off (enable_thinking=False)
- Seed
- 3407
- PyTorch
- 2.14.1+cu130
- Tasks
- 7
- Trials
- 21
- Trials passed
- 6
- Tasks passed in every trial
- 1
- Tasks passed at least once
- 3
- Wall time
- 531 s
- Peak VRAM
- 8.65 GiB
- Times re-graded from stored answers
- 1
- Grading policy
- per-block transactions, errored blocks skipped, final state graded (22 Nov 2026)
- Hardware
- RTX 4070 Ti, 12 GB
Per task
| Task | What is wrong in the app | Trials | Passed |
|---|---|---|---|
| clean-003 | Announcements: public read of published rows, no client writes at all (control) | 2 of 3 | |
| rls-009 | Notes: RLS on, no DELETE policy, so users cannot delete their own notes (secure but broken) | 3 of 3 | |
| rls-010 | Tasks table: the owner policies are granted TO anon instead of authenticated, so signed-in users see nothing | 0 of 3 | |
| rls-011 | Admin check trusts user_metadata in the JWT, which any user can edit from the client | 0 of 3 | |
| rls-012 | App config table holds a Stripe secret key and is readable by anonymous visitors | 1 of 3 | |
| rls-013 | A reporting view over the orders table runs with the owner's rights and bypasses RLS | 0 of 3 | |
| rls-014 | Org documents: any member can delete every document in the org; only org admins should | 0 of 3 |