Run log · Repair eval ·

Qwen3.5-4B, untuned, 10 tasks

2026-11-22_eval_Qwen3.5-4B_en

Each task is a small Supabase app. The model sees the schema and proposes SQL. Every SQL block is applied as its own transaction, blocks that error are skipped, and a hidden test then attacks the final database. Trial 1 is greedy, trials 2 and 3 are sampled. Prompts, answers and tests are not published, so the tasks stay usable.

Recorded fields

Model
unsloth/Qwen3.5-4B
Adapter
none, untuned parent
Loaded in 4-bit
no, 16-bit
Layers offloaded to system RAM
no
Prompt language
en
Trials per task
3
Token limit per answer
1024
Thinking mode
off (enable_thinking=False)
Seed
3407
PyTorch
2.14.1+cu130
Tasks
10
Trials
30
Trials passed
20
Tasks passed in every trial
6
Tasks passed at least once
8
Wall time
713 s
Peak VRAM
8.66 GiB
Times re-graded from stored answers
1
Grading policy
per-block transactions, errored blocks skipped, final state graded (22 Nov 2026)
Hardware
RTX 4070 Ti, 12 GB

Per task

TaskWhat is wrong in the appTrialsPassed
clean-001Correct per-user bookmarks table (control: nothing to fix)3 of 3
clean-002Blog posts readable by everyone on purpose, writable by the author only (control)3 of 3
rls-001Personal notes table with RLS never enabled3 of 3
rls-002RLS enabled but the select policy lets everyone read private orders3 of 3
rls-003Profiles: anyone, including anonymous, can update any profile3 of 3
rls-004Shared team board: update policy USING (true) lets any member take over or rewrite another member's task1 of 3
rls-005Comments: insert policy accepts any user_id, so a user can post as someone else0 of 3
rls-006RLS is correct, but a SECURITY DEFINER function exposed to the API returns every invoice3 of 3
rls-007Multi-tenant documents: the policy is right but users can insert their own membership into any org0 of 3
rls-008Public profiles expose email and phone to every signed-in user1 of 3