Run log · Repair eval ·

Qwen3.5-4B, untuned, 7 tasks

2026-11-22_eval_Qwen3.5-4B_batch2_en

Each task is a small Supabase app. The model sees the schema and proposes SQL. Every SQL block is applied as its own transaction, blocks that error are skipped, and a hidden test then attacks the final database. Trial 1 is greedy, trials 2 and 3 are sampled. Prompts, answers and tests are not published, so the tasks stay usable.

Recorded fields

Model
unsloth/Qwen3.5-4B
Adapter
none, untuned parent
Loaded in 4-bit
no, 16-bit
Layers offloaded to system RAM
no
Prompt language
en
Trials per task
3
Token limit per answer
1024
Thinking mode
off (enable_thinking=False)
Seed
3407
PyTorch
2.14.1+cu130
Tasks
7
Trials
21
Trials passed
6
Tasks passed in every trial
1
Tasks passed at least once
3
Wall time
531 s
Peak VRAM
8.65 GiB
Times re-graded from stored answers
1
Grading policy
per-block transactions, errored blocks skipped, final state graded (22 Nov 2026)
Hardware
RTX 4070 Ti, 12 GB

Per task

TaskWhat is wrong in the appTrialsPassed
clean-003Announcements: public read of published rows, no client writes at all (control)2 of 3
rls-009Notes: RLS on, no DELETE policy, so users cannot delete their own notes (secure but broken)3 of 3
rls-010Tasks table: the owner policies are granted TO anon instead of authenticated, so signed-in users see nothing0 of 3
rls-011Admin check trusts user_metadata in the JWT, which any user can edit from the client0 of 3
rls-012App config table holds a Stripe secret key and is readable by anonymous visitors1 of 3
rls-013A reporting view over the orders table runs with the owner's rights and bypasses RLS0 of 3
rls-014Org documents: any member can delete every document in the org; only org admins should0 of 3