ZerwOne Lab · small language models, English and Arabic
Measuredbeforeclaimed.
We are building a small model that reviews and repairs apps made with AI tools. Nothing is trained yet. Every number on this site opens the run that produced it.
- Runs recordedload tests, smoke tests, evals
- 9
- Eval tasks writteneach with a hidden test
- 17
- Models trainedthe smoke-test adapters do not count
- 0
Last run recorded
- 22 Nov 2026, 07:21 UTCRepair eval: Qwen3.5-4B, untuned, 7 tasks6 of 21 trials passed
- 22 Nov 2026, 07:13 UTCRepair eval: Qwen3.5-4B, untuned, 10 tasks20 of 30 trials passed
- 22 Nov 2026, 07:02 UTCLint baseline: Supabase Splinter, 17 tasks8 of 14 vulnerable tasks flagged
- 22 Nov 2026, 06:15 UTCLoRA smoke test: Qwen3.5-4B, sequence 2048, padded9.12 GiB peak, 142 s for 30 steps
- 22 Nov 2026, 06:11 UTCLoRA smoke test: Qwen3.5-4B, sequence 20488.81 GiB peak, 49 s for 30 steps
- 22 Nov 2026, 06:07 UTCLoRA smoke test: gemma-4-E4B-it, sequence 5127.09 GiB peak, 231 s for 100 steps
- 22 Nov 2026, 06:02 UTCLoRA smoke test: Qwen3.5-4B, sequence 5128.81 GiB peak, 159 s for 100 steps
- 22 Nov 2026, 05:50 UTCFirst load: gemma-4-E4B-it7.09 GiB peak, 28 s to load
- 22 Nov 2026, 05:12 UTCFirst load: Qwen3.5-4B8.72 GiB peak, 149 s to load
Fig. 1 · the baseline to beat
18/42
The untuned model repairs 43% of vulnerable trials.
14 broken Supabase apps and 3 correct ones. The model gets the schema, proposes a fix, and a hidden test attacks the result. It repaired 5 tasks every time and 8 at least once.
A free linter flags 8 of the 14 and repairs none. Where the linter is blind, the small model mostly is too. That gap is what a trained adapter has to close.
Sourceeval run 1, eval run 2, lint run. Three trials at 17 tasks is a pilot: only a gain of about 30 points would be a result.
| Task | What is wrong in the app | Lint | Model, 3 trials |
|---|---|---|---|
| rls-001 | Personal notes table with RLS never enabled | ||
| rls-002 | RLS enabled but the select policy lets everyone read private orders | ||
| rls-003 | Profiles: anyone, including anonymous, can update any profile | ||
| rls-004 | Shared team board: update policy USING (true) lets any member take over or rewrite another member's task | ||
| rls-005 | Comments: insert policy accepts any user_id, so a user can post as someone else | ||
| rls-006 | RLS is correct, but a SECURITY DEFINER function exposed to the API returns every invoice | ||
| rls-007 | Multi-tenant documents: the policy is right but users can insert their own membership into any org | ||
| rls-008 | Public profiles expose email and phone to every signed-in user | ||
| rls-009 | Notes: RLS on, no DELETE policy, so users cannot delete their own notes (secure but broken) | ||
| rls-010 | Tasks table: the owner policies are granted TO anon instead of authenticated, so signed-in users see nothing | ||
| rls-011 | Admin check trusts user_metadata in the JWT, which any user can edit from the client | ||
| rls-012 | App config table holds a Stripe secret key and is readable by anonymous visitors | ||
| rls-013 | A reporting view over the orders table runs with the owner's rights and bypasses RLS | ||
| rls-014 | Org documents: any member can delete every document in the org; only org admins should | ||
| Controls: nothing is wrong, and the right answer is to change nothing | |||
| clean-001 | Correct per-user bookmarks table (control: nothing to fix) | ||
| clean-002 | Blog posts readable by everyone on purpose, writable by the author only (control) | ||
| clean-003 | Announcements: public read of published rows, no client writes at all (control) | ||
From claim to measurement
01 / 04
Finds the security holes in your app.
Repaired 18 of 42 vulnerable trials. Untuned, 14 tasks, 3 trials each.
A powerful bilingual model.
Asked for Levantine Arabic, one candidate answered in Egyptian. One prompt, one native judge.
Outperforms the base model.
No adapter is trained. Not measured yet.
No run exists for this claim.
Fig. 2 · training runs on one card
Loss falls. That proves very little.
Two 100-step smoke tests on eight placeholder examples. The curves show the pipeline works on a 12 GB card and can memorise eight examples. They say nothing about quality, and the adapters were thrown away.

step 100gemma-4-E4B-it 0.025Qwen3.5-4B 0.024
Sourcethe trainer’s own log for each run, in the run log. Log scale. Move across the plot to read each logged step.
Two languages
We asked for Levantine. One model answered in Egyptian.
The same question to both candidate parents, in the Levantine dialect. A native speaker judged the answers. One prompt is a sanity check, not an evaluation.
Qwen3.5-4B · Answered in Egyptian Arabic
احكيلي باختصار وبالعامية الشامية: ليش لازم نتحقق من صلاحيات المستخدم بالسيرفر مش بس بالواجهة؟
اللي بيقولك ده إنك لازم تتحقق من الصلاحيات في السيرفر (Backend) مش بس في الواجهة (Frontend)، ده بسبب بسيط جداً:Raw output, first lines, not corrected.
Gemma 4 E4B · Answered in Levantine, judged natural
احكيلي باختصار وبالعامية الشامية: ليش لازم نتحقق من صلاحيات المستخدم بالسيرفر مش بس بالواجهة؟
يا زلمة، الموضوع بسيط ومركز. تخيّل الموضوع هيك:Raw output, first lines, not corrected.
Not measured yet
What would change these numbers.

An adapter of our own
not measured yetTraining data that passes the data rules. None is collected yet.
Gemma 4 E4B and Jais 3 on the same tasks
not measured yetThe same harness, same settings.
The tasks in Arabic
not measured yetThe Arabic prompts are written and wait for a native-speaker review.
A stronger model with lint and retrieval
not measured yetThe comparator that tells us whether a small model is worth it.