ZerwOne Lab · small language models, English and Arabic

Measuredbeforeclaimed.

We are building a small model that reviews and repairs apps made with AI tools. Nothing is trained yet. Every number on this site opens the run that produced it.

Runs recordedload tests, smoke tests, evals
9
Eval tasks writteneach with a hidden test
17
Models trainedthe smoke-test adapters do not count
0

Last run recorded

First load
LoRA smoke test
Lint baseline
Repair eval
Fig. 0 · every run so far · run 09 of 0922 Nov 2026, 07:21 UTCRepair eval: Qwen3.5-4B, untuned, 7 tasks6 of 21 trials passed

Fig. 1 · the baseline to beat

18/42

The untuned model repairs 43% of vulnerable trials.

14 broken Supabase apps and 3 correct ones. The model gets the schema, proposes a fix, and a hidden test attacks the result. It repaired 5 tasks every time and 8 at least once.

A free linter flags 8 of the 14 and repairs none. Where the linter is blind, the small model mostly is too. That gap is what a trained adapter has to close.

Sourceeval run 1, eval run 2, lint run. Three trials at 17 tasks is a pilot: only a gain of about 30 points would be a result.

TaskWhat is wrong in the appLintModel, 3 trials
rls-001Personal notes table with RLS never enabled
rls-002RLS enabled but the select policy lets everyone read private orders
rls-003Profiles: anyone, including anonymous, can update any profile
rls-004Shared team board: update policy USING (true) lets any member take over or rewrite another member's task
rls-005Comments: insert policy accepts any user_id, so a user can post as someone else
rls-006RLS is correct, but a SECURITY DEFINER function exposed to the API returns every invoice
rls-007Multi-tenant documents: the policy is right but users can insert their own membership into any org
rls-008Public profiles expose email and phone to every signed-in user
rls-009Notes: RLS on, no DELETE policy, so users cannot delete their own notes (secure but broken)
rls-010Tasks table: the owner policies are granted TO anon instead of authenticated, so signed-in users see nothing
rls-011Admin check trusts user_metadata in the JWT, which any user can edit from the client
rls-012App config table holds a Stripe secret key and is readable by anonymous visitors
rls-013A reporting view over the orders table runs with the owner's rights and bypasses RLS
rls-014Org documents: any member can delete every document in the org; only org admins should
Controls: nothing is wrong, and the right answer is to change nothing
clean-001Correct per-user bookmarks table (control: nothing to fix)
clean-002Blog posts readable by everyone on purpose, writable by the author only (control)
clean-003Announcements: public read of published rows, no client writes at all (control)
linter flagged it trial passed the hidden test trial failed

From claim to measurement

01 / 04

Finds the security holes in your app.

Repaired 18 of 42 vulnerable trials. Untuned, 14 tasks, 3 trials each.

Run: repair eval, Qwen3.5-4B

A powerful bilingual model.

Asked for Levantine Arabic, one candidate answered in Egyptian. One prompt, one native judge.

Run: first load, Qwen3.5-4B

Runs on consumer hardware.

Trains at 8.81 GiB peak on one 12 GB RTX 4070 Ti.

Run: LoRA smoke test

Outperforms the base model.

No adapter is trained. Not measured yet.

No run exists for this claim.

Fig. 2 · training runs on one card

Loss falls. That proves very little.

Two 100-step smoke tests on eight placeholder examples. The curves show the pipeline works on a 12 GB card and can memorise eight examples. They say nothing about quality, and the adapters were thrown away.

Line drawing of a desktop graphics card with three fans
IllustrationThe whole lab is one graphics card with 12 GB of memory. Drawn, not photographed.
0.010.1110020406080100training steploss, log scalegemma-4-E4B-it0.025Qwen3.5-4B0.024

step 100gemma-4-E4B-it 0.025Qwen3.5-4B 0.024

Sourcethe trainer’s own log for each run, in the run log. Log scale. Move across the plot to read each logged step.

Two languages

We asked for Levantine. One model answered in Egyptian.

The same question to both candidate parents, in the Levantine dialect. A native speaker judged the answers. One prompt is a sanity check, not an evaluation.

Qwen3.5-4B · Answered in Egyptian Arabic

احكيلي باختصار وبالعامية الشامية: ليش لازم نتحقق من صلاحيات المستخدم بالسيرفر مش بس بالواجهة؟

اللي بيقولك ده إنك لازم تتحقق من الصلاحيات في السيرفر (Backend) مش بس في الواجهة (Frontend)، ده بسبب بسيط جداً:

Raw output, first lines, not corrected.

Gemma 4 E4B · Answered in Levantine, judged natural

احكيلي باختصار وبالعامية الشامية: ليش لازم نتحقق من صلاحيات المستخدم بالسيرفر مش بس بالواجهة؟

يا زلمة، الموضوع بسيط ومركز. تخيّل الموضوع هيك:

Raw output, first lines, not corrected.

Not measured yet

What would change these numbers.

Line drawing of an open lab notebook with a hand-plotted chart, the newest point in violet
IllustrationThe next point goes on the chart when its run exists. Drawn, not photographed.
  • An adapter of our own

    not measured yet

    Training data that passes the data rules. None is collected yet.

  • Gemma 4 E4B and Jais 3 on the same tasks

    not measured yet

    The same harness, same settings.

  • The tasks in Arabic

    not measured yet

    The Arabic prompts are written and wait for a native-speaker review.

  • A stronger model with lint and retrieval

    not measured yet

    The comparator that tells us whether a small model is worth it.

The run log

Do not take our word for any of it.

Read the run log