System-1 decisions without lock-in: Jev vs open-weight Laya

A new kind of model answers software’s small questions directly. Instead of writing text, it takes some state and a typed question (pick one option, give a score, say whether a statement holds) and returns an answer with a probability. TypeSafe calls this a System One model; its hosted model is Jev.

A decision that sits inside your control flow is a dependency. If the only model that can answer it lives behind one vendor’s API and key, the decision is locked in. So we asked a narrow question.

The question

Can an open-weight model answer the same typed decisions on an ordinary laptop, well enough that the hosted model becomes an optional comparison rather than a requirement?

The candidate is Laya, an Apache-2.0 decision model from Convai Innovations: a ModernBERT-large encoder with a small decision head, published on Hugging Face. Its authors report beating Jev on their own benchmark. A comparison site reports the opposite on a different one. Both are claims. This note records what happened on one machine, on questions we wrote ourselves.

The method

• A public corpus of 48 hand-written rows, balanced over the three answer types (choice, score, noul), each with its expected answer written before any model ran. Corpus sha256 6e7c1a4d1a12116ac2b45ccb30ae9fc46d4048dd664debd38bd6c86f0c68e029.
• Laya loads from a pinned Hugging Face revision and runs locally.
• Jev is called only when a key is present in the environment at run time, as a comparison. Without a key the run still completes and Jev is recorded as not run; the experiment’s own test (test_no_key_means_not_run) checks that path.
• Each model answers every row sequentially. We record accuracy against the expected answers, accuracy of the answers given with at least 0.9 confidence, per-call latency and cold-load time.

Models in this recording:

• laya-typed-decisions: https://huggingface.co/convaiinnovations/laya/tree/55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851/typed-decisions, revision 55cf4c4ebb4ebe31b2550e8bdf3bd21b99753851; Apache-2.0; backend mlx 0.32.2 (Device(gpu, 0)), float32.
• jev: https://jev-ai.pro/api/v1/systemone (hosted, TypeSafe-compatible); proprietary, hosted; backend hosted API, requested model jev-1.13.0.

Machine: chip Apple M5 Max · ramGb 128 · os macOS 26.6.1 · arch arm64 · python 3.12.13 · loadAverageAtStart [35.6,47.3,46.7] · loadAverageAtEnd [43.9,47.3,46.8]. Measured 2026-09-24T13:34:28Z.

What would prove it wrong

These criteria were committed at 13:27 UTC on 24 September 2026, before the recorded measurement (measuredAt 13:34 UTC). The experiment was built in parallel branches that were later combined, so the order of commits in the repository history does not show which came first; the commit author dates give the true order. Any one of them failing refutes the claim for this corpus.

• Laya is more than 10 percentage points less accurate than Jev on the same rows.
• Laya’s answers at confidence 0.9 or above are less accurate than its answers overall, so its confidence cannot be used to decide when to escalate.
• The local port disagrees with the model’s reference implementation on any row, or its top probability differs by more than 0.01.
• Laya is slower per call on this machine than the hosted Jev call. This tests the speed claim, not the lock-in claim.

What we measured

Agreement between Laya and Jev: 77.1% of rows (37 of 48 rows). Latency is one call per row, sequential, on a shared machine; Jev’s includes the public internet. Cost for Jev is computed from a listed price, not an invoice; Laya runs locally and has no per-call price (hardware and electricity were not measured).

Against the criteria

• RED laya-typed-decisions: 81.3% against Jev 95.8% on the same rows (14.6 points behind; refuted beyond 10).
• GREEN laya-typed-decisions: answers at confidence ≥ 0.9 were 100.0% correct (2 of 2 such answers; only 2 of 48 answers reached that confidence) against 81.3% overall.
• GREEN laya-typed-decisions: median 9 ms locally against 378 ms for the hosted call, including network.
• GREEN Port parity: 0 decision mismatches against the reference implementation across 48 rows; maximum top-probability delta 0.002097.

On these 48 rows the claim is refuted as stated: 1 of 4 pre-registered checks is RED. The result is kept as recorded, not re-scored.

Notes the run recorded

• Backend: every model component (ModernBERT-large encoder, decision head, scorer, act head) runs in MLX on the Apple GPU in float32; no component falls back to torch-MPS. Tokenization uses the Hugging Face `tokenizers` library (Rust, CPU), which is not a model component.
• coldLoadS = reading and upcasting the pinned safetensors plus the tokenizer, after Python imports and after the one-off download. The first call (38 ms, Metal kernel compilation) is excluded from p50/p90/max, which cover one call per corpus row, batch size 1, sequential.
• costPer1kUsd is null for Laya: it runs locally and no per-call price applies; electricity and hardware were not measured.
• Jev: 48 sequential calls, 15 s timeout, requested jev-1.13.0; served-model labels seen: jev-1.13.0. Latency is wall time from this Mac over the public internet, including failed calls. Accuracy counts an unanswered row as wrong.
• Jev costPer1kUsd = measured mean input tokens per call x $0.242 per 1M input tokens, a vendor price CLAIM (jev-ai.pro/pricing, read 2026-09-24, highest listed annual-plan rate; listed range $0.124-$0.242; output tokens free). It is not an invoice.
• calibratedAccuracy = accuracy of the answers whose confidence is >= 0.9 (confidence = the model's reported value; for Laya noul answers, which carry none, 1 - normalized entropy of [1-p, p]).
• Decisions: choice = top key; score = most probable level; noul = probability >= 0.5.
• Corpus: 48 hand-written public rows authored before any model ran; the expected answers are the author's judgment. A small authored corpus is not a benchmark; it supports mechanism and parity claims, not general accuracy claims.
• Vendor figures (Laya eval/results.md, jev-ai.pro/compare) are CLAIMS and are not reproduced here.
• The Mac was shared with other work during the run (load average [35.6, 47.3, 46.7] at start, [43.9, 47.3, 46.8] at end); latencies are under that load.

What this does not establish

• 48 authored rows are a mechanism check, not a benchmark. They do not predict accuracy on your decisions, in other languages or in other domains.
• One machine, one run. The latencies describe this laptop under its load at the time; they are not a hosted-service comparison at scale.
• Jev’s latency includes the network and whatever the hosted service was doing at the time. It is not Jev’s model speed.
• Accuracy at 0.9 confidence can rest on very few answers. The counts beside each figure are its denominator; a handful of confident answers says little about calibration.
• Nothing here tests fine-tuning, long inputs beyond the model’s context, or label sets larger than the corpus uses.
• Vendor, model-author and comparison-site figures below are claims. We did not reproduce them, and our numbers neither confirm nor refute them.

Provenance

The figures above are rendered at build-authoring time from laya-mlx-2026-09-24.json, copied byte-for-byte from experiments/laya-vs-jev/results/laya-mlx-2026-09-24.json on branch exp/laya-vs-jev of odin-labs-ai/odin-rnd, as committed at revision a5fa7daf3961199125983f03c5b5cb86514bc83a. Its sha256 is 42e0e7bd2ad38dc94e7ffd7dcba992ca6f636f1d990ff6ae5954241d36858e59; the copy manifest records both. A repository test re-renders this page from that file and fails if they differ.

Sources

• TypeSafe AI, “Introducing System One Models & Jev” (15 Sep 2026). vendor claim, accessed 2026-09-24: Jev returns typed, probabilistic decisions instead of text; end-to-end response 70–500 ms; input $0.042 per million tokens, output free.
• Convai Innovations, Laya model card, Hugging Face revision 55cf4c4. model-author claim, accessed 2026-09-24: Apache-2.0; ModernBERT-large backbone plus a decision head, 421M parameters; fine-tuned typed-decisions checkpoint 0.766 accuracy against a published Jev 1.13.0 figure of 0.727; 32.8–39.5 ms per question on a T4 GPU, 193–464 ms on CPU. The card states its Jev figures are third-party published, not measured by the authors.
• Laya eval/results.md, Hugging Face revision 55cf4c4. model-author claim, accessed 2026-09-24: Overall in-task accuracy 0.753 (ECE 0.030); zero-shot accuracy 0.651 (ECE 0.204) on held-out task families.
• Jev AI, “Jev vs Laya” comparison page (independently operated, not TypeSafe). third-party claim, accessed 2026-09-24: JevBench v1.4.0 hard tier: Jev 74.1% against Laya 34.1%; judge tier 94.5% against 69.2%; Jev 0.65 s median on the hosted API. The bare /compare index returned HTTP 404 on the access date.
• Jev AI pricing page (independently operated reseller, not TypeSafe). third-party price claim, accessed 2026-09-24: Annual plans list $0.124–$0.242 per million input tokens, output free. The recorded Jev cost per 1,000 calls multiplies the measured mean input tokens by the highest listed rate; it is not an invoice.
• v-modal, awesome-jev-tools (community list of Jev projects, GitHub). community list, accessed 2026-09-24: Lists Laya under Calibration & Research as an open alternative to Jev, alongside other open replicas and independent Jev evaluations. A list entry is not an endorsement; the list says so itself.
• Sathish Raju, “What Is Jev? A Practical Look at TypeSafe’s System One Model”, Medium, Sep 2026. commentary, accessed 2026-09-24: Background reading only. The page refused automated retrieval (HTTP 403) on the access date, so no figure here relies on it.
• Vishal Mysore, “Jev By TypeSafe: A model you were waiting for!”, Medium, Sep 2026. commentary, accessed 2026-09-24: Background reading only. The page refused automated retrieval (HTTP 403) on the access date, so no figure here relies on it.

Laya’s weights are never committed to this repository; they are fetched at a pinned revision when the experiment runs. No API key, key prefix or response header is recorded. Failures are kept in the record as evidence.