Rendered at 09:36:52 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
ricardobeat 1 days ago [-]
Everyone is doing this to emulate Jev, but...
I took a random book excerpt with 23,000 words (±30k input tokens) and used it as context. Jev still responds in 800ms, sometimes 500ms. That's in the neighbourhood of 20-50,000 tok/s prefill, which is obviously not possible with normal LLMs, not even Cerebras is this fast.
HarHarVeryFunny 21 hours ago [-]
It seems people are just guessing at the architecture behind Jev. Obviously the functionality itself is easy to replicate, but why Jev seems to be making such a splash (beyond the doh! factor of it's huge applicability) is the ultra-low cost and speed, which may be due to architecture.
The Laya model compared in TFA shows one way Jev may be getting it's speed and low cost - by using a BERT-like bidirectional model rather than an auto-regressive one (LLM).
prometheus1992 1 days ago [-]
was the answer correct?
i have tested jev for my use cases and its horrendously wrong, but then the follow up from jev's team is "oh, you need to boil the question down further". it's a spiral of how much do you wanna dumb down the ask so that it answers it correctly. i'll pass for now.
also, 30k input tokens is a lot.
amluto 1 days ago [-]
I imagine it’s not so hard to optimize a model for this use case.
Off the top of my head, I would skip all the modern linear attention / state space stuff and use classical attention. But run prefill in a fully sliding-window mode so that “state” tokens simply don’t attend to far away tokens, or maybe also allow everything to attend to the first few tokens (and train like this). Now prefill is almost embarrassingly parallel, and you can make it fully parallel by duplicating work at block boundaries. (I’m not saying this is an awesome architecture if you want excellent results, but I’m also not convinced that Jev gives excellent results…)
The let queries attend to everything.
And architect the stack around this. Don’t try to cache the KV data — process the queries as you go so that the each input block and layer’s K and V data is computed, attended to, and discarded.
I’m curious whether Cerebras actually is a good device for this. Cerebras is kind of low on RAM, but if you don’t need to store KV data, maybe the entire computation fits on the die.
reissbaker 12 hours ago [-]
20k tok/sec prefill on B200/B300 isn't particularly noteworthy for medium-sized models like GLM-5.3-Flash, vLLM and SGLang achieve it on a reasonable number of models, especially at NVFP4.
50k tok/sec is pretty impressive though.
But... when you were doing your measurements, were you using the same random book excerpt? If you were potentially getting even partial cache hits for your 50k tok/sec measurement, it would taint the benchmark: pretty much any inference provider running any LLM would be able to hit those numbers.
mmastrac 1 days ago [-]
That's not true. I ran Cerebras as an experimental ultrafast Jev and it was faster.
ericpauley 1 days ago [-]
This has “/dev/null as a service” vibes…
walrus01 1 days ago [-]
magic 8 ball as a service.
nacs 21 hours ago [-]
Is the Magic 8 ball immune to hallucination as Jev is?
IshKebab 21 hours ago [-]
Yes. It's exactly as immune to hallucination - it can only produce answers from a small fixed set.
manojlds 22 hours ago [-]
Do we have a reliable way to count tokens for Jev yet btw?
stingraycharles 1 days ago [-]
Also, it processes all questions you ask it in parallel, which is also not possible with normal LLMs.
Xorlev 1 days ago [-]
Sure it is.
The prefill is the only blocking part, and you can prefill the whole context up to the point where they diverge, then prefill each question and decode the one token in parallel for each question.
If you batch vLLM calls with the same prompt prefix to the same process, it'll deduplicate the prompt prefix across batched requests (+/- the block size) and decode in parallel for each.
That's with a vanilla LLM. If you modify the LLM you can pull that in-graph, but it isn't really necessary.
vezycash 2 hours ago [-]
Qwen3-Next-80B-A3B can already run on a 16GB M1 MacBook at around 3–5 tok/s using aggressive memory management. Could a Jev-style controller push that to 100 tok/s on an M1?
wongarsu 20 hours ago [-]
I have run tests with qwen 3.8 and gemma 4 in a way similar to this post (based on an open source project that also does this with gemma4).
Getting competitive accuracy with Jev is fairly easy, if by accuracy you mean that the highest weighted answer is the right one. GLM 5.3 is complete overkill, much smaller llms will do
What Jev brings to the table, beyond speed, is that the reported probabilities match actual likelyhoods. If you present three options, with A and B equally likely and C impossible, jev will approximately answer with 0.5, 0.5, 0. Stock LLMs don't
walrus01 1 days ago [-]
You can turn any sufficiently smart LLM into yes/no decision model or equivalent. I already have an existing workflow with a two paragraph detailed prompt, that sends pages of stuff to an LLM and asks it to return only 7 JSON objects. Several of those objects are binary "yes or no" choices of like, whether the content contains certain things.
You can even do it with small not particularly hard to host local LLMs like a variant of Qwen 3.6 35B A3B or 3.8 27B.
opiotrek 1 days ago [-]
But does it always 100% of the time sticks to the schema? We have a prompt that is explicitly instructed to return a single html tag with the response inside it and it sometimes hallucinates
gf000 1 days ago [-]
I'm fairly sure it is 100%, and has been available for "ages". That's how every tool call and whatnot works:
I think the truth is sort of halfway between yall.
Hallucinations in tool call results _do still exist_, but basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
That all goes out the window when a string field has a “hidden” schema in that only particular strings are valid, but that restriction isn’t in the JSON schema. I have had failures when I want a field to be specifically formatted Markdown or something.
We’ll probably see something that handles this better in the future like jsonnet or Cue or dhall that has some execution capabilities so you can write a custom validator beyond what JSON schema supports.
gf000 14 hours ago [-]
> and re-prompt the LLM until it emits something with a valid schema.
Well, that's essential what happens, but on the "backend" side still, so it's significantly more efficient. Also, model providers can play with e.g. how likely are they to accept a valid/invalid character, so for example the first token may only be [, { or ".
As for what's not in the schema, that's an orthogonal question.
hbrn 14 hours ago [-]
> basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
Basically no one is doing that since 2024, have you been living under the rock? Read about constrained decoding.
> I have had failures when I want a field to be specifically formatted Markdown or something.
And Jev has an advantage here because it can’t generate anything?
22 hours ago [-]
walrus01 1 days ago [-]
yes, it does, with appropriate tuning/testing of the llm's parameters (temperature, top p, top k, using the right model, and the right prompt). You have to give it a rigid and very specific prompt to only answer in the JSON form. Also test it with LLMs that will handle being given a very low temperature to be very 'literal'. You're not asking for creative writing.
I should also add that the source comes from one of about 400 possible places and in a variety of messed up formats, it's the raw feed from a news scraper...
erkl 1 days ago [-]
I think you're operating with a different definition of 100%.
walrus01 1 days ago [-]
It hasn't failed once since the single hour I spent writing the prompt and tuning it? Close enough to 100% for my purposes.
krial 1 days ago [-]
Yes, but Jev & co literally cannot fail that way. It's 100%. Not the same thing.
walrus01 1 days ago [-]
Cannot fail? It's so smart it literally can't ever parse incoming content incorrectly and provide the wrong answer to a question like "Is at least 25% of the text content in this document written in German? (yes/no)"
david_draco 22 hours ago [-]
You are talking about the content of the response, while krial is talking about deviations in the structure of the response. The point is the number of possible states an output can take, so the next step can ingest it and it can be part of a robust pipeline.
StevenWaterman 14 hours ago [-]
Yes, you can use constrained decoding
m4y0u 2 days ago [-]
My question is why not use Jev instead? It's faster and cheaper.
kouteiheika 1 days ago [-]
> My question is why not use Jev instead? It's faster and cheaper.
Because it's proprietary? By using an open weight model you're guaranteed that you can access it forever; if one provider bans you then you can go to another one (or you can self-host). With a proprietary, single-provider model locked behind an API if your access is revoked you're screwed.
kylecazar 1 days ago [-]
There's some speculation that Jev is an open weight model with novel post-training (RLCD). So, if these folks have competitive accuracy with just the base model, it may raise some questions about the necessity of Jev's architecture. You generally don't want to find yourself competing only on price.
Fyi, I haven't tested this yet.
Fordec 1 days ago [-]
Also, while it's clearly got a lot of training on some use cases, others that probably weren't in the training set have worse good decision rates than a random number generator. If you can rebuild the architecture, you can train it on your use case.
HarHarVeryFunny 19 hours ago [-]
Whether Jev is something architecturally different from an LLM (bidirectional vs auto-regressive? different type of parallel readout head?) remains to be seen, or guessed, but LLMs are certainly fungible and are competing on price - they leapfrog each other from release to release, but overall they are all progressing in unison.
When it comes to the high volume market of business automation, it seems that ultra-low cost, rather than expensive frontier intelligence, is exactly what you want, and low latency is also nice to have for customer-facing applications like customer service chatbots.
janalsncm 1 days ago [-]
> it may raise some questions about the necessity of Jev's architecture
When I hear “architecture” I am thinking number of parameters and latency.
When I hear “accuracy” I think training recipe, data, and (later) number of parameters.
So when you say that Jev’s architecture may not be necessary, the evidence I expect to see is comparable quality at comparable latency. Not equal quality at 2x latency and 4x the cost.
kylecazar 22 hours ago [-]
The post claims comparable quality at comparable latency (the difference in the latter only stemming from serving region).
Hence my note about the risk of cost being the only moat.
dcss_gardener 1 days ago [-]
I mean just from what's known of the funding and timeline it pretty much has to be based on open weights.
But it is likely more than just a fine tune + novel training. At the very least the LM head is swapped out for a classifier one and then or also idk, bidirectional attention for the encoding pass I'm out of my depth at this point and will stop guessing. The training is probably where they have the biggest moat though, not that it's necessarily huge.
I have a project that fits jev as advertised almost comically well and I've been playing with it, and the various hacks and open versions. Jev doesn't necessarily perform better overall but it is quite different. It's sensitive to prompt phrasing in ways the others aren't, it's easy to generate questions where all the other models cluster in confidence but jev is an outlier. Not necessarily more correct, but it does feel like it's getting its answers in a different way.
I'm guessing just as much as anyone else but I've been spending a ton of time on this the last couple weeks, it landed right when I was most ready to dig into it.
ComputerGuru 1 days ago [-]
Your last bit about it being different is a known issue with models that have been trained on purely synthetic data, no?
dcss_gardener 15 hours ago [-]
Sure but I still don’t see any reason to think it’s not based on an open weight model. They don’t even make that claim as far as I know.
andrewchambers 1 days ago [-]
These questions are answered by the OP (Same speed, image support) - additionally, GLM is open weight.
schainks 12 hours ago [-]
Compliance. You can’t put Jev in a HIPAA compliant service, for example.
prjkt 1 days ago [-]
how is Jev cheaper if I can run locally. 0.5% prefill, 0.1% decode, 99.4% cached, latency is <20ms
ttoinou 1 days ago [-]
Isnt this obvious ? I would have thought people would try such things before deciding they need something like Jev
petesergeant 1 days ago [-]
It is obvious. I was going to build my own little toy doing just that ten days ago, and then found four pre-existing projects, so wrote those up instead: https://sgnt.ai/p/jev/
e12e 23 hours ago [-]
Great article, thanks for posting.
I think I'm gradually getting a picture of what jev does, and why it might be good:
1) one shot classifiers are great, and are well-known.
2) LLMs are also great classifiers, but are slow at output - and not great at guaranteed structured output.
3) everyone focused on LLMs, and forgot about classifiers.
4) any decent one shot classifier should be easy to run "fast"/"in realtime".
5) combining 4) with LLMs and a harness (a loop) open up some clever use-cases that LLMs alone are too slow for.
As I understand it, jev isn't long for this world (we'll have foss one shot classifiers we can run on modest hardware) - but it seems likely the legacy will be that classifiers will re-gain some prominence alongside generative models.
octoberfranklin 1 days ago [-]
It is.
What isn't obvious is why people keep shouting "Jev Jev Jev" all the time.
Astroturf.
goodmythical 1 days ago [-]
I am just this far from adding Jev posts to my blocklist.
I cannot fathom how people are not seeing the multiple daily posts as anything but the spam they are.
OgAstorga 1 days ago [-]
don’t worry once they get acquired for 30 billion usd then the spam will stop.
profstasiak 19 hours ago [-]
Not for people with llm brain antrophy
k__ 19 hours ago [-]
Is it non-autoregressive?
ArtRichards 1 days ago [-]
I personally love Privatemode's approach. Having Jev-like speed for confidential ai use cases is a huge enabler.
janalsncm 1 days ago [-]
If you are using an autoregressive decoder (which glm is) it is not “jev-like”. You lose all of the speed advantages that Jev has.
yogthos 1 days ago [-]
> We measured latency in separate runs with one request at a time, because timings taken under load measure the queue rather than the model.
> As Privatemode is hosted in the EU and Jev is hosted in the US, we ran four of the datasets from Germany and from the US at the same time. From Germany, Privatemode answered in 180 ms and Jev in 264 ms. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode.
Right, so it is double the latency and will no longer feel real-time to the end user.
oefrha 1 days ago [-]
Not reading TFA before commenting is okayish I guess. Confidently doubling down with a direct contraction to a short and clear quote in a reply is just polluting the discussion with noise.
Jabrov 1 days ago [-]
Is this a joke? “Jev-like” properties? People have been using LLMs as classifiers or rankers in a similar way for ages. I feel like we’re losing our minds
nazgul17 1 days ago [-]
Jev's particular architecture is such that context can be filled faster than with LLMs.
I took a random book excerpt with 23,000 words (±30k input tokens) and used it as context. Jev still responds in 800ms, sometimes 500ms. That's in the neighbourhood of 20-50,000 tok/s prefill, which is obviously not possible with normal LLMs, not even Cerebras is this fast.
The Laya model compared in TFA shows one way Jev may be getting it's speed and low cost - by using a BERT-like bidirectional model rather than an auto-regressive one (LLM).
i have tested jev for my use cases and its horrendously wrong, but then the follow up from jev's team is "oh, you need to boil the question down further". it's a spiral of how much do you wanna dumb down the ask so that it answers it correctly. i'll pass for now.
also, 30k input tokens is a lot.
Off the top of my head, I would skip all the modern linear attention / state space stuff and use classical attention. But run prefill in a fully sliding-window mode so that “state” tokens simply don’t attend to far away tokens, or maybe also allow everything to attend to the first few tokens (and train like this). Now prefill is almost embarrassingly parallel, and you can make it fully parallel by duplicating work at block boundaries. (I’m not saying this is an awesome architecture if you want excellent results, but I’m also not convinced that Jev gives excellent results…)
The let queries attend to everything.
And architect the stack around this. Don’t try to cache the KV data — process the queries as you go so that the each input block and layer’s K and V data is computed, attended to, and discarded.
I’m curious whether Cerebras actually is a good device for this. Cerebras is kind of low on RAM, but if you don’t need to store KV data, maybe the entire computation fits on the die.
50k tok/sec is pretty impressive though.
But... when you were doing your measurements, were you using the same random book excerpt? If you were potentially getting even partial cache hits for your 50k tok/sec measurement, it would taint the benchmark: pretty much any inference provider running any LLM would be able to hit those numbers.
The prefill is the only blocking part, and you can prefill the whole context up to the point where they diverge, then prefill each question and decode the one token in parallel for each question.
If you batch vLLM calls with the same prompt prefix to the same process, it'll deduplicate the prompt prefix across batched requests (+/- the block size) and decode in parallel for each.
That's with a vanilla LLM. If you modify the LLM you can pull that in-graph, but it isn't really necessary.
Getting competitive accuracy with Jev is fairly easy, if by accuracy you mean that the highest weighted answer is the right one. GLM 5.3 is complete overkill, much smaller llms will do
What Jev brings to the table, beyond speed, is that the reported probabilities match actual likelyhoods. If you present three options, with A and B equally likely and C impossible, jev will approximately answer with 0.5, 0.5, 0. Stock LLMs don't
You can even do it with small not particularly hard to host local LLMs like a variant of Qwen 3.6 35B A3B or 3.8 27B.
https://developers.openai.com/api/docs/guides/structured-out...
Hallucinations in tool call results _do still exist_, but basically everyone asks for a JSON schema for the tool call and uses that to validate and re-prompt the LLM until it emits something with a valid schema.
That all goes out the window when a string field has a “hidden” schema in that only particular strings are valid, but that restriction isn’t in the JSON schema. I have had failures when I want a field to be specifically formatted Markdown or something.
We’ll probably see something that handles this better in the future like jsonnet or Cue or dhall that has some execution capabilities so you can write a custom validator beyond what JSON schema supports.
Well, that's essential what happens, but on the "backend" side still, so it's significantly more efficient. Also, model providers can play with e.g. how likely are they to accept a valid/invalid character, so for example the first token may only be [, { or ".
As for what's not in the schema, that's an orthogonal question.
Basically no one is doing that since 2024, have you been living under the rock? Read about constrained decoding.
> I have had failures when I want a field to be specifically formatted Markdown or something.
And Jev has an advantage here because it can’t generate anything?
I should also add that the source comes from one of about 400 possible places and in a variety of messed up formats, it's the raw feed from a news scraper...
Because it's proprietary? By using an open weight model you're guaranteed that you can access it forever; if one provider bans you then you can go to another one (or you can self-host). With a proprietary, single-provider model locked behind an API if your access is revoked you're screwed.
Fyi, I haven't tested this yet.
When it comes to the high volume market of business automation, it seems that ultra-low cost, rather than expensive frontier intelligence, is exactly what you want, and low latency is also nice to have for customer-facing applications like customer service chatbots.
When I hear “architecture” I am thinking number of parameters and latency.
When I hear “accuracy” I think training recipe, data, and (later) number of parameters.
So when you say that Jev’s architecture may not be necessary, the evidence I expect to see is comparable quality at comparable latency. Not equal quality at 2x latency and 4x the cost.
Hence my note about the risk of cost being the only moat.
But it is likely more than just a fine tune + novel training. At the very least the LM head is swapped out for a classifier one and then or also idk, bidirectional attention for the encoding pass I'm out of my depth at this point and will stop guessing. The training is probably where they have the biggest moat though, not that it's necessarily huge.
I have a project that fits jev as advertised almost comically well and I've been playing with it, and the various hacks and open versions. Jev doesn't necessarily perform better overall but it is quite different. It's sensitive to prompt phrasing in ways the others aren't, it's easy to generate questions where all the other models cluster in confidence but jev is an outlier. Not necessarily more correct, but it does feel like it's getting its answers in a different way.
I'm guessing just as much as anyone else but I've been spending a ton of time on this the last couple weeks, it landed right when I was most ready to dig into it.
I think I'm gradually getting a picture of what jev does, and why it might be good:
1) one shot classifiers are great, and are well-known.
2) LLMs are also great classifiers, but are slow at output - and not great at guaranteed structured output.
3) everyone focused on LLMs, and forgot about classifiers.
4) any decent one shot classifier should be easy to run "fast"/"in realtime".
5) combining 4) with LLMs and a harness (a loop) open up some clever use-cases that LLMs alone are too slow for.
As I understand it, jev isn't long for this world (we'll have foss one shot classifiers we can run on modest hardware) - but it seems likely the legacy will be that classifiers will re-gain some prominence alongside generative models.
What isn't obvious is why people keep shouting "Jev Jev Jev" all the time.
Astroturf.
I cannot fathom how people are not seeing the multiple daily posts as anything but the spam they are.
> As Privatemode is hosted in the EU and Jev is hosted in the US, we ran four of the datasets from Germany and from the US at the same time. From Germany, Privatemode answered in 180 ms and Jev in 264 ms. From the US, the order reverses: 164 ms for Jev against 299 ms for Privatemode.
turns out there is a trick to keeping the context filled and only evaluating a handful of choice tokens https://www.youtube.com/watch?v=bcGO7xre46o