Rendered at 16:22:27 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
lhl 9 hours ago [-]
The old DeepSeek V4.1 Flash has been my workhorse model for the past month (after running tests comparing w/ Qwen 3.8 Flash Next, GLM 5.3 Flash, MiMo 2.6 Flash (including the MOPD fix)) and I've run tens of billions of tokens through it and pushed it fairly hard in both private internal benchmarking and in my own workloads to figure out where it's strong/weak.
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
kbr- 8 hours ago [-]
200+ tok/s on local infra? That's crazy. How much did you pay for your hardware?
crossroadsguy 8 hours ago [-]
I find these local infra costs crazy heavy duty. I always assume these folks are getting their values worth or that infra is serving multiple people's requests at and justifying the cost. Like I once thought of buying a 48-64 GB RAM laptop (not even clsoe to what these models need) and I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access. So I assume the usage is also heavy duty, or if nothing then for privacy and safety and then I guess the price, if one can pay, will totally be worth it.
JKCalhoun 3 hours ago [-]
Looking ahead, do we think token costs will remain where they are today? Go down? Up?
Your investment in hardware is of course fixed, the token cost amortized over time.
I suppose the other question is, do we think down the road that the models capable of being run locally will improve?
scld 3 hours ago [-]
Everything's been trending down so far, so until evidence shows otherwise, I'd assume it will continue that trend. IMO the best time to buy local infra will be when that trend begins to asymptote.
dgellow 2 hours ago [-]
There is so much down pressure, I cannot imagine the token/s price going up. It has been going down and will continue doing so, we are far from a plateau IMHO
mrheosuper 8 hours ago [-]
> I realised the equivalent cost will serve me for years of ≈ $30 per month API/sub access
The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.
Aurornis 3 hours ago [-]
API access does not have a 5-hour or weekly limit.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
amunozo 6 hours ago [-]
I still think only privacy is relevant to run on your own infra, never cost.
vizzah 4 hours ago [-]
near.ai tries to solve this
crossroadsguy 6 hours ago [-]
That is literally my next sentence :D
david-gpu 6 hours ago [-]
> The advantage is no more 5-hours or weekly session limit anymore
That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive.
lnenad 5 hours ago [-]
If the only variable is money, local models will never make any sense at these prices of infra/subscriptions.
For a lot of people, there are other variables though.
origini 5 hours ago [-]
You just need to imagine someone making big bucks in tech + being a little bit money ignorant, having a big house and a nice basement and cheap energy prices.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
johnbellone 4 hours ago [-]
Unless you have solar and batteries there aren't cheap energy prices :).
origini 3 hours ago [-]
I'm german, i pay $45 US Cents / 40 cents in Euro.
Sooooo :D
Valord 2 hours ago [-]
talking about "my setup" should require the good-old-days post footer that includes hardware details.
swiftcoder 8 hours ago [-]
This has to be a distilled model? I don't see any route to 200 tokens/s without serious datacenter hardware
naraic0o 6 hours ago [-]
you mean quantized? it’s only got 16B active parameters so it can be pretty fast at decode.
swiftcoder 6 hours ago [-]
sorry, yes, quantised
MayeulC 8 hours ago [-]
CoT = Chain of Thought (I had to look it up)
Thanks for the insights. If you edit your post to include the definition, I'll delete mine.
dgellow 7 hours ago [-]
> because I'm guaranteed it's not being changed on me
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
Tiberium 11 hours ago [-]
Somehow GPT 6.1 Sol (medium) is cheaper per task ($0.21 vs $0.26) and faster per task (2.9 vs 4.9 minutes) vs v4.1 Flash 1017 (max). I hope AA adds other reasoning effort results of Deepseek so it'd be more realistic to compare it with other models, because IMO using models at max effort is very unrealistic for most people, they then take a huge amount of time and tokens for a small performance uplift.
kadoban 11 hours ago [-]
I wonder what AA's funding and affiliations are. I get the sense that they have their hand on the scale at least a bit.
lifeisloving 9 hours ago [-]
Well yes, thats the entire business model of benchmarking companies. If you pay attention to AA, some models are on the leaderboard second they're released while others come days later, which to me implies that theres some level of coordination happening with them and a few labs.
That kind of coordination doesn't bother me, personally. That's likely just early release/access to reviewers, which isn't inherently problematic.
The part that looks weird to me is some models get tested all up and down their effort level, but this one seems to just have been tested at the level where it will come out worst (I'd expect). Maybe it'll get more coverage over time?
I've also noticed the numbers (and relative rankings) change right after some model releases, which felt a little like someone didn't like who came out on top so they tweaked it.
harrygeez 1 hours ago [-]
I can believe it. Using DS4.1 to decomp old games, it is quite capable of doing so, but the amount of mistakes it makes, by the time I get it right Sol 6.1 would have long ported it correctly more thoroughly. For some tasks Sol really shows its superiority
oefrha 11 hours ago [-]
I don’t know about 1017 but according to my limited testing of 0910 and previous v4-flash-0731 on some of my use cases, max simply used a lot more tokens (almost twice as many IIRC) for little to no benefit over high. So yeah I really don’t see a reason to use max.
amunozo 6 hours ago [-]
And not only your testing, I have seen that claim a lot of time online. I never tested it, though.
andai 11 hours ago [-]
Yeah, this always bugged me. Then again, unlike American models, surely Chinese models are far too expensive for them to test them at every difficulty ;)
lhl 9 hours ago [-]
For DS41F (both the original and 1017) I've found low (25/100) reasoning to be the optimal for my tasks at least (max doesn't tend to better, and if you can't do it w/ low imo you should just switch to a smarter model). At low it's about 1/3-1/2 max's token count (and price and speed).
crossroadsguy 9 hours ago [-]
A bit like influencers of the benchmarking world. Milking it while all this trends. Are there financial/tie-up disclosures that are audited for benchmarks?
kristopolous 11 hours ago [-]
AA prices are always kind of hand wavy for open models. I mean if you rent a cheap gpu and run inference yourself...or heck, what about a cpu instance with a bunch of ram, you can do that with llama.cpp for under a $1/hour. Ok not that, so are we talking about the median spot price per token from a basket of inference providers? Which ones? They actually vary quite a bit, even on openrouter, for this particular deepseek checkpoint it's between $0.15 and $1.20 ... that's like 8x spread...
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
RussianCow 11 hours ago [-]
> It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
Almost no one is running DeepSeek V4.1 Flash on their laptop, so comparing based on API pricing seems perfectly reasonable to me.
this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.
Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.
I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest
perplexity has a 27b decision
then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.
Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...
Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.
This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.
So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.
I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.
mrkn1 57 minutes ago [-]
[flagged]
Tiberium 11 hours ago [-]
I was quite sure they use the API pricing (by default) for open weight models that have official providers? Deepseek does have their own API.
throwawaythekey 11 hours ago [-]
I think they need another version of the charts with "subscription prices" at least. Lots of people are not paying API rates.
hamzaalibasra 3 hours ago [-]
[flagged]
slopinthebag 9 hours ago [-]
this matches my experience tbh. 6.1 sol on medium is quite good at getting things right the first time without overthinking and i prefer it over xhigh or max usually. so i'm not surprised it ends up cheaper.
nicolamanzini 54 minutes ago [-]
DeepSeek V4.1 Flash 1017 Max is live on threejseval with 12 of its 16 Three.js scenes up so far.
Same prompts as the rest of the field, blind human votes, ranked by quality and cost. It finished the 12 scenes at about $0.13 a scene. The other 4 scenes are still pending.
What I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
MaxikCZ 11 hours ago [-]
"Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context
Iolaum 10 hours ago [-]
Indeed, I m surprised how little attention qwen-3.8.-flash-next seems to be getting while it's so well fitted for 128gb unified machines.
72deluxe 6 hours ago [-]
I have a 16GB 5060 and 32GB system. RAM - are there any recommended Qwen 3.8 Flash models worth running on this ? I am using Nemotron 30B A3B at the moment
drwiggly 1 hours ago [-]
I ran the Strata project with this on 32gb ram 16gb AMD 7900XTX. It goes about 20tok/s off the ssd. I'm guessing 64gb would go to 40tok/s. https://github.com/Niko1221/Strata Quality wise its not fast enough for me to eval. I'll probably spend some tokens on it through openrouter. DeepSeek 4 flash though is a mess compared to GPT Sol 6.1.
user_7832 5 hours ago [-]
Fwiw I think you can run 3.8 flash (next?) on your setup with SSD streaming.
There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).
32 gb of ram is gonna be your blocker for iq3, u will have to go iq2 and thus the output will suffer. At the plus side, it will spew faster than iq3.
Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.
davedx 11 hours ago [-]
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
XMPPwocky 10 hours ago [-]
expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it.
akmarinov 11 hours ago [-]
You sure can, https://github.com/lxe/dsx gets you this deepseek on 2x3090s and 128 gb ram at about 30 t/s
At around 75% of the full model intelligence
veber-alex 10 hours ago [-]
About half the t/s of an online provider, heat, noise, power consumption and on top of that you only get 75% of the intelligence? Sounds like a waste of time to me.
akmarinov 9 hours ago [-]
Half the token/s - sure, no way around beating big ass data centers.
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
origini 5 hours ago [-]
I have not capped my 2x4090s so my room gets cosy warm.
But here my price per kwh is 40 cents (euro).
But if you integrated it with solar system and a heat exchange, this will be feasable
nirvdrum 10 hours ago [-]
As luck would have it, you can just pay the provider and use their service. Others may want to run locally for privacy reasons or just for the fun of it. I think at-home AI is incredibly awesome and I applaud the enthusiasts pushing it forward. Just because that's where it's at today doesn't mean it's the best we can do.
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
egeozcan 9 hours ago [-]
I use only providers because I don't have the hardware, but if, and that's a big if, the only moat left for humans is our ability to steer and make bigger picture decisions with our personality, I think it makes sense not to want to send all that conversation data to a 3rd party.
Strata is intriguing and it definitely looks like there's something useful happening there with weight streaming at usable speeds, but I've not managed to have it serve a full answer without inference crashing before long. Pity.
MaxikCZ 7 hours ago [-]
I dont think its strata problem, its been running on my system for days without any crash.
system2 10 hours ago [-]
Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game...
szundi 11 hours ago [-]
[dead]
techjamie 1 hours ago [-]
I was doing some fairly development on Flash 1017 yesterday for around ~3 hours on pi, and all told that session ran about $0.40. I probably could've saved myself more if I'd compacted the context more frequently, since at one point I was running 300k tok context requests.
But it accomplished everything I was looking for it to do, even if Claude perhaps could've done it with less guidance on my part/faster. I'm pretty impressed with it's performance.
twelvechairs 11 hours ago [-]
Mimo 2.6 pro scores 46 and is half the price (though a lot less throughput). Im not sure why nobody has made the same fuss around it
ludvigk 11 hours ago [-]
Hmm if you just compare raw token prices on Openrouter right now, Mimo 2.6 Pro is roughly 4x the price.
Does deepseek v4.1 flash 1017 really take that much longer per task?
celrod 10 hours ago [-]
For some reason the providers for MiMo 2.6 Pro are all very slow. I'd much rather use something quick. DeepSeek was great for that.
I haven't tried MiMo ultraspeed.
scotty79 5 hours ago [-]
Mimo from Xiaomi has safety refusals on things compltely unrelated to safety. It's good when it works but you need another model to continue the work when Mimo craps out.
pippo_tencent 8 hours ago [-]
[dead]
amelius 8 hours ago [-]
Just want to note that this model still does not work (read: sucks) for recognizing sheet music.
The only model that works for me is Astra, but it's expensive.
ryzvonusef 6 hours ago [-]
does the exact model matter anymore? pick one and develop context with it, imho, and just get to work.
then again, I'm not a heavy AI user, so maybe I'm missing something.
_aavaa_ 3 hours ago [-]
It does. Even looking at cases where a both a better and a worse model accomplish a given goal the difference between a solution and a good solution is important. And will save you time by not having to iterate as much with it.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
Imustaskforhelp 11 hours ago [-]
Really wild that Deepseek's model is 536+16B (passive+active) parameter model and its really comparable to Mimo v2.6 Pro which is the current open source SOTA model which is at (978+42)
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
serf 11 hours ago [-]
high tps and braindead model is a dangerous and fun thing to watch.
give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.
clhodapp 11 hours ago [-]
At risk of being blunt: Why the surprise that a model that activates 12B parameters is faster than one that activates 16B parameters which is faster than one that activates 42B parameters?
It feels like that would be the expected result, all else equal.
Imustaskforhelp 11 hours ago [-]
I think it's because of the fact that Deepseek at 16B can be comparable enough in tasks to Mimo with 42B
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
I guess what I'm really saying is just that being impressed with a model's size and being impressed with a model's speed are roughly the same thing for similar architectures and hardware.
Imustaskforhelp 9 hours ago [-]
Yes you are correct about it, though aside from a minor differences within the architecture itself, its right that being impressed with token speed is being impressed with model's size.
So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D
spwa4 7 hours ago [-]
It's too bad they couldn't make it smaller, but still very nice improvement.
The new model:
DeepSeek V4.1 Flash 1017 (Max) 552B A16B
Matches
GLM 5.3 753B A40B
MiMo 1T A42B
Nicely done!
colingauvin 2 hours ago [-]
I'm running it at 3 b.p.w. and it really feels better than Flash 4 at 4 b.p.w. for me. Some of these EXL3 calibrated quants are like magic.
It also has 196B ngram params that sit on disk. So it's really pretty comparable to GLM 5.3 in size.
Partly because I run it on very fast local infra (200+ tok/s) and partly because I'm guaranteed it's not being changed on me, I've generally preferred it over proprietary subscription models for daily tasks, as it seems about as capable at mid-level tasks (I reserve Astra/Opus mostly for planning/review).
I've replaced the previous model with 1017, and while it's early, a few observations:
- It's CoT has changed dramatically. I find the heavy Astra distillation claims to be plausible
- It has strong benchmarks scores, and from what I've seen online it's much better at 3D games and such, but that's not so relevant for my work. On my internal evals it is more token efficient, but it actually doesn't score any higher than the previous model
- It's only been a couple days, but it does seem a bit more focused and reliable. DS4 0731 and DS4.1 have been a bit sloppy (as in not-careful, although they did tend to fall into Claudisms as well), and 1017 seems to be a bit sharper.
- I'm running it through some more challenging work, but I'm not sure if it's really at GLM-5.3 level, especially for cyber/RE/hardware poking, which GLM has excelled at.
Your investment in hardware is of course fixed, the token cost amortized over time.
I suppose the other question is, do we think down the road that the models capable of being run locally will improve?
The advantage is no more 5-hours or weekly session limit anymore. Also no chance your proprietary code ends up in someone repo.
I run local models, too, but every discussion about these models uses the local quantized versions and the hosted versions interchangeably. They’re not interchangeable. The quantized versions you can run on mid-grade consumer hardware are a step down in quality. That might be fine or might not depending on your use case but it needs to be acknowledged. It’s not until you get into really expensive hardware setups that you can run something like DSv4 Flash at Q8, and you’re not breaking even on that relative to a cloud provider.
Keeping everything local is about the only reason to run local. The other discussions about saving money are missing something.
That is not a concern with pay-as-you-go models like DeepSeek. It is stupid inexpensive.
For a lot of people, there are other variables though.
Than it just makes sense that they already do this.
But to be fair, if i move, enough space for battery, solar and a basement which can take proper ML hardware + heat exchanger, is in my future vision.
Sooooo :D
Thanks for the insights. If you edit your post to include the definition, I'll delete mine.
This is something underrated. With an open weight model I can evaluate the performance for my use case, and know for a fact the model will stay the same over time. No arbitrary “model welfare” rules being introduced later on, no undocumented change to capabilities or performance
https://fixupx.com/aarnogau/status/2076059071246193012
The part that looks weird to me is some models get tested all up and down their effort level, but this one seems to just have been tested at the level where it will come out worst (I'd expect). Maybe it'll get more coverage over time?
I've also noticed the numbers (and relative rankings) change right after some model releases, which felt a little like someone didn't like who came out on top so they tweaked it.
They have some math here https://artificialanalysis.ai/methodology but ok how are the prices weighted among the providers... and even if you do that 7:2:1 math it doesn't match the providers. Specifically the model in question: https://artificialanalysis.ai/models/deepseek-v4-1-flash-101... doesn't match the first listed provider https://deepinfra.com/deepseek-ai/DeepSeek-V4.1-Flash-1017
Do the math yourself, it's actually $0.08 not the AA claimed $0.10 (and that's the more expensive of the two pricing tiers). That's a pretty large error bar.
Also many of the providers are now doing "peak" and "offpeak" pricing (deepseek.com itself for instance) so it can differ dramatically. These "peak" times depend on where the service is most popular. You can basically ride "off-peak" pricing by cascading from global provider to global provider.
The majority of the time taken in an inference call is doing the inference so the extra latency of traveling across an ocean is actually a fairly small wall-clock tax here. People still seem to be partial to nearby servers for ai-related workloads but really, try some decent ones 10,000 km away, you probably won't notice.
I know AA thinks highly of the price metric (I've been to their events, even have their hat, they brag about it) but I have learned to ignore it as noise.
It's an especially strange metric with open-weights decision models. I mean I can run those on my laptop ... cost is very abstract there.
But for closed-source models with only a handful of providers (sometimes just 1) great, the price is a very reasonable number.
For the open models a more useful thing would be like "100 tok/s spot price build" and then it's using pcpartpicker or something in some kind of partnership and can query a build for buying a machine right now that can achieve a certain benchmark number and maybe something for less interactive workloads like "5 tok/s". Of course getting some base unit of power cost in there might be important as well since the power bill is the majority of what you pay for when you rent a commodity gpu.
Almost no one is running DeepSeek V4.1 Flash on their laptop, so comparing based on API pricing seems perfectly reasonable to me.
this one is especially annoying ... i can run these on lots of hardware but there's vast differences in many aspects.
Whether I'm getting a good deal isn't what I'm thinking about if I'm getting garbage results. It's more about why am I wasting my time on the wrong tool.
I mean what I really want is a more reliable classic eval benchmark for the decision models that I can easily deploy, like the official ollama ones: https://ollama.com/library?sort=newest
perplexity has a 27b decision then there's a 3b from liquidai and another 600M one. Then there's one called kev and bespoke has nimble-v3. clef from cloudflare is there along with strands-decider from aws. Then there's laya and tev1 ... I mean the AA page seems to be the only real third party doing benchmark and evals on it but most of these are missing from it. It has a lot to be desired.
And you go over to HF and it's even more mysterious https://huggingface.co/models?pipeline_tag=text-classificati...
Maybe an agentic eval framework that tests anything that hits 100 downloads would be a decent weekend project...
Also there's like wildly different tasks. choices versus noul have way different performances and also if you are doing things like text classification such as using it for theme extraction, I mean this does work with nimble-v3 but d1 gives mostly noise and pplx-decider is in between and their performance degradation curves are dramatically different depending on the input-token length.
This is all going to be bitter lessoned eventually but I can't just sit on my hands and wait around 12 months.
So now I have a proxy that looks at the input size and do dynamic routing and sometimes fire off both and do ensemble weightings ... I mean why do I have to be clever here, this is a generic problem.
I had to discover all this myself and I really shouldn't have to do that...I guess I'm just expecting some quant results from AA that they aren't in the business of providing.
Same prompts as the rest of the field, blind human votes, ranked by quality and cost. It finished the 12 scenes at about $0.13 a scene. The other 4 scenes are still pending.
https://threejseval.com/models/deepseek-v4.1-flash-1017-max
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).
1 - https://youtu.be/IH8XmxiwliQ?si=lJPvYzi3mHNsiw8r
Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.
At around 75% of the full model intelligence
Heat, noise and power consumption are actually quite ok - i'm running 2x3090s, capped at 250W and i haven't seen heat and noise being an issue, sitting right next to it. Might be helping that it's getting colder.
75% of the intelligence with 2x3090s, with a 5090 if you have it it's around 90% and these are first day builds too, typically people hone in and improve these all the time.
It's true local doesn't make sense value wise against online, but if you have the hardware, having "unlimited" tokens with no 5h limit, no weekly limit, is super nice.
But here my price per kwh is 40 cents (euro).
But if you integrated it with solar system and a heat exchange, this will be feasable
Plus, you know, this thread was about running AI on "lame consumer hardware". There are more power efficient options available.
But it accomplished everything I was looking for it to do, even if Claude perhaps could've done it with less guidance on my part/faster. I'm pretty impressed with it's performance.
Does deepseek v4.1 flash 1017 really take that much longer per task?
The only model that works for me is Astra, but it's expensive.
then again, I'm not a heavy AI user, so maybe I'm missing something.
For this model in particular, even if it didn’t get stronger and it simply thought less that is a big win on its own. DS4 can be served at a hit tok/s, but it takes a while because it thinks for so long.
This is half the size of Mimo and comes pretty damn close to it and I feel like lots of tasks can be done by it.
I really like its speed as well but its quite verbose from the benchmarks, IMO, this should be tackled in the near future as well. I wonder if specific fine-tuning can happen just to make the model token-efficient.
Also, slightly offtopic though deekseep has good tps but Nvidia's nemotron recent 120B model also has some really crazy tps. I wonder if we are gonna see some really good tps on general hardware.
The best part is that the amount of current innovation happening within the space means that a lot of good ideas are bound to be absorbed by one or the other and we will get many of these improvements handed to us :-D
I think that such engineering is also the engineering of the trade-offs and on what axis to play on. You can be SOTA but with very low speed, extreme price and other factors. I think its interesting to see how each engineering lab approaches these trade-offs and what they pick. But whatever they pick, its best because its more choices for us to pick upon from (especially if its open weights)
give it browser access and it's a speed-run to a cloudflare bot detection freak-out in minutes.
It feels like that would be the expected result, all else equal.
I am just extremely impressed and surprised by the fact at how small we are going. My dream of local inference is perhaps coming close.
Regarding 12B one (Nemotron), still, there are other models even comparing it to qwen 9b, its tps difference is: 435 tokens/s vs 68 tokens[0]
I am not sure if this is because of how AA does its calculations for the speed but my main point was that I am impressed by the speed improvements happening in general hardware. I remember this speed when it was only expected from cerebras or groq.
Also speaking of this I found K2 Horizon[1] which seems to have 1,061.5 tps at 23b and it seems to be really close to actual open source (sharing their training code, pre-training, post-training, checkpoints etc.), I am quite impressed by them as well! [And I think that I wouldn't have found them if it wasn't for their token speed perhaps]
[0]: https://artificialanalysis.ai/models/comparisons/nemotron-3-...
[1]: https://artificialanalysis.ai/models/k2-horizon-375b-a23b & https://ifm.ai/blog/k2/
So you are right about it and I think that I am just excited about how deepseek although having the same architecture as previous times is just able to make consistently good upgrades and just honestly impressed by it all :-D
The new model:
DeepSeek V4.1 Flash 1017 (Max) 552B A16B
Matches
GLM 5.3 753B A40B
MiMo 1T A42B
Nicely done!
It also has 196B ngram params that sit on disk. So it's really pretty comparable to GLM 5.3 in size.