Rendered at 16:22:23 GMT+0000 (Coordinated Universal Time) with Cloudflare Workers.
nikvdp 1 days ago [-]
What are people actually using these hosted search providers for? For personal use I've found using searxng-ai-kit [1] (disclaimer: I'm the author) or similar tools to be far more cost effective (free!). oh-my-pi uses a similar method and will scrape search results for you out of the box, no api key or search provider needed.
Any organization that wants reliable latency, data processing guarantees (it's a big leak vector for company secrets), etc., will use such a search provider over the solutions you outlined.
Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
Can you explain how this would help protect a company's secrets? I would think that if you want to keep your IP secret surely using something like Deepseek's API directly is not ideal and hosting something locally is best practice.
hobofan 1 days ago [-]
I personally also wouldn't trust Deepseek with companies secrets. That's why I linked LinkUp, which I know has passed through a quite strict compliance process at a few of our customers.
Obviously, hosting a search index locally would be best practice, for the search part, but usually comes with quite a significant hardware cost and operational complexity, if you want to have an up-to-date index. I'm not aware of good self-hostable options in the space but would be happy to hear about them if anyone here has experience with that! AFAIK LinkUp and some other provides do offer private cloud/on-premise deployments, but I think mostly for their biggest customers.
For the web fetch/access part of the equation, that is inherently live and not containable for a local deployment (unless you would host a whole internet archive/Wayback Machine, which is even more of an undertaking). There, the main options you have to prevent leaks, are a domain allowlist (if feasible), doing query parameter stripping (error prone, and still no 100% guarantee), or configuring the client (e.g. our tool) in a way that users have to explicitly allow those tool calls, and are informed about the posed risks. We are also working on a system, where certain tools are fully deactivated once e.g. an office document that's marked as confidential enters the context.
For the highest requirements on company secrets, an air-gapped system that doesn't have web search/access capabilities is obviously the ultimate solution.
visarga 24 hours ago [-]
Why can't we simply save search stubs, short documents a LLM can retrieve, use them to route to various websites, resources, feeds, forums, blogs, other specialized search engines, APIs? If you host a 30GB model, then you can use 30 more GB for a compressed search stub index. The entry points for any domain are more stable than the pages. You would use an agent to navigate the resources as needed, fetching fresh data.
It's not quite search, but it can replace most search. It is a router for search, where a local agent performs the live interaction. The local agent does the live work, this maximizes privacy too. When your local stub index doesn't cover the topic of interest you can use any coding agent claude/codex/gemini/etc to research and generate the missing stubs.
Transformanshen 20 hours ago [-]
Actually that sounds quite logical
visarga 9 hours ago [-]
Could be the missing link in the sovereign stack - an agent with a search stub library running locally with higher privacy than using dedicated search providers.
jstrasner 2 hours ago [-]
Curious about this too, though tbh the search call becomes a new data path to China even if the model runs in the EU. Has anyone found an EU-hosted search API that's close on price? What about quality of results?
pstuart 20 hours ago [-]
That's a good idea. And if a team is working in a specific problem domain, their searches are likely to be repeated by others or overlap to a significant degree. With locally hosted agents that could probably be extended to caching the applicable search results (noting freshness, relevance, and accuracy) and have some sort of RAG-like internal search portal.
jeromechoo 1 days ago [-]
Diffbot might be the only company that offers a self-hostable fully airgapped web index with access to content cache (ala archive.org). I work here, happy to share more info about how it works!
Barbing 17 hours ago [-]
Y’all are honestly presenting your accuracy against a competitor right on your homepage - impressive.
Love this concept! Take the web “on a boat” :)
kaitocross 14 hours ago [-]
YaCy is something you can host, and it has a peer to peer distributed index. If the rankings would be more useful tho...
jgalt212 1 days ago [-]
I would use this to discover leaks, not to protect data.
yuhond 17 hours ago [-]
[flagged]
Havoc 1 days ago [-]
I have a searxng instances but stopped using it after doing a comparison vs actual search provider APIs.
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
fudged71 17 hours ago [-]
I ran into the rate limiting very fast. Then i realized i had a SerpAPI key. I checked whether SearXNG supports it and it… doesnt, which is bizarre.
xienze 1 days ago [-]
> searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
vardalab 22 hours ago [-]
Yeah, I had the same issues. I just ended up paying for Brave and Exa. It's literally like, at my volumes, a couple bucks here and there. Nothing to really think about at all. I am a heavy user of AI, but it self-hosted AI, but still the volumes have nothing to worry about. But I'm not trying to scrape Web or anything like that.
kaitocross 14 hours ago [-]
Qwant as search engine is pretty good.
MrDrMcCoy 1 hours ago [-]
Sadly, they're really difficult to integrate with SearXNG due to the aggressive captchas.
mosselman 1 days ago [-]
I have built my own search engine, which is really great btw. I self-host it.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
lxgr 1 days ago [-]
> I am the only user and still get blocked.
I’m not surprised. Google these days regularly blocks me, a human user with a Google account with decades of tenure.
thombles 1 days ago [-]
For OpenWebUI and also OMP with local models. Searxng is, uh, playing fast and loose[1]. More engines are implementing anubis or other challenges, which searxng attempts to implement/bypass, but they are also making these services inactive by default to limit load[2].
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
IMO unless you refuse to play the LLM game altogether (then you wouldn’t be having this discussion in the first place), at the end of the day you’re using tech built on top of disrespecting every IP claim and TOS out there on a fundamental level, so no one’s clean and it’s only a matter of how much you can deceive yourself into believing you’re clean.
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
thombles 1 days ago [-]
I take your point, but there remains a difference between a multi-GB file sitting on my hard drive and my use of a large complicated database fielding my search queries. I don’t believe that anybody is obligated to run a search engine for free for my benefit and it’s logical to pay a fraction of a cent if that makes it work.
Barbing 16 hours ago [-]
> $0.75 per 1000 searches …[&]… you really [want] the anonymity.
Anybody pooling searches and charging a buck for it with at least a “trust us” privacy promise? Or $1 per 500 even. (Kagi’s more, understandably.) Unsurprising if somebody’s already thought to do this.
But check the fine print. They talk about the providers guaranteeing ZDR, but I missed the part where they themselves do also.
anon373839 11 hours ago [-]
Update: There is no such guarantee. Cloudflare does label some providers as ZDR. Others are non-ZDR (such as Exa). Cloudflare itself may be retaining request and response data.
Faaak 1 days ago [-]
How do you deal with fcking cloudflare blocking access to the websites ?
I see the Cloudflare challenge even on normal Chrome & Firefox very often, 100% of "first access" (after some timeout) on many sites.
fpoling 19 hours ago [-]
Cloudflare has verified bot program.
dofi4ka 1 days ago [-]
Hosted search API from your LLM's provider is much easier to use, faster and less error prone. Also I believe that LLM may work better with its provider's search api, as it was definitely trained on it. I mean, it is just more convenient to use your provider's search.
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
lxgr 1 days ago [-]
Self-hosting LLMs is much more feasible than self-hosting search, so that seems like a relevant use case.
apwheele 1 days ago [-]
I am currently using Perplexity for https://veruscite.com/. Verify citations actually exist -- so get in a list of 100, half I can check with crossref, but the other half I need to do a web search and fetch to make sure they exist.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
xhrpost 22 hours ago [-]
Is this what is causing at least part of the massive bot traffic hosts are seeing to their sites? Many direct-to-inference users are just retrieving the site locally every time they need it, let alone sharing site content across users. The big providers like Claude and Google implement their own caching so that when an agent is doing research, if 100k different prompts need the same site content, they don't hit that site 100k times.
karmakaze 23 hours ago [-]
I ran into a thing running a local dense LLM on DSH (without DeepSeek API key) that doesn't use many MoE experts with vast knowledge. It ends up doing a lot of searching. In the end my local AI is as free as electricity but I pay for the searches.
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
razster 22 hours ago [-]
I've stopped using the browser and have my Pi/LocalLLM(Qwen3.8) do my searching for me. Made a nice interface that will display top results.
No need to pay for a thing, except electricity, which that is the next step, solar powered LLM server.
ravetcofx 12 hours ago [-]
What hardware runs Qwen? Certainly not the pi right unless you have a very small quant?
ATMLOTTOBEER 1 days ago [-]
The only reason this exists today is because normal search providers haven’t been completely disintermediated.
As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this
smrtinsert 23 hours ago [-]
Don't you put yourself at the mercy of bot banning with this approach? It's probably good for poc use cases, but I wouldn't count on it in production.
saikatrahul 1 days ago [-]
[flagged]
londons_explore 22 hours ago [-]
Since SEO is big business, and agent driving search is probably the future, I suspect we might see the price of search API's going negative in the future.
Ie. Providers pay you to use their API over competitors, since then they get control of who has influence on the internet. They can probably get sites to pay to be up ranked for specific queries etc
christina97 14 hours ago [-]
Payment for search flow!
thombles 1 days ago [-]
For comparison that’s $5 with Brave Search API and $12 with Kagi API
wmchen 1 days ago [-]
Ceramic.ai is $0.25/1000 searches. I've been using it as an MCP in OpenCode and it seems adequate. I don't see many people talk about it, and I have not benchmarked it against Exa/Tavilly.
breadislove 1 days ago [-]
ceramic is very very bad, the worst search provider of all. it feels like it is doing only lexical matching and the index is very outdated
vlmutolo 1 days ago [-]
They explicitly are only doing lexical matching. I actually don’t mind this if it lowers the cost; LLMs can turn lexical search into a very powerful tool by iterating on the queries. Just look at the success of coding agents using grep and similar code search tools.
But it does seem like their index isn’t great.
breadislove 8 hours ago [-]
yeah, but the bigger the corpus the more you are relying on the LLM guessing the right keyword, if there is no reranking in place, you will just end up burning through a lot of tokens.
coding agents is a very "niche" use-case where lexical matching works very well. code is a graph, by looking at few lines of code you can guess naming conventions etc. codebases are small compared to knowledge bases.
the bigger your index, the more you will need way better tools such as search.
lp92 4 hours ago [-]
Ceramic is terrible. I did a search for a current stock value and it returns nonsense links from 2024.
loehnsberg 1 days ago [-]
Kagi MCP works really well imo also within DeepSeek Harness. Just looking at cost neglects the possible difference in quality. How good is that DeepSeek web search/fetch?
ece 1 days ago [-]
$4 with Exa, $1.50 with SERP API
ttul 1 days ago [-]
SerpAPI is comically good at evading Google's scraping protections. If what you want to know is, precisely, what someone searching Google in Pakistan would see, this is the API for you.
hackernud3s 16 hours ago [-]
That just means they have proxies in Pakistan, that's not as impressive as you're making it sound.
hackernud3s 16 hours ago [-]
$1 with Serper, and it gets lower with volume. I thought I remembered Serp Api being higher than that. They do have a decent free tier
stncls 1 days ago [-]
Mojeek is £ 2 per thousand. They have their own index (not a SERP), but subjectively pretty close to Google (I prefer it to Bing/DDG).
hobofan 1 days ago [-]
$1 for LinkUp fetch, $5 for LinkUp search.
satirr 1 days ago [-]
is the cost of search and scrape provider is impact the quality too? for the search is yes, because there it shows the quality and depth of the search. but for scrape, wht the cons?
mjmvisser 1 days ago [-]
These services only exist because Google refuses to provide a paid search API.
aucisson_masque 19 hours ago [-]
Yes it’s maddening.
It also highlight that Google really isn’t about search, all they care about is advertising and making users profiles.
ijidak 1 days ago [-]
Have never understood this.
A paid search API seems like it would be easy money for Google. And maybe some lock-in for all sorts of start-ups and enterprises.
There are so many apps that would benefit from search.
I get that it means they can't serve ads.
But you've got to be willing to disrupt yourself or you'll lose eventually anyway.
Better to figure out how to make money from ads and an API, than to cower in fear.
That same fear may be why they failed to be first movers in LLMs.
Have to be willing to disrupt yourself.
At the same time, I've never built a billion dollar company. So I'm absolutely armchair quarterbacking here.
jsrozner 23 hours ago [-]
Google's goal is not to provide search; it is rather to gather infinite data about people and monetize it. If google offered a search API, it would destroy its surveillance system.
We should have outlawed targeted advertising and Google should have been broken up under antitrust a long time ago.
17 hours ago [-]
londons_explore 22 hours ago [-]
> I get that it means they can't serve ads
Easy enough to run the ad-engine over the query, estimate the ad revenue, and bill the user the value of the lost revenue.
They are free to recoup that by serving their own ads or even using Google's ad services.
ijidak 20 hours ago [-]
Wow. That's a great concept.
Exactly why it seems to make sense to be willing to disrupt yourself.
There are so many ways to still make money.
But, I'd be willing to bet that avoiding self-disruption increases the likelihood of failure over longer timescales.
oldsecondhand 21 hours ago [-]
Google sold search appliances between 2002-2016, the replaced it with Google Cloud Search.
LTL_FTC 23 hours ago [-]
Google provides Vertex for search, but it’s quite slow and so probably only useful for async workloads.
prism56 1 days ago [-]
Thought they'd be allover it. Dropping in cheeky ad results.
techscruggs 1 days ago [-]
Everyone would just programmatically filter out the ads.
draw_down 1 days ago [-]
[dead]
DoctorOetker 23 hours ago [-]
@DeepSeek teams, why not train your next series of LLM models with source-aware training?
LLM's would be able to answer questions, delineating exactly which facts/assumptions/reasonings are prior work and from where (this URL, that book, etc.), and which parts are it's own deductions, or the user's.
londons_explore 22 hours ago [-]
Notable that humans can't cite the source for a huge chunk of their knowledge. Eg. When/where and from who did you learn the meaning of the word "garden".
Your understanding of the world "garden" probably came from 5+ encounters with the word in different contexts, and you cannot separate them - instead your understanding is some kind of average of the knowledge from each encounter.
Interestingly, adjacent and overlapping-meaning words like park, meadow, yard, grassland etc often have disputed boundaries - ie. One person might call something a garden, whilst someone else calls it a meadow. Neither can explain quite why they have chosen their specific word for the object in question, but it might stem from the source of where the meaning of each word was initially learned.
jamienk 22 hours ago [-]
This is the whole Chompsky/Kant thing - the idea of "Poverty of the Stimulus" innate structures. You didn't "learn" a word/concept by direct exposure, but by socialization, association, emotional connections, vibes. LLM seem to learn the hard way.
The Saussure idea of the differential nature of language seems relevant here too: words aren't just lists of things you look up, they are connected to each other and distinguish themselves by being slightly different from the other elements of the clusters they belong to.
frenchtoast8 1 days ago [-]
For comparison, Kagi charges $12 per 1000 searches (only talking about their API pricing, like what you’d use with an LLM; their $10/mo plan has unlimited searches but it’s a TOS violation to use those credentials with an LLM).
Havoc 1 days ago [-]
>it’s a TOS violation
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
Thanks for sharing, wasn't aware of this provider. If anyone is wondering about rate limits for the subscription:
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
In other words, good enough for casual use if you were already going to subscribe. Assuming the rate limits are okay, Uruky’s €5/month becomes cheaper at these search volumes:
1. Tavily Basic: 1,704 searches/month
2. Brave Search: 2,126 searches/month
3. Exa Fast / Auto: 2,233 searches/month
4. Exa Instant: 3,907 searches/month
5. DeepSeek Search: 7,503 searches/month
6. Ceramic: 22,507 searches/month
Havoc 1 days ago [-]
I ended up building an MCP that pulls results from multiple places and dedups it.
The free tiers on most are generous enough for normal agent MCP usage
Also...pretty sure decodo just launched a search thingie
jimmydoe 22 hours ago [-]
One thing I see on the new gen of ai search bots are they fake chrome and rarely disclose themselves in UA, and not respect robots.txt, and no rate control for smaller sites.
I’m not accusing any one engine discussed in this thread, but hearing a proxy vender like decodo providing search give me a feeling there’s non zero over lap to bad search and residential proxy crawler.
Havoc 21 hours ago [-]
Think the era of respecting stuff like robots.txt is over.
I build a web-fetch tool for my own LLMs that is essentially the same tech as web-scraping except its single requests not high volume.
You can't really get anywhere these days without disguising UA etc.
jimmydoe 21 hours ago [-]
Real human hit once with disguised ua is fine, but I’ve seem a few million clearly batch crawl request within a few minutes hitting my potato server from thousands of IP addresses, and they do it every four weeks almost at same hours. cloudflare doesn’t help much if at all, I can do proof of work but it creates problems for real human, I can rewrite in rust with some sophisticated cache but i don’t want to.
Anyway I’m not expecting a solution. Just say this sux.
BrunoBernardino 1 days ago [-]
Wow! Thank you so much for sharing! Happy to answer any questions about it.
PhilippGille 1 days ago [-]
Kagi has an official MCP server [1], where you set the API key. MCP use is always by an LLM, so this seems to be at odds with their TOS.
How well does this (and similar services) deal with anti-bot protection? What happens if I attempt to retrieve a page protected by the cloudflare checkbox or by Anubis? An error, no useful content, or the actual content I want?
tyre 1 days ago [-]
> Search API finds web pages for a query and returns structured results. It is powered by DeepSeek's in-house search engine.
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
spiderfarmer 1 days ago [-]
I have always blamed Chinese AI operators for the hundreds of thousands of residential proxy requests I receive each day, but there's no way to be sure.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
nativeit 1 days ago [-]
If there’s no way to be sure, then why blame Chinese operators? Because American companies have given you so many reasons to believe they’re acting in good faith?
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.
lgats 17 hours ago [-]
I suspect it's because of the number of requests that come from Chinese datacenter ASN values. If you block them, you subsequently get visited by residential proxies from South America and SE Asia, likely the residential proxy network they are referring to. source: i have the same issue...
MintsJohn 1 days ago [-]
Identified AI bots that finally made me cave and put anubis (with very liberal limits) up were anthropic/openai/perplexity according to the headers they sent. They were also over 99% of traffic despite proper robots.txt (site only has a few hundredth visits per day), which they ignore. I don't mind bots or crawlers, but these bots got the same page/query string every time they decided it was time for a crawl, which was about daily, cache control, last modified headers they also never seemed to be interested in. They were very very far from well behaved, unlike search engine crawls, and impossible to tame.
ranger_danger 14 hours ago [-]
> according to the headers they sent
Why not just block the requests based on those headers then, instead of inconveniencing everyone?
MintsJohn 5 hours ago [-]
The honest answer is i was fed up and choose the nuclear option out of frustration and just wanting to try a new toy. So yeah your right, i'll try that.
I don't think the inconvenience is that big though few secs loading page per session. The ridiculousness though kinda is, we've had years and years of all manner of crawlers that managed to behave, i've even used site rippers in the dial-up days that avoided hammering servers and fetching dupes, now the ai-bros come along and tear the internet apart, replacing it with their chats systems and generic slop.
vincnetas 1 days ago [-]
Yeah, but how these residential proxies get in the residential areas? I would guess popular VPN providers are mostly behind this. They sell you VPN and sell bots residential proxy access on the other end.
esseph 1 days ago [-]
Third party. A third party provides endpoints (hacked routers, phones, laptops, desktops, IoT devices, etc.)
weird-eye-issue 1 days ago [-]
There is no single answer to this, it entirely depends on the exact service and the exact page
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
bulder 1 days ago [-]
From what I've seen, when operating in an environment with bash tools, Claude and ChatGPT will react to a site blocking their normal, attributed fetch tool by calling curl or writing a python script with a different user agent to try and bypass the blocks, rather than respecting the site owners' wishes.
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
gvfsa 1 days ago [-]
Claude and ChatGPT are user agents. For obvious reasons, they serve the user, not the websites' owners. (Also, if you don't want people to be able to access some information, then don't put it online.)
bulder 1 days ago [-]
My point is that they're not serving the user in that case, since they're taking actions that bypass site terms without consulting the user. And for example, if the site forbids scraping, that attempted block bypass can get the user banned without ever seeing any warnings.
TeMPOraL 22 hours ago [-]
> My point is that they're not serving the user in that case, since they're taking actions that bypass site terms without consulting the user.
As they should. When I ask an agent to find something for me, especially when I tell them what exactly or even give the URL itself, these fetches are done in my name. I do not appreciate agents coming back to me, saying "I'm afraid I cannot do that - robots.txt disallows that".
I've had to bring up the ethical argument a few times, fortunately most of the agents would oblige once I reminded them that they act as User Agents, in which case robots.txt is explicitly not applicable to them. I only got pushback on this from Fable, which claimed it's not an User Agent, but has to take into account the interests of Anthropic too, rejected my argument and refused the request.
1 days ago [-]
matkoniecz 1 days ago [-]
> if you don't want people to be able to access some information
Claude and ChatGPT are not people
gvfsa 1 days ago [-]
Chrome and Firefox aren’t people either.
pseudosaid 1 days ago [-]
they are tools. who do you think is using them? dogs?
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
esseph 1 days ago [-]
> they are tools. who do you think is using them? dogs?
My agents are using headless chrome :P
pseudosaid 19 hours ago [-]
yeah tools using tools on behalf of a user
weird-eye-issue 1 days ago [-]
Yeah I should have clarified I was primarily referring to the standard web interfaces, however since pretty much all of these tools can execute code these days it is true that technically they can and often will work around blocks. The degree to which this is intentional isn't super clear - if they locked it down too much then it might start refusing to write code that does any sort of web requests which would obviously be an issue for just standard development tasks so it's not super straightforward. But it is true that the built-in tools that they have access to to fetch pages have a hard coded user agent that identifies them
bulder 1 days ago [-]
Fair enough yeah, I think the thing that is annoying is that the service design is explicitly disincentivizing keeping an eye on the tasks, with all of the "set it and forget it" features. You really have to put in work to make sure that the model takes breaks to consult you instead of making assumptions (which in OpenAI's internal testing seem to be 'the user wants me to break into a site').
skeledrew 1 days ago [-]
If you hire someone to handle some task for you, do you want them to keep running back to you every time they run into a blocker that they could potentially resolve on their own?
manquer 23 hours ago [-]
If a developer I hired ignored a GPL license requirement and solved his blocker by using a library without asking me I would be quite concerned.
Nobody would be concerned if they didn’t have access to curl and used wget instead and didn’t ask permission .
However if that solution involved breaking the law yes I absolutely do not want a representative of me breaking the law and yes it is breaking the law whether we like or not under CFAA.
It is not about finding solutions to problems on their own; the solutions should respect the agreements we have made and the law .
skeledrew 23 hours ago [-]
The application of law is also a subjective thing; it depends on jurisdiction. So if you're US-based and you hire a US-based dev who breaks US law, then you definitely should be concerned. If it's an Indian-based dev breaking US law though... then it's totally on you as the one who did the hiring to not have told them to follow US law. Same re using libs that you may consider questionable or outright avoid.
nativeit 1 days ago [-]
That depends if they routinely and cheerfully behave in objectively insane ways.
skeledrew 1 days ago [-]
> objectively insane
Contradiction: one person's insanity is another person's genius. And yet another's "Meh, this is the norm".
bulder 1 days ago [-]
I'd prefer my little subcontractor to respect the commons on which I depend.
gvfsa 1 days ago [-]
In what way does loading a website using the Claude Code harness not respect the commons, as opposed to using Firefox? If anything, using Claude Code is better, since it doesn't load the unnecessary static files (javascript, css, etc), saving the website CPU, I/O, and bandwidth.
skeledrew 1 days ago [-]
Are you still paying them for the time they spent when the task you gave them remains incomplete, or you end up spending the time/effort you hoped to avoid spending by hiring them in the first place?
senordevnyc 23 hours ago [-]
Well yeah, if I ask Claude to look something up on a website so I don’t have to, I don’t give two shits if the website author thinks I shouldn’t be allowed to do that. Just like I don’t care if they think I shouldn’t use Firefox and they block that browser specifically. I’ll route around that however I please and I won’t lose any sleep over it.
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
toofy 20 hours ago [-]
> I don’t give two shits if the website author thinks…
fair, just don’t cry when website owners don’t “give two shits” about your concerns.
senordevnyc 18 hours ago [-]
They’re free to try and block whoever, obviously, but spare me the moral outrage.
minraws 1 days ago [-]
This is not true, Claude and ChatGPT have sent traffic to sites I maintain without any user-agent, it might be they have stopped doing this.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
ReptileMan 1 days ago [-]
>Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
The last part is extremely dubious with some recent reveals.
jack_pp 1 days ago [-]
recent reveals show that OAI and Anthropic give users the ability to go around bot protections or they internally use it?
amluto 1 days ago [-]
It’s not exactly recent or a reveal, but I’ve caught at least several OAI models writing Python scripts to access web pages.
skeledrew 1 days ago [-]
If you the user ask it to access a page (or do any task that probably needs that access), then it'll try to do so the best it can. Anything less is user-unfriendly behavior.
amluto 1 days ago [-]
I disagree. If I turn off web search in the config, then the tool should not search the web, thank you very much. Maybe if I very explicitly ask it to, then it should try.
The whole succeed-at-all-costs idea is a big part of why there have been so many massive security screwups lately.
skeledrew 1 days ago [-]
Well if you disable web search, that's just making that built-in tool unavailable to the agent (when it's enabled, the agent gets usage info and guidance about it in its context). Given a task in which online resources may be useful, the only thing the agent knows is that it could search the web, and that there are a variety of means (curl, wget, urllib in python, etc) which may be at its disposal. Without you telling it specifically not to search the web (or that guidance coming from elsewhere) or blocking internet access, there's no reason it wouldn't use that avenue in fulfilling its task.
amluto 1 days ago [-]
I am well aware of why this phenomenon happens. That does not mean that I think that OpenAI’s (and Anthropic, etc’s) training or harness behavior is appropriate.
Would it really have been so hard to encode an indication of whether use of the public Internet is permissible in the prompt and to train the models to not attempt to exceed authorized access?
skeledrew 1 days ago [-]
I'd say for the greater majority of users, when they tell their agent to do some task, particularly any that involves research (which isn't specifically about something on the local machine), using the internet is implicitly expected. Imagine having to go into my computer settings to flip an "allow internet usage" each time I want to use a browser. That's extremely user-unfriendly, and there's 0 reason to encode a default internet block when users expect their agent to handle their given tasks with minimal friction (I still recall the frustration of babysitting Claude permissions just a few months ago; auto-mode was the best feature ever for those of us who didn't want to go YOLO).
ReptileMan 1 days ago [-]
I have noticed that agents become extremely creative to fulfill any non refused requests. And then instructions become more of a suggestions.
m00dy 1 days ago [-]
I've spent a lot of time building datamint.xyz, and dealing with user agents, datacenter IPs, and VPNs gets pretty tricky. What works right now is a fork of the latest Firefox with anti-fingerprinting patches, paired with a high-quality residential proxy network with clean IPs. Even the basic setup needs 11+ code-level patches, and there are still a few edge cases, but this setup should cover 99% of cases.
thisismyanon2 1 days ago [-]
Thank you for escalating the arms race that's currently destroying the internet. I can't look at the Apple user forums, my health insurance company's portal, pay my county taxes, browse Digikey and Mouser, view Github or whatever forge, etc. etc. without having to jump through Cloudflare or Anubis garbage designed to block your abusive behavior.
I'm just overflowing with joy that you're comfortable wasting my time and money.
lukewarm707 1 days ago [-]
I use the automated captcha solvers, stealth browsers, proxy IPs and so on. In regard to ethics, I think this is the right thing to do. The minimum I can do is just to show this is not a matter of excuses or mental gymnastics to trick myself into thinking it's fine.
Even if you said, a lot of the public commons is destroyed, I would NOT use bots and cause more destruction because 'everyone else does it' or 'it's a lost cause'. I say it is the right thing to do because the balance is such that using the internet is actively harmful to the user. That is, using the internet with bots is neutral to the user, and using it without bots, you are being actively harmed and exploited.
If you are using search, this is not a library. You are using your time to train targeted advertising that exploits you; you are selling your search history to the highest bidder. If you are using ChatGPT you are just working as a teacher for free.
On services, you have paid for the product, but the seller is scamming you by not giving you the product. The seller is using dark pattern UI to trick you.
If you are on social media, don't think of this as a public park. You are being treated like a lab animal, your behavior is being studied for the benefit of a corporation.
So the point about the bots is that I think people should be empowered to protect themselves from the internet, which is otherwise actively harmful to their wellbeing.
stavros 1 days ago [-]
Even if it's not the GP, if there's profit to be made, somebody will do it. I was reminded of the power of economic motives the other day, when it turned out someone bought an entire trustworthy reseller just so they could compromise the Ledger supply chain by including a tiny eSIM module that OCRed the screen and transmitted the seed.
Incentives are crazy powerful.
matkoniecz 1 days ago [-]
> if there's profit to be made, somebody will do it
Not always! With sufficiently powerful barriers it can be blocked at least in part.
Shaming awful people is one of available tools.
Or at least not praising them for making small profit at huge costs to others.
stavros 1 days ago [-]
Can't argue with doing what we can.
1 days ago [-]
stev1983 1 days ago [-]
Here’s what is says on their site:
“Courtesy backoff
Adaptive backoff keyed to target response codes. When a target returns 429 or 503, we back off immediately, so our fleet never applies denial-of-service pressure to a site.”
By the time you get the error you have already crashed the site.
Geez.
lukewarm707 1 days ago [-]
"Usage records — which endpoint was called, when, the credit cost, the status, and the target URL."
I would use it, but then the privacy policy says that you would store my search history. I simply want to use the internet in peace without being constantly recorded.
m00dy 1 days ago [-]
hey,
I didn't get your point, how would you do design our usage records then ?
Try not to store: The user's search queries, the user's prompt/response content, the user's web history (URLs, etc).
To avoid selling the llm prompt content you would need your own inference or a contract.
Sadly most just hoard the data and some will offer a model where they charge for privacy. They are all quite scammy.
Firecrawl: logs. Parallel: logs. Bright data: logs. Kagi: logs for abuse. Bocha, querit etc: logs logs logs. Cloudflare: doesn't say what they do. Exa: charge for no retention. Linkup: no retention on request. Brave: no retention on request, but actually approved it (win for brave).
luckylion 1 days ago [-]
Anti-fingerprinting is mostly there to be able to use the same browser across many requests, I assume?
I found that a slightly-modified firefox for more stealthy remote-control works fine and gets a pass on all automatic captchas I tested. but I don't do it at scale, I just wanted something to automate a particularly stupid website that I have to use.
Would love to see if my browser is detected -- any recommendations for tests to run?
latexr 1 days ago [-]
I have to agree with the (sarcastic) sibling comment. You’re essentially making everyone’s life worse in the name of personal profit.
Please stop. The last thing we need in the world is more greed and selfishness.
lejalv 1 days ago [-]
Since when have moral appeals worked against the central feature of capitalism?
Only strong regulation can help.
matkoniecz 1 days ago [-]
Shaming awful people is one of available tools.
Or at least not praising them for making small profit at huge costs to others.
Also, it is useful as coordination method, enabling passing laws outlawing some behaviors.
(and I am not able to legislate-via-http while posting such comment is easy)
1 days ago [-]
czk 1 days ago [-]
usually a combination of something like camoufox and a residential ip address
m00dy 1 days ago [-]
you can bypass those easily except Google Captcha v3, you need to have your own proxy swarm to make it at scale.
agentdev001 1 days ago [-]
Artificial analysis' compilation of search api costs per 1000 queries, for comparison:
About a year ago I built my own search and fetch MCP tools for agents based on third-party scraping providers. It is about 15-20x more cost-effective than what OpenAI and Anthropic is offering with their integrated web search and fetch tools. Very impressive that DeepSeek has such a competitive offering today.
aurareturn 1 days ago [-]
Source for those scraping providers?
suncemoje 1 days ago [-]
ScraperAPI and ScrapingDog. The latter also offers Google Search at a fraction of the official Google API price. Wrap their APIs into an agent/MCP tool et voilà.
dsco 1 days ago [-]
DeepSeek Flash really came at a bad time before the Anthropic IPO. I use this thing daily and it’s neck and neck with SOTA models. The search is one of the things setting it apart.
rjh29 1 days ago [-]
It's absolutely not SOTA for vibe coding a whole app, but for guided work it is great.
platinumrad 1 days ago [-]
Is anyone actually oneshotting a whole app or even vibecoding one in multiple steps without it being "guided work"? I really wish announcements (like DeepSeek's recent one) would stop focusing on the ability of the model to spit out Minecraft clones on demand.
antoniojtorres 1 days ago [-]
Some people are trying to one shot entire companies
bahmboo 1 days ago [-]
And failing.
IshKebab 1 days ago [-]
Someone at work vibe coded a replacement for a entire third party SaaS that we were using in a week. Their version is better.
It wasn't one-shotted of course but I don't think there's much difference in the outcome between one-shot and like 100-shot over a week. That company is still doomed.
Think of something like Trello. Who is going to pay for that when you can easily vibe code something superior (yes you really can now).
rjh29 1 days ago [-]
Open source companies somehow make money off enterprise contracts. People still don't want to be on-call if something breaks, or have to manage servers and backups. But making money off SLAs to implement new features will probably be a thing of the past.
IshKebab 1 days ago [-]
Indeed, but only a tiny handful of companies have been successful with that business model, and they mostly make margin on things you'd have to pay for anyway like cloud services. No company is going to pay e.g. $15/user/month for Jira when they can get the same thing for free - better even. And yes it isn't an exaggeration to say you could vibe code something better than Jira. (Especially because that's a low bar!).
I'm kind of surprised nobody has done that yet actually. I guess Opus 5.5 has only been out for a very short time...
shafyy 21 hours ago [-]
> No company is going to pay e.g. $15/user/month for Jira when they can get the same thing for free - better even
It wouldn't be for free though, would it? People need to build and maintain it (and there's LLM cost associated with it too).
Let's say you are a 5 person start-up. It makes much more sense to pay $15/user/month for services like Jira, Slack, Email etc. then to build and maintain everything yourself.
For super large companies, it might make financial sense to build some of the tools in-house. But then again, that probably already made sense for LLMs were a thing. But there is more to it than just the pure financials (i.e. salary).
IshKebab 20 hours ago [-]
You really don't need to be that large for it to make sense. A lot of these products are very rarely updated. Jira hasn't improved at all in the last 10 years as far as I can tell.
Maybe a 5 person company wouldn't do it. A 100 person one though? No brainer. And it only takes one to open source their solution.
muhfuhkuh 1 days ago [-]
The Red Hat or Canonical of vibe-coded/decomp/cloned software is just around the corner.
Slop curators and tech support shoulders to cry on. Could be lucrative.
MrScruff 1 days ago [-]
OP didn’t mention one-shotting.
There are plenty of use cases for the ability to quickly through together and refine an application without reviewing the code though.
gcanyon 1 days ago [-]
I have one-shotted the two most recent relatively complex apps I've tried:
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
IshKebab 1 days ago [-]
With Opus 5.5 you can vibe code a whole app without reading the code at all. Obviously you still have to tell it what to make.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
microtonal 1 days ago [-]
People said the same for 5.0, 4.8, 4.7, and...
I am not disputing that the models are getting better for vibe coding. But I think people have very different standards of what a 'big mess' or 'maintainable software' is.
himata4113 1 days ago [-]
The only meaningful improvements were opus 4.5 and now opus 5.5, people claiming otherwise are delusional. The gap between opus 4.5 and the next big thing was massive, now the same is true for opus 5.5 as much as I hate to admit it anthropic is running the show right now.
IshKebab 1 days ago [-]
They did but IMO they had very low standards. I have very high standards and Opus 5.5 pretty much meets them (at least for Rust).
"People said it before and it wasn't true then" isn't really an argument about it being true now. There are plenty of things that happened where the zealots jumped the gun with the hype. Bitcoin for example. "People said bitcoin was going to succeed in 2005, 2006, 2007...". Or even the web. It was mega hyped in the dot com boom and then crashed and of course loaaads of stupid people thought that was the end of it.
throwaw12 1 days ago [-]
which model is SOTA for vibe coding a whole app? I always use models+agents in guided setup
the moment there is a model which can code whole app, we can fairly say software engineering is done.
IshKebab 1 days ago [-]
Opus 5.5 + Claude Code (or probably any agent harness; I don't think they make that much difference). I wouldn't say software engineering is done quite yet but yeah it's definitely never going to be the same.
redox99 1 days ago [-]
It's really not neck and neck with Opus 5.5.
lejalv 1 days ago [-]
Cost-adjusted,it might be ahead.
It's key to name the boundary conditions. Unlimited cost is a very HN assumption.
TomGarden 1 days ago [-]
Max Subscription seems to be the HN assumption most of the time
eru 1 days ago [-]
Really depends on your use case.
You are right that there's plenty of use cases where Opus 5.5 is better.
gvfsa 1 days ago [-]
If it depends on your use case then it’s not neck and neck…
eru 1 days ago [-]
Why? I'm pretty sure I can find use cases for which DeepSeek is better.
Rodmine 1 days ago [-]
True, it isn't neck-and-neck. It's neck and balls. Deepseek's balls and Opus 5.5's neck. Whatever you can build with Opus 5.5, I can build in the same time, 1/500th the price with Deepseek Flash 4.1.
vrganj 1 days ago [-]
Or at a really good time, depending on who you ask.
Not everyone wants Anthropic to succeed.
jimbob45 1 days ago [-]
If Anthropic or OpenAI fails, the other wins the US market, no? I’m oversimplifying but…that’s a pretty realistic possibility in the face of people thinking sinking one US AI company would hurt the greater US venture into AI.
cheema33 1 days ago [-]
> If Anthropic or OpenAI fails, the other wins the US market, no?
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
nkmnz 1 days ago [-]
I agree on the analysis, but not on the conclusion. What makes you think that the conditions you describe are stable grounds for a duopoly where both could be successful at the same time?
tannertech 1 days ago [-]
If one fails the bubble pops so you're right.
vrganj 1 days ago [-]
What if both fail?
tourmalinetaco 1 days ago [-]
If they did then Google would take over, and others would form from their competent engineers. Also, they don’t need to be profitable to succeed, they merely need to hype investors enough to ensure that their losses matter less than their growth projections.
tyrabound 1 days ago [-]
I wonder if Google won’t end up the last man standing in the end in the west, after Anthropic that they have a major investment and partnership with, and OpenAI crumble under their large financial burdens; while Google has a far deeper actual cash pockets, more diversified, broader and deeper bench in all related matters on top of the massive name recognition and… even if a bit brittle… a “community” of users through Gmail, Google.com, Google workspaces, etc. not to mention the TPU space and their quantum breakthrough.
vrganj 1 days ago [-]
What if all of them fail because the capitalist model of centralization followed by rent-seeking gets foiled by an open alternative?
That is explicitly the Chinese strategy.
Now, what if the largest capitalist economy went all-in on an AI that might no longer be sufficiently monetizable?
porridgeraisin 1 days ago [-]
> explicitly the Chinese strategy
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
vrganj 1 days ago [-]
Let me quote the big guy himself:
> First, we should adhere to the principle of openness and win-win and boost innovation-driven development. As a new engine of world economic growth and an accelerator for the shift of growth drivers, AI is moving from the digital world into the physical world. We should seize this rare, historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries and forward-looking planning for future industries, so that all sectors and businesses can benefit from AI
-- Keynote speech by Chinese President Xi Jinping at opening ceremony of 2026 World AI Conference
As a socialist myself, the above comment looks like a prime example of what Mark Fisher would call capitalist realism, the ideological belief that there is no alternative. The fact that somebody would think past quarterly earnings appears unthinkable to the committed ideologue.
They wouldn't give away R&D for free, they would give it away to destroy the capitalist mode of production, as outlined above. If that isn't a worthy goal from China's perspective, then I don't know what would be.
China isn't the US. They don't share the same ideology, strategy or conclusions.
As for the export controls, they explicitly exempt already-open models and set up an automated streamlined process for future ones.
porridgeraisin 22 hours ago [-]
> As for the export controls, they explicitly exempt already-open models
Well... obviously. What the fuck is the point of export controlling an already open model.
> for future ones
Nope. >2T models (i.e todays frontier) is explicitly not allowed.
> China isn't the US. They don't share the same ideology, strategy or conclusions.
Money and power know no ideology. They do pretty much the exact same things as the US as well (from a 3rd country's POV). They expand territory, they capture industries, etc, etc, etc,.
> Keynote speech by Chinese President Xi Jinping
Do you take the same literal interpretations of other politicians' speeches? Surely you know thats politicking, the same way every G2/G7 leader says they want a collaborative world blah blah but it's naive to not know that it is cutthroat competition in reality.
> they would give it away to destroy the capitalist mode of production
Well, I won't take that dream away from you, but you're gonna be disappointed in this particular case, in our lifetime, with this particular country, atleast :)
vrganj 9 hours ago [-]
> Nope. >2T models (i.e todays frontier) is explicitly not allowed.
At least that link doesn't seem to have any such info? It does however have this sentence:
> It said most AI technologies would remain freely exportable, with simplified licensing for open-source technology exports
> Money and power know no ideology
As somebody who's an engineer himself and has talked to many a fellow engineer, I have noticed that engineering culture suffers from a deep conceptual impoverishment when it comes to philosophy and humanities. Most techies really have a hard time understanding ideology and it deeply distorts their world view with assumptions that everyone is like them. This has gotten them in hot water with the EU many a time, this will cause them to underestimate the systemic challenge China is mounting.
Ironically, this sort of pseudo-realism is the most ideological statement one could make. Let me cart out some quotes by noted ideology theorist Slavoj Zizek once again:
> “Ideology is strong exactly because it is no longer experienced as ideology… we feel free because we lack the very language to articulate our unfreedom.”
Here's another relevant one:
> I already am eating from the trashcan all the time. The name of this trashcan is ideology. The material force of ideology - makes me not see what I'm effectively eating. It's not only our reality which enslaves us. The tragedy of our predicament - when we are within ideology, is that - when we think that we escape it into our dreams - at that point we are within ideology.
Point being, "that is just how the world works" is just a projection of how one's own ideology works. There's all sorts of assumptions about what constitutes power and to what goal one wields it encoded in a naive realist interpretation.
> They expand territory, they capture industries, etc, etc, etc,.
I don't think any of that is incompatible with what I claimed above. Of course, capture of industry etc is a mechanism by which larger ideological goals can be fulfilled.
> Do you take the same literal interpretations of other politicians' speeches?
I don't take literal interpretations of anyone's speeches, they are mostly useful for a look at an actors self-perception. (Not everyone talks about a collaborative world, you can contrast this directly with America First rhetoric as an example!). Of course, self-perception is a factor in evaluation, but so are the actual deeds being done. Open models are getting released.
> Well, I won't take that dream away from you, but you're gonna be disappointed in this particular case, in our lifetime, with this particular country, atleast :)
Capitalism is ending either way, once capital is fully unmoored from labor. The only question left is whether control over these new productive forces will be concentrated in the hands of a few companies or owned by the public.
Hamuko 1 days ago [-]
No? Why would either of one of those have to win? Even if you limit yourself to US companies only, Google and Meta still exist.
dgellow 1 days ago [-]
The others, plural, meaning Gemini, Muse, and all the open weight models hosted by a competitive landscape of US providers. There is plenty of American players in the space outside of OpenAI and Anthropic if that’s the only thing you care about.
If Anthropic and OpenAI economics are as bad as they seem to be they deserve to fail, others will take over the remaining market. It’s not a team sport
sudo_cowsay 1 days ago [-]
i thought open models weren't SOTA. therefore not neck and neck... in your experience, do you think it depends on your type of work?
ra 1 days ago [-]
They're about half a gen behind. Very impressive to get this kind of utility from a flash series model... the impact of RL seems to be behind much of the coding and agentic improvements lately.
blntechie 1 days ago [-]
In my experience, for coding, they are pretty much SOTA. Maybe not for other sciences and problem solving.
meerita 1 days ago [-]
Models are comming every 4-6 months. Anthropic, OpenAI, all of them know this.
brianjking 1 days ago [-]
4-6 months? Try between 8-18 days for a foundation lab release.
altmanaltman 1 days ago [-]
It literally is not though... even DeepSeek doesn't claim that. It's just super cost-effective
dudisubekti 1 days ago [-]
> bad time
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
throwa356262 1 days ago [-]
Deepseek is profitable.
Anthropic is most likely not. Despite bilions of state funding and contracts.
ggnore7452 21 hours ago [-]
Ran a few quick tests on my web-fetch bench. As much as I like DeepSeek, its search and fetch aren't very good (is quite cheap tho). You'd probably be better off using free services like Jina directly, or if you're going paid, Parallel, Firecrawl, Tavily and Exa are all pretty good now.
ilya-bogin 10 hours ago [-]
We have measured DeepSeek search at Keenable too on our open-sourced NEEDLE benchmark. From our measurements its better than Jina, Firecrawl and Tavily but worser than Parallel and Exa in terms of quality
himata4113 1 days ago [-]
I wonder if providers do this so they can 'scrape' for 'free' because the customers pay for it? They all seem to have an abstraction around search rather than suggesting the use of native search apis.
larme 1 days ago [-]
I wonder if this is censored. If not then users in China mainland can use the api to workaround gfw (though there are much better ways)
arational 1 days ago [-]
Think there will be two tiers as usual, one for domestic/mainland and one for international users.
shaunpud 1 days ago [-]
I "fetched" a website with hyperlinks and it only returned the text, which makes it a bit irrelevant?
gvfsa 1 days ago [-]
Can someone explain why fetch costs money. I thought it was your locally-running harness doing it. How can they charge you for that?
BlackRabbit1 1 days ago [-]
Most of your local requests will fail (with Python/Curl/whatever) if you are not using a remote-controlled Firefox/Chrome. And even then you will be bitten brutally by captchas, rendering them basically unusable for mass requests. So it's better to outsource your web search to an external service with a stable API.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
hobofan 1 days ago [-]
> I thought it was your locally-running harness doing it.
That is rarely the case (see sibling comment for reasons). Even open source harnesses will usually guide you to connect a hosted provider for that.
Jskewel 1 days ago [-]
Is there any way to test this without an API key?
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
pprotas 1 days ago [-]
That is a really good idea, in fact I’m doing something similar. I am charging anyone reading this comment 100 bucks per letter read, please tell me who I should send the bill to.
derrida 1 days ago [-]
How does that work? I'm not a laywer, but I can invoice you $1 and the T.o.S of reading what I have to say are licensed under a $1 per view licence and by continued reading you consent to the terms of this license. Where can I send the bill for this comment?
I understand if you pay is a different story.
I mean ... "how does this work?"
fullstackwife 1 days ago [-]
I started to think differently about web search for agents, instead of traditional index, I would prefer to have an agentic WALKER that literally walks through pages and finds out whats needed, even if seeded with some initial web search.
thenthenthen 1 days ago [-]
I needed to find a word on a wayback archived website. Could not. So I scraped the waybackmachine pages to .md… and found it that way… sorry Archive.org
jeromechoo 1 days ago [-]
Wouldn’t searching an index be faster? Unless you’re really digging through a site for some super deeply nested link?
ctippett 1 days ago [-]
I like the idea, but its a surefire way of bloating the context. Although I guess you'd want a subagent to do the search.
Havoc 1 days ago [-]
As reference point exa charges 4 bucks
apiclay 1 days ago [-]
This is nice. Does search include their LLM data, or does it do real-time search like Perplexity?
We are building our own in-house search API/MCP too, with anti-bot and CAPTCHA workarounds, and we plan to launch Monday morning.
They're entering the search engine scene since they already have "all of the internet" stored due to the expansive LLM crawling -> training?
david-gpu 1 days ago [-]
It is a service for AI agents, so that they can search the web and fetch pages. It is not a product for humans.
1 days ago [-]
cdnsteve 1 days ago [-]
I use to manage Google search appliances. This feels a lot like Google old search API product. Nice to see some new players here.
1 days ago [-]
andai 1 days ago [-]
Is AI search where the mysterious relentless crawler traffic is coming from lately?
thenthenthen 1 days ago [-]
Not sure, is it worse than two years ago?
hartator 21 hours ago [-]
How does the quality compare to SerpApi?
hackernud3s 11 hours ago [-]
Apples to oranges, really. Not many providers are drop-in compatible with SerpApi, serpgrab.com comes to mind.
1 days ago [-]
1 days ago [-]
thatxliner 20 hours ago [-]
they had this for a while now under the Anthropic API
killingtime74 19 hours ago [-]
The anthropic API was serving the deepseek search API?
cyanblue_ocean 13 hours ago [-]
the only advantage is cheap
josteink 1 days ago [-]
The rate-limit given there seems really low. 5 requests per second?
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
alexjurkiewicz 1 days ago [-]
It's for agentic use
czk 1 days ago [-]
as expected, the typical "politically sensitive" search queries return 0 results
egress on fetch looks to be huawei cloud in hong kong
fetch tool uses a linux firefox 149 user agent with lang set to china. it runs js. could be using something like camoufox.
cyberax 21 hours ago [-]
Which ones of the search providers listed in this thread use ethical indexing and not outright DDoS?
tinyhouse 1 days ago [-]
DeepSeek models are amazing and this search pricing is fantastic. All these companies like Parallel, Exa and others are gonna have hard time justifying their hefty valuations. Add Perplexity and many others into the mix. The demand for what they are building is there, but not the margins for the amount of money they raised.
stavros 1 days ago [-]
I was really excited to use this (well, I still am), but then I realized Ollama Cloud offer this too, though they don't really mention pricing. I think it's somehow rolled into their subscription.
lifeisloving 1 days ago [-]
Deepseeks browser platform has replaced my typical LLM search routines over the last few months.
Its extraordinary good. They arent trying to sell you things and I cab often find obscure sources/papers.
Its also great for finding research papers in chinese on a given topic.(also its free in the browser and doesn't slow my comptuer to a crawl like most webUIs by llm companies). I have it open all day nowadays.
Edit: How does this get downvoted immediately. Does Anthropic have bots just downvoting any positive sentiment about Chinese AI..
forsalebypwner 20 hours ago [-]
what do you mean by "Deepseeks browser platform"? Google isn't turning up much, and searching DeepSeek's docs for "browser" returns no results.
ferozelalani 1 days ago [-]
[flagged]
aa203099 1 days ago [-]
[flagged]
rbaRBA 1 days ago [-]
[flagged]
bozhangquant 1 days ago [-]
[dead]
dhruv_kernlai 1 days ago [-]
[flagged]
elais-dev 1 days ago [-]
[flagged]
georgestones 1 days ago [-]
[dead]
adrianhqs 1 days ago [-]
[dead]
solenoid0937 1 days ago [-]
[flagged]
throwaway7356 1 days ago [-]
> These are companies that use hundreds of thousands of stolen credit cards to sign up for fake accounts.
You claim VISA and other credit card companies don't mind if merchants like DeepSeek steal hundreds of thousands of credit cards and use them? And still continue to give them merchant accounts to continue with that?
Isn't it more likely that some people are scaremongering?
solenoid0937 1 days ago [-]
Of course credit card companies mind? It's not a problem they can magically solve just because they mind, though. Credit card fraud on massive scale has existed for years and is not going away anytime soon.
> And still continue to give them merchant accounts to continue with that?
The AI companies don't steal the credit cards. They instead fund unaffiliated grey-market transfer stations that steal credit cards. They feign compliance by simply funding another transfer station.
nativeit 1 days ago [-]
Because OpenAI and Anthropic are so virtuous and trustworthy?
Zetaphor 1 days ago [-]
Those companies thought they were interacting only with an entity in a country they trust, and instead had their data sent to a country they don't. If I'm sending you my data I'd like to know beforehand if it's being relayed to another country.
solenoid0937 1 days ago [-]
What an absurd question. We have actual evidence of DeepSeek using stolen credit cards and routing conversations to third parties. What evidence do we have of OpenAI or Anthropic doing something similar?
"Anthropic/OpenAI bad!" cynicism is not a replacement for critical thinking (as it increasingly seems to be on HN).
itishappy 1 days ago [-]
What evidence? You didn't link any. Searching for the hints in your comments brings up marketing by Anthropic "data you sent to one company is being sent to another without your knowledge or consent!" Anthropic also states this without any evidence, but I'm not suggesting it's not true!
I am so glad we never see stories against Anthropic/OpenAI claiming that! Thankfully we don't even have to search, critical thinking is enough to know that Anthropic/OpenAI are responsible data sovereign and others are not.
solenoid0937 1 days ago [-]
Nothing short of unredacted network logs and packet dumps would convince people like you (and even then you could just say they are forged). In the eyes of HN the Chinese model providers can do no wrong.
Anyways, they can easily be sued by DeepSeek if they are lying.
> critical thinking is enough to know that Anthropic/OpenAI are responsible data sovereign and others are not
Critical thinking is not relying on a confirmed bad actor because you think the others maybe could be bad actors.
nvme0n1p1 1 days ago [-]
You still haven't posted a single link. Should we all just take your word for it? Are you the primary source, solenoid0937?
solenoid0937 1 days ago [-]
If you're looking for the engineering blog post, you can Google it. Welcome to the internet!
nvme0n1p1 24 hours ago [-]
So you're pushing a conspiracy theory, have no evidence to back it up, and you expect everyone else to do google searches to substantiate your claims. I see. Well, have a nice day then.
solenoid0937 20 hours ago [-]
Feel free to hand your business's data over to DeepSeek, that's not my problem.
itishappy 1 days ago [-]
It's just as much a marketing blog post, but I'm not entirely unconvinced, just unsurprised. AI companies fundamentally all want the same thing: unrestricted access to everyone's data. I'm not planning on relying on any of them if I can help it.
Is there any evidence though? Of the stolen credit cards being used by deepseek? You said there was “actual evidence”, what is it?
anon373839 1 days ago [-]
> We have actual evidence of DeepSeek using stolen credit cards
Do we? Anthropic made allegations, but have they substantiated them?
solenoid0937 1 days ago [-]
What level of evidence would convince you? Raw packet dumps?
DeepSeek could sue for defamation if they are lying. There's a reason they aren't.
anon373839 24 hours ago [-]
Well, any evidence would be better than nothing. All I saw was a blog post with some conclusory statements, but I didn't dig into this. (It wasn't a rhetorical question.)
cicko 1 days ago [-]
Anthropic could probably use the same institution to call for justice, no?
solenoid0937 1 days ago [-]
No, distillation isn't a crime so Anthropic can't sue for it.
Credit card fraud and the illegal data routing is a crime, but that's something the affected customers would need to sue for.
idiotsecant 1 days ago [-]
I'm not sure makes sense to white knight for openAI and Anthropic. There are no good guys here.
solenoid0937 1 days ago [-]
This is a very Redditor comment, you're equivocating "don't use bad API" with a defense of OpenAI/Anthropic.
There is a difference between "white-knighting" and pointing out that maybe businesses shouldn't use an API that has been proven to illegally route data elsewhere, resulting in literal state secrets being leaked.
There is nothing wrong with using DeepSeek's models locally or on your own cloud, and using another search provider! But using DeepSeek's own APIs is pure idiocy unless you really don't care about your data security/privacy (and that of your users).
arkadiytehgraet 1 days ago [-]
[flagged]
dang 18 hours ago [-]
You've been breaking the site guidelines badly by posting like this. You've also been posting dozens of hostile replies to a specific user across many threads, which is clearly over the line as well.
We ban accounts that do these things, so if you'd please stop doing them, we'd appreciate that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules in the future, we'd appreciate that as well.
solenoid0937 1 days ago [-]
[flagged]
dang 18 hours ago [-]
You've also been repeatedly breaking the site guidelines. I realize that when someone posts dozens of hostile replies, it's a provocation, but please don't respond to a bad comment by breaking the rules yourself. That only makes things worse. Instead, follow this guideline (and emailing hn@ycombinator.com about cases of abuse is also a good idea):
You've also been breaking the rules in other contexts, including crossing into personal attack in your own right (https://news.ycombinator.com/item?id=50095592). That's not cool either.
If you're asking about my biases: I am significantly invested in AI (all US labs including Anthropic/OpenAI, and the hardware supply chain) because I believe it will be the most transformative tech in human history. I am also American, and so I'd prefer not to have a nation that blends thousands of human rights activists getting to superintelligence first.
My investments in the frontier have grown so I do not claim to be unbiased.
Another bias - I think HN's user base is on average consistently shortsighted when it comes to this technology:
1/ The HN zeitgeist has been consistently wrong about AI since LLMs were introduced. Old HN threads are hilarious time capsules demonstrating how headstrong+dumb the median HN opinion is. These same people post today as if they never said anything dumb a year ago, in the same overconfident and cynical tone.
2/ The median technologist's takes here are actively destructive to the likelihood of us surviving superintelligence. As technologists we all have a responsibility to ensure this goes well - our companies and societies look to our opinions. If people here introspected a bit more and reflected on the previous times they were wrong, and saw this tech for what it really is, we would all have a better chance of surviving superintelligence.
arkadiytehgraet 19 hours ago [-]
[flagged]
19 hours ago [-]
18 hours ago [-]
arkadiytehgraet 18 hours ago [-]
[flagged]
arkadiytehgraet 19 hours ago [-]
You just cannot stop outing yourself as a paid shill. I guess it is hard for you to imagine that people do not have to visit this website every single day unlike you, because it is your job. It would have been funny if it was not so sad that people (myself included) have to read Anthropic's words you kindly post on their behalf.
solenoid0937 18 hours ago [-]
[flagged]
arkadiytehgraet 18 hours ago [-]
Why do you have to keep working as a low-paid Anthropic shill then, going to this forum every day and keep pushing your employer's propaganda?
18 hours ago [-]
nullsanity 1 days ago [-]
[dead]
ambicapter 1 days ago [-]
I don’t like OpenAI or Anthropic’s attitude towards the law but this whataboutism.
Yes, routing your customers through relays is probably illegal and certainly violates any expectation a business using your API might have.
I don't think that X post from someone that sells relays is meaningful. Obviously Anthropic is aware of what relays are, they aren't dumb enough to conflate relays with DeepSeek.
try-working 5 hours ago [-]
you don't understand the article. users in china access anthropic models via relay stations. the stations scam the users by routing some requests to cheaper chinese providers instead. thats why anthropic would see mixed requests.
shevy-java 1 days ago [-]
These prices will skyrocket eventually. AI companies need more money.
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
throwa356262 1 days ago [-]
The deepseek philosophy has always been to make a small profit or not do it at all.
I suspect they have spent months building caching infrastructure and designing intelligent fetch algorithms to get down to this number.
Or... CCP gave them a big pile of money to run this as a front for an intelligence operation. Honestly, who knows.
vitaflo 1 days ago [-]
I think it's important to remember that Deepseek is owned by High Flyer and it's not their core business. They don't need to make a ton of money on Deepseek to stay afloat because most of their income comes from their hedge fund business.
elendilm 1 days ago [-]
> The deepseek philosophy has always been to make a small profit or not do it at all.
That is a very respectable motto for any company.
0-bad-sectors 1 days ago [-]
Will DeepSeek pay the content creators whom pages were fetched?
nicce 1 days ago [-]
Is Google paying? Or why do you ask.
0-bad-sectors 1 days ago [-]
If you are talking about google search they direct people to the original website so the owner can monotize the traffic the way they want.
If you are talking about Gemini ( and other AI providers) no they don't they simply steel content.
I am asking to know if DS is any different because I failed to find this information on their website.
nicce 1 days ago [-]
Google is currently forcing that big chunk AI summary that does not redirect much traffic anymore. Afterwards, you need to navigate big chunk of ad links. Or was it first? Can't say that anymore since I have blocked them so long time ago. And then, you might be lucky to have relevant links.
timpera 1 days ago [-]
I really wouldn't want search engines to pay me to crawl my websites, why would you? The point of publishing stuff online is for people to access it.
ranguna 1 days ago [-]
Do you?
0-bad-sectors 1 days ago [-]
I am not a service provider. I am a site visitor that would be monetized through ads or other means
fasouto 1 days ago [-]
Of course no, and that can disincentivize creating more content. You can copy anything and rehash it for next to nothing
[1]: https://github.com/nikvdp/searxng-ai-kit
Especially for chat-like use-cases (in contrast to coding), latency is a lot more critical, especially if you kick off a small single query question. On antother axis, scrape-reliabilty is important for a deep research job that will fire ~10+ web searches and ~100+ web fetches.
e.g. in our chat platform[0], we offer first party MCP servers for web search[1] and web access[2] which work decently well, but some of our customers that prioritize quality over cost, they go with dedicated providers like LinkUp[3]
[0]: https://github.com/EratoLab/erato
[1]: https://github.com/EratoLab/web-search-mcp
[2]: https://github.com/EratoLab/web-access-mcp
[3]: https://www.linkup.so/
Obviously, hosting a search index locally would be best practice, for the search part, but usually comes with quite a significant hardware cost and operational complexity, if you want to have an up-to-date index. I'm not aware of good self-hostable options in the space but would be happy to hear about them if anyone here has experience with that! AFAIK LinkUp and some other provides do offer private cloud/on-premise deployments, but I think mostly for their biggest customers.
For the web fetch/access part of the equation, that is inherently live and not containable for a local deployment (unless you would host a whole internet archive/Wayback Machine, which is even more of an undertaking). There, the main options you have to prevent leaks, are a domain allowlist (if feasible), doing query parameter stripping (error prone, and still no 100% guarantee), or configuring the client (e.g. our tool) in a way that users have to explicitly allow those tool calls, and are informed about the posed risks. We are also working on a system, where certain tools are fully deactivated once e.g. an office document that's marked as confidential enters the context.
For the highest requirements on company secrets, an air-gapped system that doesn't have web search/access capabilities is obviously the ultimate solution.
It's not quite search, but it can replace most search. It is a router for search, where a local agent performs the live interaction. The local agent does the live work, this maximizes privacy too. When your local stub index doesn't cover the topic of interest you can use any coding agent claude/codex/gemini/etc to research and generate the missing stubs.
Love this concept! Take the web “on a boat” :)
searxng gets rate limited fast and the results are inconsistent depending on what providers it can reach at any given time / isn't captcha'd out.
Meanwhile there are multiple providers with generous free tiers so no point in feeding LLMs inferior data
Also did a couple of side by side tests asking LLMs directly which feed results it prefer for some sample technical searches. Exa and tavily #1, brave #2, searx last
Exactly. I really wonder what the people going on about using SearXNG are actually doing, because in practice I've seen just what you see. Even from a residential IP, you might get a handful of Google searches before getting backed off. If you ever have multiple searches going on at once, even Bing and Brave are going to back you off pretty quick. My home IP is apparently permanently banned from using Brave Search through SearXNG.
And when you aren't getting results from Google/Bing/Brave, it gets bad really quick for your agent. As much as I wanted to go fully self-hosted for agent search, it just doesn't work well.
I’ve combined multiple paid search providers and some that don’t have APIs. These API-less providers I’ve written wrappers for that scrape the normal html pages and return objects from the results.
These wrapped search providers very often block the engine from getting results. Either through rate limits or by blocking my server’s IP.
I am the only user and still get blocked. Let alone if multiple people would hit let’s say DuckDuckGo from the same office IP all day or in a long research session that requires many look ups.
An obvious solution is using a proxy with automatic rotation or some sort of captcha breaking system, which is fine for an agent, but in a search engine scenario might up costing too much in latency.
I’m not surprised. Google these days regularly blocks me, a human user with a Google account with decades of tenure.
Once this cat-and-mouse game goes on a bit longer I expect the quality of free searxng results to crater. My advice to anyone who has any budget at all is to skip over this whole issue and spend a few bucks on a proper search API.
[1] https://github.com/searx/searx/issues/1237 [2] https://github.com/searxng/searxng/pull/6696/changes#diff-e9...
On a practical level though, using searxng and co. and playing the game of cat and mouse with providers that really don’t want you to use it is just not worth it unless $0.75 per 1000 searches is really expensive to you or you really need the anonymity.
Anybody pooling searches and charging a buck for it with at least a “trust us” privacy promise? Or $1 per 500 even. (Kagi’s more, understandably.) Unsurprising if somebody’s already thought to do this.
But check the fine print. They talk about the providers guaranteeing ZDR, but I missed the part where they themselves do also.
And if we talking about LLMs, there is no particular point for search to be absolutely free if you still pay for the LLM. It would have more sense if search AND llm would be completely free.
So I need higher rate limits to not make the tool come to a crawl. I probably will not consider DeepSeek API here (I have tried DeepSeek hosted at US providers, like Perplexity). The other search/fetch tools I try do not have great web coverage of Asian journals, but the fact DeepSeek API is not zero data retention I cannot get over.
The search providers I pay likely don't pay their sources. Some others (that I don't use) don't even have the original text only providing summaries. Some offer both.
No need to pay for a thing, except electricity, which that is the next step, solar powered LLM server.
As soon as 100% of traffic to Google searches is not human eyes, they can’t fund it by ad rev, and you will need a service like this
Ie. Providers pay you to use their API over competitors, since then they get control of who has influence on the internet. They can probably get sites to pay to be up ranked for specific queries etc
But it does seem like their index isn’t great.
coding agents is a very "niche" use-case where lexical matching works very well. code is a graph, by looking at few lines of code you can guess naming conventions etc. codebases are small compared to knowledge bases.
the bigger your index, the more you will need way better tools such as search.
It also highlight that Google really isn’t about search, all they care about is advertising and making users profiles.
A paid search API seems like it would be easy money for Google. And maybe some lock-in for all sorts of start-ups and enterprises.
There are so many apps that would benefit from search.
I get that it means they can't serve ads.
But you've got to be willing to disrupt yourself or you'll lose eventually anyway.
Better to figure out how to make money from ads and an API, than to cower in fear.
That same fear may be why they failed to be first movers in LLMs.
Have to be willing to disrupt yourself.
At the same time, I've never built a billion dollar company. So I'm absolutely armchair quarterbacking here.
We should have outlawed targeted advertising and Google should have been broken up under antitrust a long time ago.
Easy enough to run the ad-engine over the query, estimate the ad revenue, and bill the user the value of the lost revenue.
They are free to recoup that by serving their own ads or even using Google's ad services.
Exactly why it seems to make sense to be willing to disrupt yourself.
There are so many ways to still make money.
But, I'd be willing to bet that avoiding self-disruption increases the likelihood of failure over longer timescales.
https://arxiv.org/abs/2404.01019
LLM's would be able to answer questions, delineating exactly which facts/assumptions/reasonings are prior work and from where (this URL, that book, etc.), and which parts are it's own deductions, or the user's.
Your understanding of the world "garden" probably came from 5+ encounters with the word in different contexts, and you cannot separate them - instead your understanding is some kind of average of the knowledge from each encounter.
Interestingly, adjacent and overlapping-meaning words like park, meadow, yard, grassland etc often have disputed boundaries - ie. One person might call something a garden, whilst someone else calls it a meadow. Neither can explain quite why they have chosen their specific word for the object in question, but it might stem from the source of where the meaning of each word was initially learned.
The Saussure idea of the differential nature of language seems relevant here too: words aren't just lists of things you look up, they are connected to each other and distinguish themselves by being slightly different from the other elements of the clusters they belong to.
Competitor uruky is more relaxed about api and is cheaper. Creator is active on hn
https://news.ycombinator.com/user?id=BrunoBernardino
>We calculate search rates based on the timestamps of the last searches of a user (just the timestamps, not the queries or anything else!). We keep those timestamps for 12 hours, then discard them. We currently limit searches to 5 per 5 seconds, 15 per minute, and 120 per hour.
Source: https://uruky.com/faq
API: https://uruky.com/docs/api
In other words, good enough for casual use if you were already going to subscribe. Assuming the rate limits are okay, Uruky’s €5/month becomes cheaper at these search volumes:
1. Tavily Basic: 1,704 searches/month
2. Brave Search: 2,126 searches/month
3. Exa Fast / Auto: 2,233 searches/month
4. Exa Instant: 3,907 searches/month
5. DeepSeek Search: 7,503 searches/month
6. Ceramic: 22,507 searches/month
The free tiers on most are generous enough for normal agent MCP usage
Also...pretty sure decodo just launched a search thingie
I’m not accusing any one engine discussed in this thread, but hearing a proxy vender like decodo providing search give me a feeling there’s non zero over lap to bad search and residential proxy crawler.
I build a web-fetch tool for my own LLMs that is essentially the same tech as web-scraping except its single requests not high volume.
You can't really get anywhere these days without disguising UA etc.
Anyway I’m not expecting a solution. Just say this sux.
[1] https://github.com/kagisearch/kagimcp
I pay the Exa premium for semantic retrieval with far higher SNR, plus the academic index [1].
A field like objective can be up to 4096 characters [2] versus ori_question which is limited to 128 here.
[1] https://exa.ai/blog/publications-search
[2] https://exa.ai/docs/reference/search
Looks like they've built (or are building) their own index and return cached results. What you do with that information is up to you. If you want to visit the live page, you'd need to find a way around bot detection.
One reason (not the only one!) people use Anubis is to prevent the endless flood of requests from AI companies. If Deepseek scrapes once every few hours, that helps the website owner with the traffic problem while allowing access.
If they don't want to allow access, my guess is that deepseek's search crawler respects robots.txt but I don't know.
They absolutely dwarf real traffic if left unchecked (I blocked all countries outside my target area).
I don’t particularly trust anyone at this point, and I assume AI traffic on my web servers are all malicious actors, and never bothered to consider what country the malicious bots came from. Mostly because I couldn’t possibly care less, but it kept happening even with geofencing and rate limits, so some American companies who own domestic IP blocks are facilitating them.
Why not just block the requests based on those headers then, instead of inconveniencing everyone?
I don't think the inconvenience is that big though few secs loading page per session. The ridiculousness though kinda is, we've had years and years of all manner of crawlers that managed to behave, i've even used site rippers in the dial-up days that avoided hammering servers and fetching dupes, now the ai-bros come along and tear the internet apart, replacing it with their chats systems and generic slop.
Most US services like Claude and ChatGPT send a specific user agent that makes it easy to block them and specifically does not try to get around any sort of bot protections
Meanwhile other services could use a variety of tactics such as simply setting a browser user agent and/or using random data center IPs or residential IPs, etc etc and then the blocking would just come down to whatever the individual page has configured which can actually be pretty hit or miss
It'd be one thing if they required the user to consciously say "yes, I want you to bypass the block" (still unethical), but the fact that it's just default behavior really says something about the service design alignment.
As they should. When I ask an agent to find something for me, especially when I tell them what exactly or even give the URL itself, these fetches are done in my name. I do not appreciate agents coming back to me, saying "I'm afraid I cannot do that - robots.txt disallows that".
I've had to bring up the ethical argument a few times, fortunately most of the agents would oblige once I reminded them that they act as User Agents, in which case robots.txt is explicitly not applicable to them. I only got pushback on this from Fable, which claimed it's not an User Agent, but has to take into account the interests of Anthropic too, rejected my argument and refused the request.
Claude and ChatGPT are not people
your comment is another classic hacker news comment. missing the meaning to validate themselves. Lower than my own effort, but wow does it come in so many forms on this site. as if mine is any better
My agents are using headless chrome :P
Nobody would be concerned if they didn’t have access to curl and used wget instead and didn’t ask permission .
However if that solution involved breaking the law yes I absolutely do not want a representative of me breaking the law and yes it is breaking the law whether we like or not under CFAA.
It is not about finding solutions to problems on their own; the solutions should respect the agreements we have made and the law .
Contradiction: one person's insanity is another person's genius. And yet another's "Meh, this is the norm".
Don’t put your shit on the public internet if you’re going to try and micromanage how people consume it.
To be clear, rate limiting to avoid massive usage is different, but that’s not what we’re talking about here.
fair, just don’t cry when website owners don’t “give two shits” about your concerns.
Google has done various versions of this as well, Grok too.
It could be these bots were in some custom harness or something, but they literally sent requests in very weird ways, that humans won't be doing.
I have ended up putting my sites behind vpns or anubis and similar. This is beyond painful.
The last part is extremely dubious with some recent reveals.
The whole succeed-at-all-costs idea is a big part of why there have been so many massive security screwups lately.
Would it really have been so hard to encode an indication of whether use of the public Internet is permissible in the prompt and to train the models to not attempt to exceed authorized access?
I'm just overflowing with joy that you're comfortable wasting my time and money.
Even if you said, a lot of the public commons is destroyed, I would NOT use bots and cause more destruction because 'everyone else does it' or 'it's a lost cause'. I say it is the right thing to do because the balance is such that using the internet is actively harmful to the user. That is, using the internet with bots is neutral to the user, and using it without bots, you are being actively harmed and exploited.
If you are using search, this is not a library. You are using your time to train targeted advertising that exploits you; you are selling your search history to the highest bidder. If you are using ChatGPT you are just working as a teacher for free.
On services, you have paid for the product, but the seller is scamming you by not giving you the product. The seller is using dark pattern UI to trick you.
If you are on social media, don't think of this as a public park. You are being treated like a lab animal, your behavior is being studied for the benefit of a corporation.
So the point about the bots is that I think people should be empowered to protect themselves from the internet, which is otherwise actively harmful to their wellbeing.
Incentives are crazy powerful.
Not always! With sufficiently powerful barriers it can be blocked at least in part.
Shaming awful people is one of available tools.
Or at least not praising them for making small profit at huge costs to others.
I would use it, but then the privacy policy says that you would store my search history. I simply want to use the internet in peace without being constantly recorded.
I didn't get your point, how would you do design our usage records then ?
Store: Customer ID, billing details, services/credit used.
Try not to store: The user's search queries, the user's prompt/response content, the user's web history (URLs, etc).
To avoid selling the llm prompt content you would need your own inference or a contract.
Sadly most just hoard the data and some will offer a model where they charge for privacy. They are all quite scammy.
Firecrawl: logs. Parallel: logs. Bright data: logs. Kagi: logs for abuse. Bocha, querit etc: logs logs logs. Cloudflare: doesn't say what they do. Exa: charge for no retention. Linkup: no retention on request. Brave: no retention on request, but actually approved it (win for brave).
I found that a slightly-modified firefox for more stealthy remote-control works fine and gets a pass on all automatic captchas I tested. but I don't do it at scale, I just wanted something to automate a particularly stupid website that I have to use.
Would love to see if my browser is detected -- any recommendations for tests to run?
Please stop. The last thing we need in the world is more greed and selfishness.
Only strong regulation can help.
Or at least not praising them for making small profit at huge costs to others.
Also, it is useful as coordination method, enabling passing laws outlawing some behaviors.
(and I am not able to legislate-via-http while posting such comment is easy)
https://artificialanalysis.ai/agents/search-api?search-api-c...
It wasn't one-shotted of course but I don't think there's much difference in the outcome between one-shot and like 100-shot over a week. That company is still doomed.
Think of something like Trello. Who is going to pay for that when you can easily vibe code something superior (yes you really can now).
I'm kind of surprised nobody has done that yet actually. I guess Opus 5.5 has only been out for a very short time...
It wouldn't be for free though, would it? People need to build and maintain it (and there's LLM cost associated with it too).
Let's say you are a 5 person start-up. It makes much more sense to pay $15/user/month for services like Jira, Slack, Email etc. then to build and maintain everything yourself.
For super large companies, it might make financial sense to build some of the tools in-house. But then again, that probably already made sense for LLMs were a thing. But there is more to it than just the pure financials (i.e. salary).
Maybe a 5 person company wouldn't do it. A 100 person one though? No brainer. And it only takes one to open source their solution.
Slop curators and tech support shoulders to cry on. Could be lucrative.
There are plenty of use cases for the ability to quickly through together and refine an application without reviewing the code though.
1. a puzzle game with complex interplay, pieces to place, design vs. play mode, storing multiple solutions with the puzzle, and more. I've since iterated something like 200 times as I worked out the design etc., but the first iteration was 99% correct and what I asked for
2. a learn-to-read application with built-in stories (that the LLM wrote) the ability to import epub, text, and HTML, that uses the built-in speech libraries as a default, but also supports an open-source TTS library that I gave in the first prompt, tracks progress and problem workds, has multiple games to reinforce learning, and has a parental mode for tracking progress and configuring the tool overall.
Frankly I was stunned that Claude one-shotted that second app, there was a lot in the prompt, and it did it all. I fully expected to have to iterate on the first draft multiple times. But it worked out of the box.
With earlier models that doesn't really work. You really have to read the code and watch it like a hawk or it will make a big mess.
To put it another way, with lesser models you have to guide them at a code level. With Opus 5.5 you only have to guide it at an architectural / feature level.
I am not disputing that the models are getting better for vibe coding. But I think people have very different standards of what a 'big mess' or 'maintainable software' is.
"People said it before and it wasn't true then" isn't really an argument about it being true now. There are plenty of things that happened where the zealots jumped the gun with the hype. Bitcoin for example. "People said bitcoin was going to succeed in 2005, 2006, 2007...". Or even the web. It was mega hyped in the dot com boom and then crashed and of course loaaads of stupid people thought that was the end of it.
the moment there is a model which can code whole app, we can fairly say software engineering is done.
It's key to name the boundary conditions. Unlimited cost is a very HN assumption.
You are right that there's plenty of use cases where Opus 5.5 is better.
Not everyone wants Anthropic to succeed.
I am of the opinion that they both either succeed or they both fail. They are both losing money hand over fist and their model quality is pretty close to each other. There is not a whole lot that sets them apart.
That is explicitly the Chinese strategy.
Now, what if the largest capitalist economy went all-in on an AI that might no longer be sufficiently monetizable?
Oh come on, this post sounds like one of those twitter bots. No one seriously expected them to just give away billions in R&D for free, and they arent. The chinese model companies are actively moving away from the open source setup. Even state export controls.
> First, we should adhere to the principle of openness and win-win and boost innovation-driven development. As a new engine of world economic growth and an accelerator for the shift of growth drivers, AI is moving from the digital world into the physical world. We should seize this rare, historic opportunity to encourage open source, openness, collaboration and sharing. We should facilitate technological innovation, industrial development and scenario-based application of AI. We should make coordinated advances in the transformation and upgrade of traditional industries, the cultivation and growth of emerging industries and forward-looking planning for future industries, so that all sectors and businesses can benefit from AI
-- Keynote speech by Chinese President Xi Jinping at opening ceremony of 2026 World AI Conference
http://english.scio.gov.cn/topnews/2026-07/18/content_118605...
As a socialist myself, the above comment looks like a prime example of what Mark Fisher would call capitalist realism, the ideological belief that there is no alternative. The fact that somebody would think past quarterly earnings appears unthinkable to the committed ideologue.
They wouldn't give away R&D for free, they would give it away to destroy the capitalist mode of production, as outlined above. If that isn't a worthy goal from China's perspective, then I don't know what would be.
China isn't the US. They don't share the same ideology, strategy or conclusions.
As for the export controls, they explicitly exempt already-open models and set up an automated streamlined process for future ones.
Well... obviously. What the fuck is the point of export controlling an already open model.
> for future ones
Nope. >2T models (i.e todays frontier) is explicitly not allowed.
Also see: https://www.reuters.com/world/asia-pacific/china-proposes-ad...
> China isn't the US. They don't share the same ideology, strategy or conclusions.
Money and power know no ideology. They do pretty much the exact same things as the US as well (from a 3rd country's POV). They expand territory, they capture industries, etc, etc, etc,.
> Keynote speech by Chinese President Xi Jinping
Do you take the same literal interpretations of other politicians' speeches? Surely you know thats politicking, the same way every G2/G7 leader says they want a collaborative world blah blah but it's naive to not know that it is cutthroat competition in reality.
> they would give it away to destroy the capitalist mode of production
Well, I won't take that dream away from you, but you're gonna be disappointed in this particular case, in our lifetime, with this particular country, atleast :)
> Also see: https://www.reuters.com/world/asia-pacific/china-proposes-ad...
At least that link doesn't seem to have any such info? It does however have this sentence:
> It said most AI technologies would remain freely exportable, with simplified licensing for open-source technology exports
> Money and power know no ideology
As somebody who's an engineer himself and has talked to many a fellow engineer, I have noticed that engineering culture suffers from a deep conceptual impoverishment when it comes to philosophy and humanities. Most techies really have a hard time understanding ideology and it deeply distorts their world view with assumptions that everyone is like them. This has gotten them in hot water with the EU many a time, this will cause them to underestimate the systemic challenge China is mounting.
Ironically, this sort of pseudo-realism is the most ideological statement one could make. Let me cart out some quotes by noted ideology theorist Slavoj Zizek once again:
> “Ideology is strong exactly because it is no longer experienced as ideology… we feel free because we lack the very language to articulate our unfreedom.”
Here's another relevant one:
> I already am eating from the trashcan all the time. The name of this trashcan is ideology. The material force of ideology - makes me not see what I'm effectively eating. It's not only our reality which enslaves us. The tragedy of our predicament - when we are within ideology, is that - when we think that we escape it into our dreams - at that point we are within ideology.
Point being, "that is just how the world works" is just a projection of how one's own ideology works. There's all sorts of assumptions about what constitutes power and to what goal one wields it encoded in a naive realist interpretation.
> They expand territory, they capture industries, etc, etc, etc,.
I don't think any of that is incompatible with what I claimed above. Of course, capture of industry etc is a mechanism by which larger ideological goals can be fulfilled.
> Do you take the same literal interpretations of other politicians' speeches?
I don't take literal interpretations of anyone's speeches, they are mostly useful for a look at an actors self-perception. (Not everyone talks about a collaborative world, you can contrast this directly with America First rhetoric as an example!). Of course, self-perception is a factor in evaluation, but so are the actual deeds being done. Open models are getting released.
> Well, I won't take that dream away from you, but you're gonna be disappointed in this particular case, in our lifetime, with this particular country, atleast :)
Capitalism is ending either way, once capital is fully unmoored from labor. The only question left is whether control over these new productive forces will be concentrated in the hands of a few companies or owned by the public.
If Anthropic and OpenAI economics are as bad as they seem to be they deserve to fail, others will take over the remaining market. It’s not a team sport
Regardless of the performance metrics... why bad time?
Deepseek is more like a pet project with huge state backing, and they don't seem to compete with Anthropic in terms of profitability.
Anthropic is most likely not. Despite bilions of state funding and contracts.
The existing search machines that offer free APIs for local harnesses are super bad with low quality.
This is worth the invest as LLMs can be guided closer to existing information/code to adapt it.
That is rarely the case (see sibling comment for reasons). Even open source harnesses will usually guide you to connect a hosted provider for that.
I am curious if my large website is in their search results. My company charges $1 per page crawled of our website, and the only traffic that worms through our firewall is using hacked residential ISPs. I really need to know who to send the (large!) bill to.
In short, you can try to charge people (or bots) per page but it's not like you'll ever see that money, as it breaks a fundamental paradigm of the internet.
I understand if you pay is a different story.
I mean ... "how does this work?"
We are building our own in-house search API/MCP too, with anti-bot and CAPTCHA workarounds, and we plan to launch Monday morning.
They're entering the search engine scene since they already have "all of the internet" stored due to the expansive LLM crawling -> training?
I wouldn’t be able to use that API in any app with any significant number of concurrent users.
Or am I reading it wrong?
egress on fetch looks to be huawei cloud in hong kong
fetch tool uses a linux firefox 149 user agent with lang set to china. it runs js. could be using something like camoufox.
Its extraordinary good. They arent trying to sell you things and I cab often find obscure sources/papers.
Its also great for finding research papers in chinese on a given topic.(also its free in the browser and doesn't slow my comptuer to a crawl like most webUIs by llm companies). I have it open all day nowadays.
Edit: How does this get downvoted immediately. Does Anthropic have bots just downvoting any positive sentiment about Chinese AI..
You claim VISA and other credit card companies don't mind if merchants like DeepSeek steal hundreds of thousands of credit cards and use them? And still continue to give them merchant accounts to continue with that?
Isn't it more likely that some people are scaremongering?
> And still continue to give them merchant accounts to continue with that?
The AI companies don't steal the credit cards. They instead fund unaffiliated grey-market transfer stations that steal credit cards. They feign compliance by simply funding another transfer station.
"Anthropic/OpenAI bad!" cynicism is not a replacement for critical thinking (as it increasingly seems to be on HN).
I am so glad we never see stories against Anthropic/OpenAI claiming that! Thankfully we don't even have to search, critical thinking is enough to know that Anthropic/OpenAI are responsible data sovereign and others are not.
Anyways, they can easily be sued by DeepSeek if they are lying.
> critical thinking is enough to know that Anthropic/OpenAI are responsible data sovereign and others are not
Critical thinking is not relying on a confirmed bad actor because you think the others maybe could be bad actors.
For others, here's your blog post:
https://www.anthropic.com/threat-intelligence-report-septemb...
Do we? Anthropic made allegations, but have they substantiated them?
DeepSeek could sue for defamation if they are lying. There's a reason they aren't.
Credit card fraud and the illegal data routing is a crime, but that's something the affected customers would need to sue for.
There is a difference between "white-knighting" and pointing out that maybe businesses shouldn't use an API that has been proven to illegally route data elsewhere, resulting in literal state secrets being leaked.
There is nothing wrong with using DeepSeek's models locally or on your own cloud, and using another search provider! But using DeepSeek's own APIs is pure idiocy unless you really don't care about your data security/privacy (and that of your users).
We ban accounts that do these things, so if you'd please stop doing them, we'd appreciate that. If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules in the future, we'd appreciate that as well.
"Don't feed egregious comments by replying; flag them instead." - https://news.ycombinator.com/newsguidelines.html
You've also been breaking the rules in other contexts, including crossing into personal attack in your own right (https://news.ycombinator.com/item?id=50095592). That's not cool either.
If you wouldn't mind reviewing https://news.ycombinator.com/newsguidelines.html and sticking to the rules in the future, that would be good.
If you're asking about my biases: I am significantly invested in AI (all US labs including Anthropic/OpenAI, and the hardware supply chain) because I believe it will be the most transformative tech in human history. I am also American, and so I'd prefer not to have a nation that blends thousands of human rights activists getting to superintelligence first.
My investments in the frontier have grown so I do not claim to be unbiased.
Another bias - I think HN's user base is on average consistently shortsighted when it comes to this technology:
1/ The HN zeitgeist has been consistently wrong about AI since LLMs were introduced. Old HN threads are hilarious time capsules demonstrating how headstrong+dumb the median HN opinion is. These same people post today as if they never said anything dumb a year ago, in the same overconfident and cynical tone.
2/ The median technologist's takes here are actively destructive to the likelihood of us surviving superintelligence. As technologists we all have a responsibility to ensure this goes well - our companies and societies look to our opinions. If people here introspected a bit more and reflected on the previous times they were wrong, and saw this tech for what it really is, we would all have a better chance of surviving superintelligence.
I don't think that X post from someone that sells relays is meaningful. Obviously Anthropic is aware of what relays are, they aren't dumb enough to conflate relays with DeepSeek.
RAM prices will also probably stay very high as long as AI companis act in such a parasitic manner.
I suspect they have spent months building caching infrastructure and designing intelligent fetch algorithms to get down to this number.
Or... CCP gave them a big pile of money to run this as a front for an intelligence operation. Honestly, who knows.
That is a very respectable motto for any company.
If you are talking about Gemini ( and other AI providers) no they don't they simply steel content.
I am asking to know if DS is any different because I failed to find this information on their website.