I want someone to prove his LLM can be as insightful and accurate as paid one.
I mean, you can train a model that’s domain-specific that some commercial provider doesn’t have a proprietary model to address. A model can only store so much information, and you can choose to weight that information towards training on what’s important to you. Or providers may just not offer a model in the field that you want to deal with at all.
But I don’t think that, for random individual user who just wants a general-purpose chatbot, he’s likely going to get better performance out of something self-hosted. Probably it’ll cost more for the hardware, since the local hardware isn’t likely to be saturated and probably will not have shared costs, though you don’t say that cost is something that you care about.
I think that the top reason for wanting to run an LLM model locally is the one you explicitly ruled out: privacy. You aren’t leaking information to someone’s computers.
Some other possible benefits of running locally:
-
Because you can guarantee access to the computational hardware. If my Internet connection goes down, neither does whatever I’m doing with the LLM.
-
Latency isn’t a factor, either from the network or shared computational systems. Right now, I don’t have anything that has much by way of real-time constraints, but I’m confident that applications will exist.
-
A cloud LLM provider can change the terms of their service. I mean, sure, in theory you could set up some kind of contract that locks in a service (though the VMWare customers dealing with Broadcom right now may not feel that that’s the strongest of guarantees). But if I’m running something locally, I can keep it doing so as long as I want, and I know the costs. Lot of certainty there.
-
I don’t have to worry about LLM behavior changing underfoot, either from the service provider fiddling with things or new regulations being passed.
They’re all right, I suppose, but it wasn’t dissatisfaction with search results that caused me to want to use Kagi. Rather, that I wanted to use a search engine that has a sustainable business model that didn’t involve data-mining me or showing me ads.
If Google or whoever offered some kind of comparable commercial “private search” service with a no-log, no-data-mining, no-ad offering, I’d probably sit down and to compare the results, see what I think. I kind of wish Google would do that with YouTube, but alas, they don’t…
Kagi does have a feature where they will let you search the complete Threadiverse that I make use of, since I spend a lot of time here; there isn’t really a fantastic way to accomplish this on Google or another search engine that I’m aware of. They call that their “Fediverse Forums” search lens; that’s probably the Kagi-specific feature that I get the most use out of.
They have other features, like fiddling with the priorities of sites and stuff like that, but I don’t really use that stuff. They do let you customize the output and stuff. You can set up search aliases and stuff, but I can do most of that browser-side in Firefox.
They have the ability to run a variety of LLM models on their hardware, provide that as a service. I have the hardware to run those on my own hardware and have the software set up to do so, so I don’t use that functionality. If I didn’t, I’d probably find some commercial service like them that had a no-log, no-data-mining policy, as it’s more economical to share hardware that one is only using 1% of the time or whatever.
I dunno. They have some sort of free trial thing, if you want to see what their search results are like.