Pumpkin Escobar

First a caveat/warning - you’ll need a beefy GPU to run larger models, there are some smaller models that perform pretty well.

Adding a medium amount of extra information for you or anyone else that might want to get into running models locally

Tools

• Ollama - great app for downloading/managing/running models locally
• OpenWebUI - A web app that provides a UI like the ChatGPT web app, but can use local models
• continue.dev - A VS Code extension that can use ollama to give a github copilot-like AI assistant running against a local model (can also connect to Anthropic Claude, etc…)

Models

If you look at https://ollama.com/library?sort=featured you can see models

Model size is measured by parameter count. Generally higher parameter models are better (more “smart”, more accurate) but it’s very challenging/slow to run anything over 25b parameters on consumer GPUs. I tend to find 8-13b parameter models are a sort of sweet spot, the 1-4b parameter models are meant more for really low power devices, they’ll give you OK results for simple requests and summarizing, but they’re not going to wow you.

If you look at the ‘tags’ for the models listed below, you’ll see things like 8b-instruct-q8_0 or 8b-instruct-q4_0. The q part refers to quantization, or shrinking/compressing a model and the number after that is roughly how aggressively it was compressed. Note the size of each tag and how the size reduces as the quantization gets more aggressive (smaller numbers). You can roughly think of this size number as “how much video ram do I need to run this model”. For me, I try to aim for q8 models, fp16 if they can run in my GPU. I wouldn’t try to use anything below q4 quantization, there seems to be a lot of quality loss below q4. Models can run partially or even fully on a CPU but that’s much slower. Ollama doesn’t yet support these new NPUs found in new laptops/processors, but work is happening there.

• Llama 3.1 - The 8b instruct model is pretty good, decent speed and good quality. This is a good “default” model to use
• Llama 3.2 - This model was just released yesterday. I’m only seeing the 1b and 3b models right now. They’ve changed the 8b model to 11b, I’m assuming the 11b model is going to be my new goto when it’s available.
• Deepseek Coder v2 - A great coding assistant model
• Command-r - This is a more niche model, mainly useful for RAG. It’s only available in a 35b parameter model, so not all that feasible to run locally
• Mistral small - A really good model, in the ballpark of Llama. I haven’t had quite as much luck with this as with Llama but it is good and I just saw that a new version was released 8 days ago, will need to check it out again

It’s a good thing that real open source models are getting good enough to compete with or exceed OpenAI.

It has been on my list to figure out how to move to forgejo, need to do it soon before the migration process breaks or gets awful.

FWIW they didn’t merge it, they closed the PR without merging, link to line that still exists on master.

The recent comments are from the announcement of the ladybird browser project which is forked from some browser code from Serenity OS, I guess people are digging into who wrote the code.

Not arguing that the new comments on the PR are good/bad or anything, just a bit of context.

Been 100% linux for like 6-9 months now, these stories make me thankful for finally making the switch.

I’ve tried to make the switch 3-4 times in the past and was stopped by 2 main things:

• Drivers / Laptops were tough to get set up
• Gaming

The experience was so much better this time and I really have no regrets. I don’t imagine I’ll ever run Windows again outside of a VM

Elon “Nick Cannon” Musk

Rip up the Reddit contract and don’t use that data to train the model. It’s the definition of a garbage in garbage out problem.

Oh no, not titties!

I like that. If there was a site that did like The Razzies for movies but for technology enshitification, I would definitely watch, and probably follow a blog if it was done well

dumbest fucking timeline. A subscription for a feature that requires no infrastructure and is part of the physical thing you just paid $40k for.

Holy shit, .15 euros per page? Why not just run to der Kinkos? I haven’t checked but I imagine it’s cheaper there. I get the convenience of having a printer at home but this is like if every cup of coffee you make at home cost you the Starbucks $8.25.

FWIW his white whale or inspiration is more like the Chinese “we do everything” apps / platforms https://wise.com/us/blog/chinese-payment-app