Only 3.8B parameters according to the paper, so it ought to be quite easy on the hardware as well if they do.

Assuming they already own a PC, if someone buys two 3090 for it they’ll probably also have to upgrade their PSU so that might be worth including in the budget. But it’s definitely a relatively low cost way to get more VRAM, there are people who run 3 or 4 RTX3090 too.

For LLMs it entirely depends on what size models you want to use and how fast you want it to run. Since there’s diminishing returns to increasing model sizes, i.e. a 14B model isn’t twice as good as a 7B model, the best bang for the buck will be achieved with the smallest model you think has acceptable quality. And if you think generation speeds of around 1 token/second are acceptable, you’ll probably get more value for money using partial offloading.

If your answer is “I don’t know what models I want to run” then a second-hand RTX3090 is probably your best bet. If you want to run larger models, building a rig with multiple (used) RTX3090 is probably still the cheapest way to do it.

A static website and Immich

Definitely not cheap, but at least not as bad as having to buy an A100 for €7000 to get 40GB VRAM. I’m hoping second hand GPU prices will plummet after Christmas

You can buy 2x second hand RTX3090 for about the same price of one new RTX4090, though you’ll probably need to get a new PSU as well. Or rent the hardware through runpod.io or similar for around $1/hour. Still a lot of money for most people but it’s not completely unachievable… Spend some time in the local LLM community and 48GB VRAM will start to feel like the bare minimum if you want to use any of the better models :S

There are tons of options for running LLMs locally nowadays, though none come close to GPT4 or Claude 2 etc. One place to start is /c/localllama@sh.itjust.works

Looks really cool, but a video of more than 3 seconds would’ve been nice :)

Static html+css page generated with this: https://github.com/maximtrp/tab

Do you mean that you want to build the docker image on one computer, export it to a different computer where it’s going to run, and there shouldn’t be any traces of the build process on the first computer? Perhaps it’s possible with the –output option… Otherwise you could write a small script which combines the commands for docker build, export to file, delete local image, and clean up the system.

https://github.com/miroslavpejic85/mirotalk might be an option. There’s both a server based version and a p2p version IIRC.

I asked someone about this a few days ago, and they claimed to have over 30000 photos in Nextcloud without issues

I suppose “a few” is quite open to interpretation, but I have 50k photos now so if it can handle 100k without getting sluggish it’ll probably be fine for the foreseeable future.

Does Nextcloud handle large numbers of photos nowadays? IIRC when I was comparing programs some years ago I read that both it and Owncloud struggled when you got to a few 10000s of photos.

Ah, nice.

Btw. perhaps you’d like to add:

build: .

to docker-compose.yml so you can just write “docker-compose build” instead of having to do it with a separate docker command. I would submit a PR for it but I have made a bunch of other changes to that file so it’s probably faster if you do it.

Awesome work! Going to try out koboldcpp right away. Currently running llama.cpp in docker on my workstation because it would be such a mess to get cuda toolkit installed natively…

Out of curiosity, isn’t conda a bit redundant in docker since it already is an isolated environment?