not_IO@lemmy.blahaj.zone to Programmer Humor@programming.devEnglish · 6 个月前might be a form of Jevons Paradoxlemmy.blahaj.zoneimagemessage-square292linkfedilinkarrow-up11.81Karrow-down116file-text
arrow-up11.8Karrow-down1imagemight be a form of Jevons Paradoxlemmy.blahaj.zonenot_IO@lemmy.blahaj.zone to Programmer Humor@programming.devEnglish · 6 个月前message-square292linkfedilinkfile-text
minus-squareVlyn@lemmy.ziplinkfedilinkEnglisharrow-up18arrow-down1·6 个月前With 32 and 64 GB systems I’ve never run out of RAM, so the RAM isn’t the issue at all. Optimization just sucks.
minus-squareSamskara@sh.itjust.workslinkfedilinkarrow-up2·5 个月前Have you ever tried running a decent sized LLM locally?
minus-squareVlyn@lemmy.ziplinkfedilinkEnglisharrow-up2·5 个月前Decent sized for what? Creative writing and roleplay? Plenty, but I try to fit it into my 16 GB VRAM as otherwise it’s too slow for my liking. Coding/complex tasks? No, that would need 128GB and upwards and it would still be awfully slow. Except you use a Mac with unified memory. For image and video generation you’d want to fit it into GPU VRAM again, system RAM would be way too slow.
minus-squareSamskara@sh.itjust.workslinkfedilinkarrow-up1·5 个月前I use a Mac with unified memory, so that distinction slipped my mind.
With 32 and 64 GB systems I’ve never run out of RAM, so the RAM isn’t the issue at all.
Optimization just sucks.
Have you ever tried running a decent sized LLM locally?
Decent sized for what?
Creative writing and roleplay? Plenty, but I try to fit it into my 16 GB VRAM as otherwise it’s too slow for my liking.
Coding/complex tasks? No, that would need 128GB and upwards and it would still be awfully slow. Except you use a Mac with unified memory.
For image and video generation you’d want to fit it into GPU VRAM again, system RAM would be way too slow.
I use a Mac with unified memory, so that distinction slipped my mind.