Does anyone run their own AI service?
If so,
Which one?
What hardware?
Do you have a way to access it from outside or from a phone?
And, if you’re open to sharing, what do you do with it?
I know AI-enabling hardware is insanely expensive right now, but some of the smaller (<32GB) more efficient models (think deepseek- or qwen-based) don’t absolutely need specialized hardware to run and perform well enough for many purposes.
I am so aware of the absolutely abysmal privacy practices of basically all AI companies that I’m not comfortable using them to help me with web searches and the such, because there’s no way that all this information won’t be going into a profile. This makes me want to have my own “helper” service.
I have nearly the same setup, except OpenCode for the harness. I run 200k context for anything that doesn’t need vision, or 150k otherwise. Typically get about 30 t/s decode With MTP, starts at close to 800 t/s for prefill and drops to the 200s for a long context. I’d imagine higher decode speeds are possible with some tinkering, but I haven’t gotten around to it.
OpenCode recently got hacked and advises changing API keys fyi.
I get about 45 t/s after some tweaking, getting the thermals under control helped a lot; mainly by reducing the power consumption to 261 W (and downclocking a little bit).
What vision model do you use?