tekBlueprint

I previously wrote an article CI/CD with Gitea for HomeLab, which included some information about a local server for project needs. This server isn’t limited to CI/CD; I also use it to run local AI models.

Prerequisites for local deployment

Spending time and resources on setting up an environment for local models isn’t always necessary, and not everyone wants it. I had several reasons:

1. Privacy – It’s no secret that everything written in public chats remains with the owners of those chats. I think there was an old news story somewhere about users’ conversation contents being leaked publicly. In other words, you no longer control everything you do there.
2. Availability – Blocking is the new reality. Many resources are constantly blocked under various pretexts. Only the truly crazy and courageous can hope for the constant availability of public resources now.
3. Self-education – It’s still interesting to understand how things work from the ground up, rather than just having a ready-made chat window. Yes, this requires personal time and resources.
4. Cost – a debatable metric and depends on usage volume. If they’re high, it might be more cost-effective to have local servers than a subscription.

It’s also worth mentioning that local models will never match their cloud counterparts in quality and speed. This is simply because they require far less-than-consumer-grade hardware. Don’t expect to be able to build the same solution at home as ChatGPT. The full model won’t even be available openly, only stripped-down versions. So, don’t get too excited about the idea, lest you be disappointed later. :)

Affordable Hardware for 2025

There are actually a ton of options here, and it all depends on your budget. I don’t see the point in considering server hardware (A100, H100, H200, L40s, etc.). The cost is very high and requires both the graphics cards themselves and the corresponding servers. Overall, it’s not suitable for home use.

Building a server with regular CPUs and DDR is also pointless; it will result in very low inference speeds. It simply won’t work properly.

The choice depends on your budget. If we consider more “affordable” options for home use, it looks something like this:

NVIDIA RTX PRO™ 6000 Blackwell, Apple Mac Studio M3 Ultra, NVIDIA DGX Spark – for me, they were immediately eliminated. Prices fluctuate depending on the exchange rate, and the numbers run into the millions. But wealthy individuals might want to take a closer look. If you’re still interested in how they perform in AI, there are video reviews, such as this one from Alex Ziskind.

GeForce RTX 3090 – supports NVLink, so they’re often used together. But you need to find cards that haven’t been used in mining. The combined power consumption can be a drawback. Additionally, you’ll need a suitable case with good cooling and a motherboard with two PCIe x8 lanes.

GeForce RTX 4090 – there’s no point in buying one now, as there’s the GeForce RTX 5090. You can probably find one on the used market, but unlike the 3090, it doesn’t have NVLink. This means the maximum memory limit is 24 GiB.

GeForce RTX 5090 is currently the only option with maximum performance. The price is unreasonable, as is the case with all NVIDIA flagship graphics cards. Unfortunately, Huang has long been a monopolist and does whatever he wants.

AMD Ryzen™ AI Max+ 395 is a surprisingly good SoC from AMD. Yes, it’s not up to par with the Apple Mac Studio M3 Ultra, but it’s also about a third of the price. MiniPCs based on it are now plentiful; the base is the same (standard motherboards), only the packaging changes.

There’s also a vast field of experimentation with combining various available hardware into a computing cluster. For example, for distributed computing, you can use vLLM or exo, combining available PCs into a single cluster. However, I should point out right away that the final performance will be greatly affected by the link speed between the PCs. Thunderbolt 4/5 is a good choice. Alternatively, you can distribute isolated agents across PCs rather than sharing a common model between them.

For more exotic options, you could use OCuLink. An external graphics card is connected via NVMe (MiniPCs typically have two slots). However, the cost of this external graphics card and its box will be significantly higher than the MiniPC or laptop itself. It won’t work at full capacity, either.

What I used and what problems I encountered

At first, I used what I had: an old PC graphics card (GeForce 970 @ 4GiB) and an integrated graphics card on my work laptop (RTX 3050 @ 6GiB). But memory issues surfaced almost immediately. At some point, I ran out of patience and bought a GeForce RTX 4090 during its lowest price period (2023), before the price jump. The speed boost was huge, and I used it for everything I did until almost the end of 2025.

The problem with a single graphics card is that it can’t always be used for multiple models simultaneously. You have to either constantly switch, or use smaller models that fit in 24 GiB, but at a cost in quality. A good example is LM Studio. In server mode, it can only use one model and switches between them. One solution was to run models for vectorization and ranking on a work laptop with an RTX 3050, and large LDMs (32GiB) on a PC. The problem of constantly switching between models disappeared.

Over time, the models themselves grew in size, and 24GiB of VRAM wasn’t always enough (especially for LDM). So, around the end of 2025, I decided to get a MiniPC with an Al Max+ 395 128GiB. I also needed a separate server for other tasks, so I could combine business with pleasure. I bought the cheapest one from AliExpress, which was risky, but it arrived intact. However, I strongly suspect it’s not new. With the MiniPC, I can now load models up to 72GiB without MoE or 120GiB+ with MoE. I can immediately say that models with more than 14GiB without MoE will run slowly due to low memory bandwidth. MoE models work quite fast, for example, with gpt-oss-120b-MXFP4. You can read about MoE here – Mixture of Experts (MoE)

Performance testing

I’ll skip everything related to LDM; these are more specific neural networks and are best tested separately. We’ll only discuss the more or less popular LDMs.

For performance testing, I typically use llama.cpp, and the target metric is the number of tokens per second. Generally, you can roughly estimate the final performance based on memory bandwidth and model size (assuming the CPU/GPU can fully utilize the memory).

For example, the size of Qwen2.5-Coder-14B-Instruct-GGUF = 11.29 GiB, memory bandwidth for AI Max+ 395 = 256 GB/s. The absolute peak speed will be 22.67 tokens per second.

This calculation is not applicable to MoE models, as the number of active experts may vary. Below are the test results for some of the models I used. The same llama.cpp parameters were used for testing: ngl: 99, backend: Vulkan | CUDA, version: 6881 (8b11deea4). The version of llama.cpp is not the latest (October 30), but I don’t think it’s critical. The flash attention flag was not used, as it does not work correctly with all models.

AMD Ryzen™ AI Max+ 395

The main use is for MoE models, or small models for specialized tasks. That is, instead of one huge one, we use many small ones without having to unload them from memory.

Below you can see that MoE models deliver decent performance. However, without MoE, it’s noticeably slower, ~10 tokens per second, which isn’t all that much. Otherwise, you need to reduce the number of parameters and the quantization level -> you lose quality.

Memory usage example for gpt-oss 120B MXFP4 MoE with a context window of 131072 tokens (default).

Device 0 [Radeon 8060S Graphics] Integrated GPU RX: N/A TX: N/A  
GPU 607MHz  MEM 995MHz  TEMP  38°C   CPU-FAN   POW  13 W  
GPU[                                0%] MEM[|||||||||||||||||66.485Gi/96.000Gi]

GeForce RTX 4090

Here, as they say, the results are clear. With a dedicated graphics card, a significant difference in speed becomes apparent. Memory bandwidth is 4 times higher (1008 GBs / 256 GBs = 3,9).

I don’t want to repeat the manipulations manufacturers make in their MiniPC advertising. They loaded larger models onto a graphics card (4090/5090) larger than the available video memory and showed how slow it was compared to their boxes. Naturally, it will be slower when split between the graphics card and a regular CPU/DDR. Therefore, only suitable models are listed below.

The difference between Vulkan and CUDA is too little to be meaningful. It’s possible that you can enable something specific for CUDA that will improve performance, but I haven’t looked into it.

Here’s an example of what happens when some layers don’t fit in video memory and are offloaded to RAM. The resulting performance on the RTX 4090 + DDR4(64 GiB) is three times slower than on the AI Max+ 395 (LPDDR5x). It’s unusable.

gpt-oss 120B was launched with --n-cpu-moe 20. A higher or lower value only resulted in slowdowns.

Final configuration

Ultimately, I’m using everything available. Currently, a PC with a dedicated graphics card is used less frequently. Most models have migrated to MiniPCs. The table below shows a rough breakdown of the main task groups.

I also plan to install and configure Open WebUI to unite all of this together, but I haven’t gotten around to it yet.

It’s important to note that AMD has issues with ROCm support on Linux. It’s only available in Ubuntu 20.24 and, judging by the reviews, it’s not working properly. I hope AMD will finally bring a stable alternative to Nvidia’s CUDA monopoly to a stable state. In the meantime, the Vulkan backend implementation is currently the most stable.

I’ve also experimented with additional services, but I rarely use them right now:

• n8n – a pioneer, so to speak, a convenient tool for organizing processes.
• RAGFlow – RAG, working with documents and the knowledge base.
• OpenHands – a VSCode-based webIDE with an agent.
• Ollama – I don’t understand why it’s so popular, but I tried using it.

A couple of examples for llama.cpp

And here’s an example of running llama.cpp as an OpenAI compatible server, if anyone’s interested.

#!/usr/bin/env bash  
  
MODEL=gpt-oss-120b-MXFP4-00001-of-00002.gguf  
  
llama-server --host 0.0.0.0 --port 1234 \
 -c 0 \
 --batch-size 2048 --ubatch-size 2048 \
 --temp 1 --top-p 1 --top-k 0 \
 --flash-attn on --no-warmup --jinja \
 --alias "gpt-oss" -m ${MODEL}

#!/usr/bin/env bash

MODEL=Qwen3-Coder-30B-A3B-Instruct-Q8_0.gguf

llama-server --host 0.0.0.0 --port 1234 \
 -c 65536 \
 --batch-size 1024 --ubatch-size 256 \
 --n-predict 2048 \
 --temp 0.7 --top-p 0.8 --top-k 20 \
 --repeat-penalty 1.05 \
 -ctk q8_0 -ctv q8_0 \
 --flash-attn on --no-warmup --jinja \
 --alias "qwen-coder" -m ${MODEL}

Is it worth it?

Let me say right away that I started experimenting with generative models back in 2022. That’s when the need for hardware that could run all this locally arose. Cloud services weren’t as widespread as they are now. The total budget for all this was a GeForce RTX 4090 graphics card (2023) and a MiniPC AMD Ryzen™ AI Max+ 395 (2025).

Was it worth it? Considering how much time was spent experimenting with LLM/LDM, it’s more likely yes than no. I suspect subscriptions, token purchases, and paid accounts would have eaten up more money over three years. Besides, this hardware is and will be used for other purposes, not just AI.

And looking at the bigger picture, I’m sure prices for public services will increase in the future. Currently, many AI services are essentially unprofitable, and this race against price gouging will end sooner or later. Investments will need to be recouped, and the ones being “cheated” will be regular users and corporate clients, who will no longer be able to function properly without their familiar tools.

Conclusion

Using local AI models ultimately depends on your immediate needs and interests. In my case, it was justified–at first, I was simply interested and needed it for my projects, then for my main job.

If everything is limited to work-related tasks, and on top of that, it’s free and paid for by the employer, then it’s clearly not worth it.