braindead-local-intelligence

A local, OpenAI-compatible LLM and image generator on AMD GPUs, protected by a token. Both are always available on a single URL; the requested model is loaded on demand (only one in memory at a time), then unloaded after 10 min without requests to free the GPU and RAM.

Setup (copy and paste)

Requirements: Linux, podman ≥ 5.0, an AMD graphics card.

1. Grant GPU access, then log out and back in (or reboot):

sudo usermod -aG render,video $USER

2. Install, from the project folder:

bin/ia install
ia doctor

ia doctor should end with "All good." and list the AMD card. If ia is not found: open a new terminal (the command is in ~/.local/bin).

3. Download the models (Qwen3-VL 4B for text, which also understands images, and SD 1.5 to generate images; ~8 GB, already configured):

M=~/.local/share/ia-locale/models
curl -fL -C - -o $M/llm/Qwen3-VL-4B-Instruct-Q4_K_M.gguf \
  https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF/resolve/main/Qwen3-VL-4B-Instruct-Q4_K_M.gguf
curl -fL -C - -o $M/llm/Qwen3-VL-4B-Instruct-mmproj-F16.gguf \
  https://huggingface.co/unsloth/Qwen3-VL-4B-Instruct-GGUF/resolve/main/mmproj-F16.gguf
curl -fL -C - -o $M/sd/v1-5-pruned-emaonly.safetensors \
  https://huggingface.co/stable-diffusion-v1-5/stable-diffusion-v1-5/resolve/main/v1-5-pruned-emaonly.safetensors

4. Start:

ia start

The first start downloads the engines (several GB): allow a few minutes. When it is done, ia start prints the settings to copy into your application.

Connect an application (bot, Pydantic AI…)

Any OpenAI-compatible client needs three values. ia info prints them already filled in:

$ ia info
Settings for an OpenAI client (bot, Pydantic AI…):
  base_url : http://192.168.1.20:8080/v1
  api_key  : 3f9c…
  model    : local (text)  or  sd-cpp-local (images)

Setting	Value
base_url	http://SERVER_IP:8080/v1 (text and images, with /v1)
api_key	the token, shown by ia token
model	local for text, sd-cpp-local for images

Which model name? Always local or sd-cpp-local, whatever file you downloaded. These names are fixed: switching models (see below) changes nothing on the application side. The name also tells the server what to load into memory. To see which model is loaded right now: ia status.

Python examples (Pydantic AI) and common errors: docs/clients.md.

Browser

On the server machine, open http://127.0.0.1:3000/: a ChatGPT-like interface (Open WebUI), no account needed. To let the model search the web, enable Web Search in the + menu of the input box.

Text or image: there is a single chat model, local. For an image, enable Image (image generation) in the input box, describe the image and send; disable it to go back to text. The server switches between the LLM and the image generator on its own: the first image, and the first text answer after it, wait for the model to load. Image size: IMAGE_SIZE in config/open-webui.conf (match it to the image model, then ia restart).

Commands

Command	What it does
ia start	Start everything
ia stop	Stop everything
ia restart	Restart (after changing models)
ia info	Show the URL, token and model names to copy
ia status	Show what is running and which model is loaded
ia unload	Unload the model now (frees GPU and RAM)
ia jobs	Show the running engine: model file, running time, last image prompt
ia cancel	Stop a generation in progress (kills the engine, the rest keeps running)
ia ttl 600	Delay before automatic unloading, in seconds (0 = never)
ia logs	Show the logs when something goes wrong
ia token new	Change the token (the old one stops working)
ia listen 127.0.0.1	Block access from the network (ia listen 0.0.0.0 to reopen)
ia pull	Update the engines (then ia restart)
ia doctor	Check GPU, podman and permissions
ia uninstall	Uninstall everything (keeps models, config and token)

Switching models

Copy and paste a block from docs/models.md (smaller, bigger, better quality, MoE for 12 GB of VRAM), then ia restart.

Web search

The LLM can search the web, like OpenAI's web_search: the client adds the {"type": "web_search"} tool to its request. The server runs the searches (SearXNG, local, no API key) and returns the final answer. Examples: docs/clients.md.

Network access

The API listens on the whole network (port 8080) and requires the token. Open the port in the firewall if needed:

sudo firewall-cmd --permanent --add-port=8080/tcp && sudo firewall-cmd --reload  # Fedora
sudo ufw allow 8080/tcp                                                          # Debian/Ubuntu

The connection is unencrypted HTTP: the token is sent in clear text. On a trusted network that is enough; from the Internet, go through a VPN (WireGuard, Tailscale) or an HTTPS reverse proxy.

The browser interface (port 3000) is only reachable from the machine itself.

Testing

TOKEN=$(ia token)

# Text (add ,"tools":[{"type":"web_search"}] for web search)
curl http://127.0.0.1:8080/v1/chat/completions -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"model":"local","messages":[{"role":"user","content":"Hello!"}]}'

# Image → cat.png
curl -s http://127.0.0.1:8080/v1/images/generations -H "Authorization: Bearer $TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"model":"sd-cpp-local","prompt":"an astronaut cat","size":"512x512"}' \
  | jq -r '.data[0].b64_json' | base64 -d > cat.png

Other ports (optional)

Not needed for normal use, kept for troubleshooting and compatibility:

• 8081: same API, bypassing web search (fallback if it is down). Token required.
• 7860: former image port, same as 8081. Token required.
• 8090: llama-swap interface (loaded model, logs), no token, only from the machine itself.

How it works in detail and troubleshooting: docs/configuration.md.