One GPU, Four Tenants: How My AI Hub Shares 32 GB

My PC has one graphics card, an RTX 5090 with 32 GB of memory, and four kinds of work that each want most of it:

A coding agent on a local model. Oh My Pi (omp) talks to one local LLM endpoint, served either by llama.cpp or by NInfer. NInfer at 262k tokens of context with vision holds 29.5 of the card’s 32.6 GB.
Pictures, in ComfyUI, through the recipes in Comfy Studio. A Qwen Image render wants 15 to 20 GB.
Video, with MiniMax H3, also in ComfyUI. It wants the whole card: about 39 GB of weights streamed through 32.
Speech and music in audio.cpp: 4 GB for a transcription, up to 15 for a song.

No two of the big ones fit at the same time. For a while that was my problem to solve by hand, every time. This post is about how the hub solves it now.

The hub itself, what it is and how I reach it from my phone, is in My GPU Workstation in My Pocket. I won’t repeat that here.

The failure that started it

The first program that used three of these apps in a row was a small batch job of mine: audio.cpp transcribes a recording, Gemma 31B on the LLM endpoint turns the transcript into a one-sentence picture description, and ComfyUI paints it. On its first real run, in early October, Gemma failed to load on the second pass. The LLM server answered with a 500, because ComfyUI was still holding about 15 GB from the render before.

The fix at the time was one line: tell ComfyUI to free its memory before asking Gemma. It worked, and it was only safe by luck. Nobody else was rendering. Had I been making a picture in ComfyUI’s own page at that moment, the batch job would have pulled the model out from under me.

That is the whole problem in one sentence: whether an app may be unloaded depends on whether somebody else is using it, and only one place can know that. No single program can. So the rule became: programs ask, and only the hub evicts.

What the hub can measure

Less than I hoped. On Windows, under the normal display driver model (WDDM), nvidia-smi reports per-process memory as [N/A]. I measured it before building anything. So the hub never knows what each process holds. It knows two numbers: the board’s free memory, and the size the asking program says it needs.

Everything below rests on those two numbers. It turns out to be enough.

Three ways to hold the card

There are three mechanisms, built in this order, each for a case the previous one could not serve.

	what it is for	what the hub does
Lease	borrowing the LLM endpoint with particular settings	writes a profile, starts the engine, puts the old settings back afterwards
Claim	using ComfyUI, audio.cpp or llama.cpp for a while	starts the app, unloads idle apps until the claimed size is free, protects it
Park	the agent’s own job needs the card the agent is using	stops the engine, and brings it back exactly as it was

Leases: the engine slot has settings worth borrowing

The LLM endpoint holds one engine at a time, llama.cpp or NInfer, and the hub’s card for it has a lot of settings: which engine, which model file, whether vision is on, the KV-cache format, the context length, the speculative decoding mode. Comfy Studio and audio.cpp Studio both use the endpoint to write prompts, and they want particular combinations. A prompt for a 2,000-token enhancement doesn’t need a 200k-token KV pool sitting in memory.

So since late September they lease it. A program names a profile, a whole set of the card’s settings kept in the hub’s config, and the hub:

copies the card’s current settings aside on the first lease,
writes the profile and starts or swaps the engine,
on the last release writes the old settings back byte for byte, and deletes a file that wasn’t there before.

Restoring means the settings, not the engine. I made that call because restarting whatever ran before costs 20 to 60 seconds nobody asked for. The card saying “restart to apply” is the cheaper surprise.

Three rules keep it honest. A NInfer start is priced before it begins and refused, naming who holds the memory, because NInfer allocates its whole pool at startup and cannot make room once it is loading. A hand on the card ends a lease: if I press Start or Stop myself, the program loses its loan. And a lease nobody touches for 20 minutes lapses, so a crashed program can’t pin the engine forever.

Claims: a size, and a promise

ComfyUI and audio.cpp have no settings worth borrowing. What they load is decided per request. What a program needs from them is the other half of what a lease does:

the app running,
room on the card,
not being unloaded in the middle of its work.

That is a claim, built on 6 October: “I am using ComfyUI and need about 20 GB; make it so, and keep it so until I let go.”

aihub --client my-batch claim comfyui --gb 20 -- python render_all.py

The script runs as a child. The claim is touched every minute and released when the script exits. On a claim the hub, under one lock:

starts the app if it is down and waits for it to answer,
measures the board and, if there isn’t enough free, unloads the other GPU apps one at a time, re-measuring after each, so nothing is evicted that the claim didn’t need. Unloads, never stops: a stopped ComfyUI costs the next render its startup.
skips anything protected: an app somebody else has claimed, an engine on lease, or an app visibly busy without a claim. That last one is what protects me: a job in ComfyUI’s queue, the coding agent in the middle of a turn, a llama.cpp model with a request in flight.
refuses, naming the holder, only when a protected app is in the way. Memory nobody can be named for is granted anyway, with how short it fell, because no program could act on a refusal over it.

Two details took a real run each to find. llama.cpp’s router answers “is this model busy” only per model, and asking about a model that isn’t loaded loads it. My first probe of an idle Gemma started a 20 GB load that then failed behind ComfyUI. Now only loaded models are asked. And a release unloads nothing on purpose: the app stays warm for the next stage that wants it, and the next claim evicts it only if it needs the room.

A hand on the card ends a claim too. If I press Free VRAM on ComfyUI while a program holds it, the program learns at its next touch and stops, with “ended on the hub’s card”. It has to stop rather than carry on, because carrying on would mean restarting the app I just freed.

Arbitration is plain HTTP on purpose. A claim is one request, so anything that can send one can take part, not just my own client library. And the work never runs inside the hub: the hub arbitrates the card, the programs keep their own failure modes.

Two exceptions came later.

On 21 October, a video upscale ran beside a MiniMax render with nobody the wiser. The upscaler (NVIDIA’s video super resolution, and RIFE for frame interpolation) runs inside the calling program, not in an app the hub manages, so there was nothing to claim. Now a program can claim the card itself, gpu, with a size. It starts nothing, but while it stands the hub names it as a holder when somebody else’s room comes up short.

And NInfer is the one thing allowed to have apps stopped rather than unloaded. An unloaded ComfyUI still holds its CUDA context, about 640 MB that no unload releases. At NInfer’s heaviest settings that is the whole margin above the engine’s own 1 GB of headroom. One night it made NInfer’s return after a park fail three times. So when a NInfer start is refused for room, the hub stops the idle, unclaimed apps and tries once more. Only after a refusal, never priced in advance: the hub’s own estimate for that setting was 1.2 GB too high, and the same start had fit beside ComfyUI twice that evening. A pre-check would have stopped ComfyUI for starts that never needed it.

Measuring a size, and why a reading afterwards is wrong

Every claim’s size is the program’s own guess, so the guesses had better be measured. The method: sample the board’s used memory every half second through one run, from cold, take the peak above where it started, and round up.

The reason for “peak” is MiniMax Music 3 in audio.cpp. During a 30-second song it peaks at 11.8 GiB. After the take it lets go of all but 1.6 GiB. If you look at nvidia-smi after a run, you see 1.6 and conclude the model is small.

MiniMax Music 3: the peak during a take, and what stays loaded after it.

A claim sized on what stays loaded leaves the gap open. The hub hands it to the next program, that program loads into it, and the next take fails. And a load that misses doesn’t slow down, it fails outright, so there is no gentle way to find out a guess was small.

Disk size is no guide either. VoxCPM2 peaks at 1.2 times its files, Ace-Step at 3.2 times. The measured peaks, as the client library uses them today:

model	app	claimed
Qwen3-ASR 0.6B (transcription)	audio.cpp	4 GB
VoxCPM2 / Fish S2 Pro (speech)	audio.cpp	6 / 7 GB
MiniMax Music 3 / Ace-Step 1.5 turbo (music)	audio.cpp	12 / 15 GB
Gemma 31B, Qwen3.8 27B	llama.cpp	23 GB
Qwen Image 2.1	ComfyUI	15, the recipe says 20
MiniMax H3 video	ComfyUI	29 to 30, i.e. everything

The MiniMax H3 number is a cold number on purpose. A cold load peaks higher than a warm one, and the one 15-second clip that failed was the one that started cold.

The park: when the agent’s own job needs the card

Claims work for programs that start with the card mostly free. They fail for the case I wanted most: omp on NInfer, on my phone, writing a workflow that renders something and running it. NInfer holds 29.5 GB. A claim for ComfyUI is refused, naming NInfer, because a claim never stops an engine somebody chose on the card.

That rule is right for a program I didn’t start. It is wrong for my own agent asking to step aside for a while. So on 7 October the hub got the inverse of a lease. A lease brings an engine for a program and takes it away afterwards. A park takes the engine away for a program, and promises it back afterwards, exactly as it was.

Before building it I measured what would happen:

Stopping NInfer does not kill omp. They are separate processes. Only an LLM request in flight is affected, and omp retries a refused connection for about 8.5 minutes.
omp already has background jobs. Its bash tool can run a command with async: true, tell the model “end your reply and wait to be woken”, and when the job ends, inject its output and start a new turn by itself.
A warm NInfer restart takes 23 to 24 seconds to serving.

So the waking was already solved. What was missing was the hub’s half: take the engine away for the length of the job, and have it back before omp wakes.

aihub park -- aihub render qwen21-t2i "A lamb in a scarf" -o lamb.png

The agent runs that as a background job and ends its reply. Then:

The park is refused with “turn” until the reply has really ended, and the client asks again. The job starts while the model is still writing “started”.
The hub writes down how the engine runs, before it touches anything, so a crash can be recovered. Then it stops NInfer.
The command runs. Inside it is ordinary: it claims ComfyUI with a size, as any program would.
When the command exits, the client asks for the engine back. The hub first makes room, because ComfyUI is still warm, then starts NInfer with the settings it was parked with.
The client exits only once the engine serves again. That exit is what wakes omp, so omp wakes onto a working engine.

The card's memory during the first park, 7 October, to scale. GB as nvidia-smi reports them.

The first real run: I asked omp for a picture. It read the skill, started the park as a background job and ended its reply. Board use fell from 29.5 to 1.3 GB, the workflow claimed ComfyUI, the render took 18 seconds, ComfyUI was unloaded and NInfer started again. omp woke 69 seconds after the job started, on a serving engine, and reported the file. Nothing in the session was lost.

One park, from the agent starting the job to the agent waking up.

Who may park, and who brings it back

Only the agent’s own job may park while an agent is up. The hub mints a token per session and puts it in the agent’s environment. A park without it is refused while a session is running. With no session, anybody may park: that is an engine nobody is using, and it comes back unchanged.

The hub owns the return, not the workflow. A sweeper watches the parking process and brings the engine back the moment it is gone. That covers a crashed script, a killed client and a hub restart, since the park is written to disk like the leases and claims. omp retries for about 8.5 minutes and a return takes about 25 seconds, so even a wake that somehow beats the return finds an engine. The first version also lapsed a park on the 20-minute idle clock. A long render touches its claims, not the park, and would have had the engine brought back into the middle of it. A watched park now lapses only when its process is gone.

I can end it early. The card has Bring back now. It ends the job’s claims, so the workflow stops at its next touch, and returns the engine. While parked, the chat on my phone refuses to send a prompt and says why, with the button in the sentence. A prompt to a stopped engine would just sit in omp’s retries.

The same evening I tried the three cases that could go wrong, on the real hub: a parker killed mid-park (the engine served again about 30 seconds later), Bring back now under a ComfyUI claim (the workflow stopped, nothing left behind), and a return with another program still holding ComfyUI (the return waited, named the holder, and went ahead as soon as the claim was released).

They also found three bugs. One is worth telling: the park said “parked” as soon as its record was written, which is first, for crash recovery. A claim taken on that word came up 17.7 GB short, because NInfer was still shutting down. It says “parking” now, until the engine is really down.

The deadlock the model walked into

On 18 October, on a long render-then-video task, the park sat at “3 min 25 s so far” until the client gave up. The model had started the park correctly, done some other work, and then, with its next step depending on the pictures, called omp’s wait tool. The background job’s own message offers it: “Completely blocked? Call wait”. But wait holds the reply open, and the park waits for the reply to end. Each waited for the other.

The obvious fix was wrong. Aborting the turn looks to omp like the person interrupting, and then omp suppresses the automatic wake: the job’s result would arrive and wake nobody. Turning wait off wasn’t possible either; omp has no switch for it alone. Now the hub notices a park refused while the agent sits in a wait, and sends one short message into the session. omp ends a wait on any incoming message, the engine is still up, and the model gets a turn to read why and end its reply.

Workflows report their own progress

Once parks worked, the phone showed “NInfer parked” and a claim row, “holds ComfyUI, 20 GB”, and nothing about which step was running or how long was left.

The hub can’t work that out. It doesn’t know the plan, and omp hands a job’s output to the model only when the job ends. So since 17 October the workflow reports itself. It sends its whole run, every step with its counts and timings, to the hub on every change. The whole record each time, from one writer per record, so there are no merge rules and a lost update is fixed by the next. The hub only keeps it, stamps the times, and marks a run vanished when its process is gone without saying so. A failed report never fails the work: a progress line must not break a batch.

My client library’s batch runner reports by itself, and so does every aihub wrapper. aihub park -- python flow.py shows up as one run: park, the flow’s stages, the return.

The estimate of time left is deliberately modest. Within a stage it uses the mean of the items done after the first, because the first carries the model’s load. Stages that haven’t started are estimated only from the last run of the same workflow, and say “unknown” otherwise. On the second run of a test, the return was estimated at 29.9 seconds and took 29.2, and the per-picture rate was 5.4 against 5.4. The load was the miss: 20 seconds estimated, about 4 taken, because the second run read the model from a warm file cache.

Inside a single render the hub follows ComfyUI node by node over its websocket. That only works if each prompt names the hub’s socket as its client: ComfyUI broadcasts the sampler’s steps to everyone, but which node is running, and when the prompt ends, only to the client that is named. My first version didn’t name it, and the card showed the sampler stuck at 20/20 through the whole decode.

The image that crashes the engine

One more rule is about pictures, not memory, but it frees memory in the worst way. NInfer’s vision decoder crashes on WebP: corrupted size vs. prev_size, the process gone, 20 GB of memory suddenly free, and every program using the endpoint left without a server.

It hit every picture I sent from my phone. omp picks the smallest of PNG, JPEG and WebP when it prepares an attachment, and WebP nearly always wins. What reached the chat was “Unable to connect”, retried ten times against a server that no longer existed. The hub now starts omp with OMP_NO_WEBP=1, always, because the engine can change under a session that is already running. The client library refuses WebP before sending, and Comfy Studio re-encodes any reference picture that isn’t PNG or JPEG. llama.cpp has no such problem, which is exactly why it was easy to miss.

What I didn’t build

Most of the design is in what was turned down:

A lease per app. ComfyUI and audio.cpp have no settings to snapshot. It would have been the lease’s machinery with nothing inside.
Mutual exclusion between apps. ComfyUI workflows call the LLM for prompt enhancement, so both must be able to run together. A claim makes room on demand; it never bars an app.
The hub running the jobs. A queue in the hub that renders on a program’s behalf would put every workflow’s failure modes next to the dashboard, and a hub restart would kill a two-hour backfill. This came up three times, for claims, for the park and for progress, with the same answer each time.
Pricing memory per process. Not available, measured.
Forcing out a claimed app. A lease can be forced because the card’s owner may override their own dashboard. A claim’s holder is another program in the middle of a render, and overriding it ruins that render. Waiting is the polite form.
Stopping NInfer and leaning on omp’s retries. It works by accident for jobs under 8.5 minutes, puts error turns on the phone, and fails outright beyond that. One parked render ran for 2 hours 15 minutes.
A smaller NInfer beside the render. Qwen Image wants 15 to 20 GB, and any NInfer profile takes more than what is left. A coding agent at 64k context is a different tool.
Inferring progress from the files a workflow writes. The hub wouldn’t know the plan, when a step started, or a running workflow from an abandoned one.

What is still open

A release unloads nothing, so ComfyUI keeps its memory until the next claim needs it. That is right for batches and wrong for me looking at a full card on my phone. An idle timer would fix it; I haven’t decided.

The cost of waking up is unmeasured at large contexts. NInfer’s cache goes with the engine, so after a park omp re-reads the whole conversation. At 9k tokens that was invisible. At 150k it won’t be.

And pi, the other coding agent the hub runs, has no background jobs, so none of the park works for it yet.

Still, the everyday result is what I wanted: I ask the agent on my phone for a picture or a short clip, it starts a job and goes quiet, the card changes hands twice, and a minute or two later the agent picks up where it left off, on the same engine with the same settings, without me touching anything.