zkzkAgent Adds Nosana GPU Inference While Keeping Linux Tools Local
- zkzk gamal
- 539 reads
zkzkAgent can send demanding model work to Nosana while files, commands and safety checks stay on the user’s machine.

zkzkAgent’s move to a larger model changes where its reasoning happens, while leaving the work on a Linux machine under local control. The agent can use a 27B model served through Nosana’s decentralized GPU network; its graph, tools, files and confirmation checks still run on the user’s hardware. That split gives modest laptops another option when a small local model struggles with a demanding task.

The project previously ran with Ollama and a 2B model, qwen3.5:2b. That model fits in laptop memory and responds quickly, but planning a software project puts more pressure on it. zkzkAgent’s planner lays out each step and supplies the content of the files it intends to create. A 27B model won’t fit on the author’s laptop, so using one had meant relying on a centralized provider. Nosana provides a remote inference path without moving the agent’s execution environment off the machine.
Remote inference, local execution
The integration follows a narrow rule: only model requests go to Nosana. zkzkAgent uses a multi-node LangGraph setup, with separate nodes for classifying requests, answering questions, writing plans and executing tools. Its 25 tools cover files, applications, networking, processes, packages and deployment. The graph routes work and records why it chose a path; the tools themselves act on the local Linux system.
That boundary matters because the model can propose an action without being able to run it directly. Destructive operations, including removing files, emptying the trash and installing or removing packages, are blocked until the user types yes. If a remote model suggests remove_file, the local tool still needs that confirmation before it can act.
The backend switch is configured in models/LLM.py using settings in .env. With MODEL_TYPE="NOSANA", the agent reads a Nosana API key, base URL and model name, then connects through LangChain’s ChatOpenAI client. Nosana’s inference endpoint accepts the OpenAI chat format, so the rest of the graph can work with the same llm interface it uses for other backends. The CLI also supports streaming, conversation history and slash commands, alongside Docker and optional voice modes.
There are a couple of practical details in the connection. A model appears in Nosana’s GET /models response only while a GPU is serving it. If NOSANA_MODEL is left empty, zkzkAgent checks that list and selects an available model. The client also requests streamed usage data; the final chunk includes token counts, which provide the basis for tracking inference cost.
Users who want to run their own model server have another route: the repository includes nosana-ollama.json, a job definition that starts an Ollama container on a GPU node and pulls qwen3:14b. The shared inference API and a user’s own Ollama deployment are distinct ways to put the model on a remote GPU, while keeping the agent’s tools local.

A project build tests the full loop
For a practical trial, the developer pointed zkzkAgent at the 27B Nosana model and asked it to create zkzk_ai, a calculator with a FastAPI backend and an HTML, CSS and JavaScript frontend, then open the project in VS Code. The classifier sent the request to the planner, which produced a nine-step plan with the proposed file contents. Execution waited until the user typed ok.
After that confirmation, the agent made 10 tool calls over 10 graph iterations, within a maximum of 15. It created ~/zkzk_ai, wrote app.py and the three frontend files, added requirements.txt, README.md and .gitignore, and opened the directory in VS Code. The cycle is straightforward: Nosana’s model chooses a tool, the tool runs locally, and its result returns to the model for the next decision.
A follow-up request asked the agent to rename the calculator “zkzk.” Again, it drafted a plan and waited for confirmation. It read static/index.html, then updated the page title and heading, the FastAPI application title and the README. The calculator logic stayed untouched.
The developer then launched the app with uvicorn app:app. The browser loaded the page and its static assets, and a request to /api/calculate returned successfully. Across the two tasks, the saved log records 18 model calls to inference.nosana.com, each returning HTTP 200. File writes, shell commands and launching the app took place locally.
The generated calculator code also got several safety and usability details right in the author’s review. It parses expressions with Python’s ast module instead of calling eval, allowing numbers, parentheses and selected arithmetic operators. An attempt to pass __import__("os") was rejected because a function call is not an allowed expression element. Division by zero and complex results such as (-1)**0.5 produce readable errors. Both the browser and API cap expressions at 500 characters. Keyboard controls support evaluation, deletion, clearing and powers.
The code for the calculator is available at zkzkGamal/calculator, and the execution log is in the zkzkAgent repository.
The trade-offs still need measuring
This setup makes zkzkAgent local-first rather than local-only. Sensitive work can stay on Ollama; heavier planning can use a larger model on Nosana. That flexibility comes with a clear boundary to understand: the model calls leave the machine when Nosana is selected, even though the tools and resulting file operations remain local. Teams deciding which tasks fit each mode will need to consider the information sent in prompts as well as the model’s capability.
The current demonstration shows a working task flow, but it doesn’t establish how much the larger model improves results or what that improvement costs. The developer plans to run five tasks on both the local 2B model and the 27B model, comparing success rate, tool calls, elapsed time, token use and cost. Those measurements should help show whether remote inference earns its place for planning, and how often the smaller model is already enough.
Other planned work includes assigning different models to different graph nodes: short classification and chat could remain local, while planning and execution use Nosana. Automatic fallback to Ollama would keep the agent usable when the remote endpoint can’t be reached. Both ideas would add resilience or reduce the amount of conversation sent off-device, but neither is implemented yet.
The safety boundary also has a gap the author identifies. Plan approval exists, and certain destructive tools require confirmation, but run_command can currently execute any shell command. Since the model chooses its arguments, adding a confirmation step or an allow-list is a meaningful next task. The project’s next test is not just a stronger model; it’s whether the added capability can be measured and paired with controls that fit the broader set of actions.
zkzkAgent’s setup instructions and code are at github.com/zkzkGamal/zkzkAgent. To try it, clone the repository, install its requirements, copy .env.example to .env, set MODEL_TYPE="NOSANA" and a Nosana API key, then run ./cli.sh.