How to Build a Zendesk AI Chatbot with Ollama (Model Agnostic)

Most “AI chatbot” tutorials hardcode a single model into the application logic.
Then, the model you picked on day one turns out to be:

too slow,
too expensive to self-host at your ticket volume, or
worse at your actual domain than something released three weeks later.

Instead of swapping a value in a configuration file, you end up rewriting the entire integration.
This guide shows you how to build a support chatbot for Zendesk, backed by Ollama and structured so that the model remains a runtime decision rather than an architectural one.

You will run everything through Ollama during the development process, but nothing in your code will assume which model is generating the answers. When testing shows that Model Y performs better than Model X, you only need to change one line.

Target reader: An engineer or technical founder who already uses Zendesk for customer support and wants to add an AI layer they can control and run on their own infrastructure.

The architecture in one picture

Zendesk (messaging / Web Widget)
-> webhook: new message
-> Your bot service (Node/TS)

1. Retrieval layer – embeddings –> vector store (help center chunks)
2. LLM client (model-agnostic interface) –> Ollama
(/v1/chat/completions)
3. Guardrails + human handoff
4. Reply back to Zendesk Conversations API
-> Eval harness (runs the same test set against model X and model Y)

The important line is the LLM client. Everything above and below it must not care whether the answer came from llama3.1:8b, qwen2.5:14b, or mistral-small.

Why model-agnostic is the whole point

Ollama makes this easier than most people realise, because it exposes an OpenAI-compatible endpoint (/v1/chat/completions) in addition to its native /api/chat. If you write your client against the OpenAI-compatible surface, three things become true:

Switching models is a config change (model: "qwen2.5:14b"), not a code change.
Ollama applies each model’s own chat template server-side, so you do not embed model-specific prompt formatting (no manual [INST] tags, no im_start markers) in your code.
If you ever need to burst to a hosted provider for one workload, the same client can point at any OpenAI-compatible API.

The cost of model-agnosticism is discipline: you cannot lean on quirks of one model. Anything that only works because of one model’s specific behaviour is a bug waiting to surface after you switch.

Prerequisites

A host with Ollama installed. For self-hosting at real support volume you want a GPU box; an 8B model runs on a 16GB card, a 14B quantised model wants 24GB.
A Zendesk account with the Conversations API / messaging enabled, and an API token.
Node 20+ (examples below are TypeScript). The same shape works in Python.

Pull two or three candidate models up front so the test phase has something to compare:

 ollama pull llama3.1:8b
 ollama pull qwen2.5:14b
 ollama pull mistral-small
 ollama pull nomic-embed-text

Step 1: Put the model behind a config, not in the code

Define models as data. Nothing else in the system reads these fields directly except the LLM client.

// models.json
  {
    "candidateA": {
      "baseUrl": "http://localhost:11434/v1",
      "model": "llama3.1:8b",
      "temperature": 0.2,
      "num_ctx": 8192,
      "supportsTools": true
    },
    "candidateB": {
      "baseUrl": "http://localhost:11434/v1",
      "model": "qwen2.5:14b",
      "temperature": 0.2,
      "num_ctx": 16384,
      "supportsTools": true
    }
  }

num_ctx is here for a reason: models differ in usable context window, and it is one of the few places where a swap can silently truncate your retrieved documents. Keep it explicit per model, not hardcoded.

Step 2: One thin LLM client, one interface

This is the seam that keeps the model swappable. Every caller talks to chat(); none of them know the model name.

// llm-client.ts
  import OpenAI from "openai";

  export interface ChatMessage {
    role: "system" | "user" | "assistant";
    content: string;
  }

  export interface ModelConfig {
    baseUrl: string; model: string; temperature: number;
    num_ctx: number; supportsTools: boolean;
  }

  export class LlmClient {
    private client: OpenAI;
    constructor(private cfg: ModelConfig) {
      // Ollama's OpenAI-compatible endpoint. apiKey is required by
the SDK but unused.
      this.client = new OpenAI({ baseURL: cfg.baseUrl, apiKey: "ollama" });
    }

    async chat(messages: ChatMessage[]): Promise<string> {
      const res = await this.client.chat.completions.create({
        model: this.cfg.model,
        messages,
        temperature: this.cfg.temperature,
        // Ollama-specific knob passes through; other providers ignore it.
        options: { num_ctx: this.cfg.num_ctx },
      });
      return res.choices[0]?.message?.content ?? "";
    }
  }

Notice what is NOT in here: no prompt templating, no [INST], no model-family conditionals. If you find yourself writing if (model.startsWith("llama")) anywhere, you have broken the abstraction.

Retrieval over your help center (keep embeddings separable)

The chatbot should answer from your actual Zendesk help center, not from the model’s training data. That means retrieval-augmented generation: chunk your articles, embed them, store the vectors, and at query time pull the most relevant chunks into the prompt.

One trap for model-agnosticism: the embedding model and the chat model are two different swappable things, and they are NOT equally cheap to swap. Changing the chat model is free. Changing the embedding model means re-indexing your entire knowledge base, because vectors from different embedding models are not comparable. Treat them as separate configs with a loud comment.

// retriever.ts
  import OpenAI from "openai";
  const embedder = new OpenAI({ baseURL: "http://localhost:11434/v1",
apiKey: "ollama" });

  // WARNING: changing this model invalidates every stored vector.
Re-index on change.
  const EMBED_MODEL = "nomic-embed-text";

  export async function embed(text: string): Promise<number[]> {
    const res = await embedder.embeddings.create({ model: EMBED_MODEL,
input: text });
    return res.data[0].embedding;
  }

  export async function retrieve(query: string, k = 5): Promise<string[]> {
    const qVec = await embed(query);
    return vectorStore.search(qVec, k); // pgvector, Qdrant, or similar
  }

Pull the help center content through the Zendesk Help Center API once, chunk it (roughly 500 to 800 tokens per chunk with overlap), embed, and store. Re-run on article updates.

Step 4: Assemble the prompt without model-specific formatting

Build the conversation as plain role-tagged messages. Ollama turns them into whatever template the chosen model expects. Keep the system prompt behaviour-focused, not model-tuned.

// orchestrator.ts
  export async function answer(llm: LlmClient, userMessage: string,
history: ChatMessage[]) {
    const context = await retrieve(userMessage, 5);
    const grounded = context.length > 0;

    const messages: ChatMessage[] = [
      { role: "system", content:
        "You are a support assistant for <Company>. Answer only from
the provided " +
        "context. If the context does not contain the answer, say you
are not sure and " +
        "offer to connect a human. Never invent policy, prices, or
account details." },
      ...history,
      { role: "user", content:
        "Context:\n" + context.join("\n---\n") + "\n\nQuestion: " +
userMessage },
    ];

    const reply = await llm.chat(messages);
    return { reply, grounded };
  }

The instruction to fall back to a human when context is missing is your single most important guardrail. It is also model-agnostic: it works regardless of which model you swap in, because it constrains behaviour through the prompt and the retrieval result, not through model internals.

Step 5: Wire it into Zendesk

Zendesk messaging delivers new messages to your service via a webhook, and you reply through the Conversations API. Keep Zendesk-specific code at the edges so it never touches the model layer.

// zendesk-webhook.ts (Express)
  app.post("/zendesk/webhook", async (req, res) => {
    const { conversationId, message, authorType } = parseZendeskEvent(req.body);
    if (authorType !== "user") return res.sendStatus(200); // ignore
our own messages

    const llm = new LlmClient(loadActiveModel()); // reads models.json
-> active key
    const { reply, grounded } = await answer(llm, message, await
loadHistory(conversationId));

    if (!grounded) {
      await zendesk.handoffToAgent(conversationId); // human takes over
    } else {
      await zendesk.postMessage(conversationId, reply);
    }
    res.sendStatus(200);
  });

loadActiveModel() reads which config key is live. In the test phase, this is how you flip between model X and model Y, or route a percentage of traffic to each.

Step 6: The evaluation harness (this is where model choice gets decided)

Model-agnostic architecture is only useful if you actually measure which model to run. Build a fixed test set before you pick a winner. Pull 100 to 200 real resolved tickets, record the question and the correct resolution, and score each candidate model against the same set.

// eval.ts
  const models = ["candidateA", "candidateB"];
  for (const key of models) {
    const llm = new LlmClient(configs[key]);
    let correct = 0, hallucinated = 0, totalMs = 0;

    for (const t of testSet) {
      const start = Date.now();
      const { reply, grounded } = await answer(llm, t.question, []);
      totalMs += Date.now() - start;

      const verdict = judge(reply, t.expectedAnswer, grounded); //
LLM-as-judge or manual
      if (verdict === "correct") correct++;
      if (verdict === "hallucinated") hallucinated++;
    }

    console.log(key, {
      accuracy: correct / testSet.length,
      hallucinationRate: hallucinated / testSet.length,
      avgLatencyMs: totalMs / testSet.length,
    });
  }

Decide on the metrics that matter for support specifically:

Deflection rate: how many tickets it resolves without a human.
Hallucination rate: how often it states something not in the context. For support this is the metric that can actually hurt you, so weight it heavily.
Latency and tokens/sec: a 14B model may answer better but halve your throughput on the same GPU. That is a real cost, not a footnote.
Handoff correctness: does it escalate the right cases and not the easy ones.

Because the architecture is model-agnostic, this harness runs the exact same code path production uses. You are testing the real system, not a proxy.

What goes wrong

Silent context truncation on swap. You move to a model you configure at 4k and half your retrieved chunks fall off the end. Answers get worse and it looks like the model is dumber than it is. Set num_ctx per model and log when the prompt exceeds it.

Tool-calling assumptions. If you add function calling for account lookups, not every local model supports it well. Keep supportsTools in config and degrade gracefully rather than assuming.
Embedding model drift. Someone “upgrades” the embedding model without re-indexing. Retrieval quietly returns garbage because old and new vectors are not comparable. Guard this with a version stamp on the index.
Judging with the same model that answers. If your eval judge is the same model as the candidate, it flatters itself. Use a separate, stronger judge or a human pass on a sample.
Prompt tuned to one model. You tighten the system prompt until model X behaves, then model Y regresses. That means you overfit the prompt. Re-run the eval set after any prompt change, across all candidates.