Appendix B — Running AI Locally and in the Cloud

“The plans of the diligent lead surely to abundance, but everyone who is hasty comes only to poverty.” — Proverbs 21:5 (ESV)

Every other appendix in this book gets your tools working. This one gets your workload working — the AI itself. By the end you’ll have a real language model answering on your own machine, a web UI in front of it, an understanding of how a GPU serving stack differs from the laptop version, and four cloud providers wired up safely so you can reach a frontier model when you need one.

Here is the one fact that makes all of this tractable: the OpenAI-compatible HTTP API is the lingua franca of LLM serving. Almost every runner — local or cloud — speaks /v1/chat/completions. Point the same SDK at a different base_url, hand it a key, and your code doesn’t change. Learn the shape once and you can talk to anything. As of 2026 this is the single most durable skill in this entire appendix; model names will churn, the protocol won’t.

This is a get-it-working guide, not a survey. Pick the path you need and follow it to a verify step. Every install ends with a command that proves it worked. Don’t skip it.

Coach’s Note — Run something local before you reach for the cloud. A model on your own hardware costs nothing per token, leaks nothing off your machine, and teaches you what the cloud is actually selling you. The cloud is for capability and scale you can’t host — not for things you were too impatient to install.

B.1 — Run a Model Locally with Ollama

Ollama is the fastest way to a working local model. It’s a CLI plus a background daemon that pulls quantized GGUF weights, manages them, and serves them over an OpenAI-compatible endpoint at localhost:11434. It’s the right tool for personal, dev, and low-concurrency work — roughly one to three users. (As of 2026 the current line is Ollama v0.30.x; re-check the version when you install.)

Install

• macOS: download the app from ollama.com/download, or brew install ollama.
• Windows: download and run the installer from ollama.com/download.
• Linux: the official one-liner installs the binary and a systemd service:

curl -fsSL https://ollama.com/install.sh | sh

Pull and run a model

ollama pull downloads weights; ollama run drops you into an interactive chat (and starts the daemon if it isn’t running). Start small — an 8B-class model in a 4-bit quant needs only ~5 GB of memory and runs on a normal laptop:

ollama pull llama3.1:8b
ollama run llama3.1:8b

Type a question, get an answer, Ctrl+D to exit. List what you’ve pulled with ollama list; remove one with ollama rm llama3.1:8b.

Coach’s Note — The tag after the colon is the quantization, and it is a real engineering choice, not a detail. A 4-bit quant (the common default is Q4_K_M, roughly 4.5–4.9 bits per weight) cuts memory to about 0.5 GB per billion parameters versus ~2 GB/B at FP16 — at the cost of some quality on hard reasoning and math. Pull a model at two different quants, ask both the same ten hard questions, and watch the knee in the curve. That’s Chapter 2’s lesson with your own hands on it.

The endpoint is the whole point

The daemon serves an OpenAI-compatible API at http://localhost:11434/v1. Prove it with plain curl — no SDK, no key that means anything:

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Say hello in five words."}]
  }'

You’ll get back a JSON response in the exact shape OpenAI’s API returns. That’s not a coincidence — it’s the design. Hold onto that :11434/v1 address; B.4 points real SDK code at it.

B.2 — Put a Face On It with Open WebUI

A terminal chat is fine for you. It’s useless for the rest of a team. Open WebUI is the dominant self-hosted, ChatGPT-style front end (~142K GitHub stars as of 2026). It is a UI and orchestration layer — not an inference engine. It does not run the model; it talks to a backend that does, like your Ollama daemon. It adds chat history, document upload and RAG, multi-user accounts with role-based access control (RBAC), and MCP tool connections on top.

Coach’s Note — Open WebUI adopted a branding-protection license in 2025 (v0.6.6) that drew real community pushback. It’s still free to self-host for ordinary use, but if you’re deploying it at an organization, read the license before you assume “open source” means what you think. Verify the current terms — license posture is one of the fast-moving facts in this space.

Run it in Docker, pointed at Ollama

Docker is the clean way to run it (see Appendix A for Docker itself). This command starts Open WebUI and tells it where your Ollama daemon lives:

docker run -d \
  --name open-webui \
  -p 3000:8080 \
  -e OLLAMA_BASE_URL=http://host.docker.internal:11434 \
  -v open-webui:/app/backend/data \
  ghcr.io/open-webui/open-webui:main

A few things worth understanding rather than copying blind:

• -p 3000:8080 maps the container’s port 8080 to localhost:3000 on your machine — that’s the URL you’ll open.
• OLLAMA_BASE_URL points the UI at the Ollama daemon. host.docker.internal is how a container reaches a service on the host on macOS and Windows. On Linux, run with --network=host and use http://localhost:11434 instead.
• -v open-webui:/app/backend/data keeps your accounts and chat history in a named volume so they survive a container restart.

Verify it worked

Open http://localhost:3000 in a browser. Create the first account (it becomes the admin), pick llama3.1:8b from the model dropdown, and send a message. If it answers, your front end and back end are talking. Now try the part that earns its keep: upload a PDF and ask a question grounded in it (that’s RAG), then create a second, non-admin user under Settings and watch RBAC scope what they can see. That’s the front-end/back-end split that the rest of your career as an AI administrator runs on.

B.3 — Optional: A Production GPU Server with vLLM

Ollama is one barista. When you need to serve many concurrent users from a real GPU, you want a production serving engine, and as of 2026 the reference open choice is vLLM (a PyTorch Foundation project). vLLM brings the machinery that personal runners don’t: PagedAttention for memory-efficient KV-cache handling, continuous batching so the GPU stays busy across many in-flight requests, FP8 serving, prefix caching, and multi-GPU/multi-node scaling. (SGLang is the other major open engine; NVIDIA’s NIM containers are the vendor-supported, NVIDIA-GPU-only path for air-gapped or SLA-bound shops.)

This section is optional and needs an NVIDIA GPU with enough VRAM for your model. Skip it if you don’t have the hardware — nothing later depends on it.

Coach’s Note — Don’t reach for vLLM on a laptop. The split is clean: Ollama / LM Studio / llama.cpp for personal, dev, and ~1–3 users; vLLM / SGLang for production GPU serving where throughput is the metric you’re paid to move. Using the heavy tool for the light job is its own kind of mistake.

Run it in a container

The simplest start is the official image. Size your model to your card — a gpt-oss-20b-class model runs in roughly 16 GB; gpt-oss-120b fits on a single 80 GB GPU. This serves an OpenAI-compatible API on port 8000:

docker run --gpus all \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model meta-llama/Llama-3.1-8B-Instruct

For a model too large for one card, split it across GPUs with tensor parallelism (set the size to your GPU count):

  --tensor-parallel-size 2

Verify it worked

Same protocol, different port. vLLM exposes the model list at /v1/models:

curl http://localhost:8000/v1/models
curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model":"meta-llama/Llama-3.1-8B-Instruct","messages":[{"role":"user","content":"Ping."}]}'

Coach’s Note — vLLM versions move weekly, and as of 2026 at least one recent release (v0.23.0) was pulled for a tensor-parallel bug shortly after shipping. Pin a known-good version line in production, read the release notes before you upgrade, and never let an unattended :latest tag decide which build runs your inference tonight.

B.4 — Point the OpenAI SDK at a Local Endpoint

Here is the payoff of “one protocol to rule them all.” The official openai Python SDK will happily talk to Ollama, vLLM, llama.cpp, LM Studio — anything that speaks the OpenAI-compatible API. You change exactly two things: the base_url and a api_key that, for a local server, is a throwaway dummy string.

Install the SDK into a project venv (see Appendix A):

pip install openai

Then point it at the Ollama daemon from B.1:

from openai import OpenAI

client = OpenAI(
    base_url="http://localhost:11434/v1",  # Ollama. vLLM: :8000/v1
    api_key="ollama",                       # local servers ignore this; any string works
)

resp = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Explain a load balancer in two sentences."}],
)
print(resp.choices[0].message.content)

That’s the whole trick. Want to run the same script against vLLM instead? Change base_url to http://localhost:8000/v1 and model to the vLLM model name. Against a different local runner? Swap the port: llama.cpp serves :8080/v1, LM Studio serves :1234/v1. The code in between never changes. When you graduate to the cloud in B.5, you’ll change the same two lines — plus, crucially, you’ll be handing over a real key, and real keys have to be handled with care.

Coach’s Note — Write your application against this interface and you’ve bought yourself portability. The model you prototype against on localhost and the frontier model you ship against in production differ by a base_url, a key, and a model ID. That is a governance win as much as an engineering one: you can move a workload on-prem for data-residency reasons, or off it for capability, without a rewrite.

B.5 — Cloud API Access: OpenAI, Azure, Bedrock, Vertex

When a workload needs a frontier model, more capacity than you can host, or a vendor’s managed agent runtime, you go to the cloud. As of 2026 the four major platforms — OpenAI, Microsoft Foundry (Azure), Amazon Bedrock, and Google Vertex AI — share a common shape: a model catalog, a pay-per-token serverless default, a reserved-capacity tier billed by time in a vendor-specific unit, and a managed agent runtime. (Costs are covered in Chapter 12; this appendix is about getting access.)

B.5.0 — Which one should you actually sign up for?

All four are documented below because you will meet all four in the field, and because the differences between them are course content. But for coursework, they are not equally easy to get into, and the friction has nothing to do with the learning:

Coach’s Note — Bedrock is not bad technology; it is bad onboarding, and those are different complaints. Its IAM-based auth is genuinely the most enterprise-realistic of the four, which is exactly why it is worth reading about. But three gates stand between a new student and their first successful call — account activation, IAM credentials, and per-model access approval — and each can stall for hours. Learn Bedrock’s shape here; do your graded deployments somewhere with one gate instead of three.

First, the rule that governs everything in this section.

B.5.1 — The security of keys (read this before you generate one)

An API key is a bearer credential: whoever holds it can spend your money and act as you, no questions asked. Treat it like the password to your bank.

• Never commit a key to git. Not to a private repo, not “temporarily,” not in a comment. Keys have been leaked by the millions through public commits; bots scan GitHub for them within seconds of a push. Put real keys in environment variables or a secrets manager (Azure Key Vault, AWS Secrets Manager, Google Secret Manager), and read them at runtime.
• Add the secret file to .gitignore before you create it. A typical pattern: keep keys in a .env file, and make sure your very first commit already ignores it:

printf ".env\n" >> .gitignore

Then in code, load from the environment — never hard-code:

import os
from openai import OpenAI

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

• Scope and rotate. Use the narrowest permissions the platform offers (project-scoped or restricted keys), set spend limits, and rotate keys on a schedule and immediately on any suspected leak.
• If a key leaks, revoke it first, investigate second. A live leaked key is an open faucet on your budget.

Coach’s Note — This is vocation, not paranoia. You are a steward of resources that aren’t ultimately yours — the organization’s budget, the users’ data, the trust placed in the system. A committed key is a small act of carelessness that becomes someone else’s large problem at 3 a.m. Diligence here is faithfulness in a little thing. Build the .gitignore-first, env-var-always habit until it’s reflex, and you’ll never be the cautionary tale.

B.5.2 — OpenAI

Create an account at platform.openai.com, add billing, and generate a key under API keys (prefer a project-scoped key). Then it’s the same SDK you already used locally — just no base_url override:

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ["OPENAI_API_KEY"])

resp = client.chat.completions.create(
    model="gpt-5.5",  # verify the current model ID; the GPT-5.x line moves fast
    messages=[{"role": "user", "content": "One-sentence health check."}],
)
print(resp.choices[0].message.content)

Set the key in your shell instead of in code:

export OPENAI_API_KEY="sk-...your-real-key..."

Two deprecations worth pinning, as of 2026: OpenAI is winding down self-serve fine-tuning, and the older Assistants API sunsets August 26, 2026 — new agentic work should target the Responses + Conversations APIs and the Agents SDK. Re-confirm both dates before you build on them.

B.5.3 — Microsoft Foundry (Azure)

Microsoft’s platform was renamed from Azure AI Foundry to Microsoft Foundry (effective January 1, 2026), though the underlying “Azure OpenAI” service name persists in many places — don’t be thrown by seeing both. You deploy a model into a resource and get a per-resource endpoint and key. Azure prefers the Entra ID (managed-identity) auth path over raw keys for production; use it where you can. With a key, the SDK pattern looks like:

import os
from openai import AzureOpenAI

client = AzureOpenAI(
    azure_endpoint=os.environ["AZURE_OPENAI_ENDPOINT"],  # https://<resource>.openai.azure.com
    api_key=os.environ["AZURE_OPENAI_API_KEY"],
    api_version="2024-10-21",  # pin an API version; verify the current one
)

resp = client.chat.completions.create(
    model="<your-deployment-name>",  # the deployment name you chose, not the raw model ID
    messages=[{"role": "user", "content": "Health check."}],
)
print(resp.choices[0].message.content)

Note the Azure-isms: you target your deployment name, not the bare model ID, and you pin an api_version. Reserved capacity here is sold in PTUs (provisioned throughput units); see Chapter 12.

B.5.4 — Amazon Bedrock

Bedrock is AWS’s managed model service, offering Claude, Llama 4, Amazon Nova 2, Mistral, Cohere, and more behind one API. Authentication is standard AWS IAM, not a single bearer key — you use IAM credentials/roles via the boto3 SDK and configure access with aws configure (or, better, an instance/role profile). You must also request access to specific models in the Bedrock console before you can call them.

import boto3, json

client = boto3.client("bedrock-runtime", region_name="us-east-1")

resp = client.converse(
    modelId="anthropic.claude-sonnet-4-...",  # verify the exact current model ID in-console
    messages=[{"role": "user", "content": [{"text": "Health check."}]}],
)
print(resp["output"]["message"]["content"][0]["text"])

Because auth is IAM, the security discipline shifts from “guard the key” to “least-privilege the role”: grant only the Bedrock actions and model ARNs the workload needs, and never bake long-lived access keys into code or images. Reserved capacity is sold in “model units”; Bedrock’s managed agent runtime is AgentCore.

B.5.5 — Google Vertex AI

Google’s platform is Vertex AI (the brand is migrating toward the Gemini Enterprise Agent Platform, announced April 2026, but “Vertex AI” persists in URLs and SDKs — expect both names). Auth uses Google Cloud IAM via Application Default Credentials (ADC) — a service account or gcloud auth application-default login — rather than a static key string. Enable the Vertex AI API on your project first.

from google import genai

# Reads ADC + GOOGLE_CLOUD_PROJECT / location from the environment
client = genai.Client(vertexai=True, project="your-project-id", location="us-central1")

resp = client.models.generate_content(
    model="gemini-3-flash",  # verify the live model ID; the Gemini ladder is crowded
    contents="Health check.",
)
print(resp.text)

Set up local credentials once with:

gcloud auth application-default login

Reserved capacity here is sold in GSUs (generative AI scale units). As with Azure and AWS, prefer IAM/identity-based auth over static secrets — it’s the more secure default and the one production should standardize on.

Coach’s Note — Notice the pattern across all four: OpenAI hands you a bearer key; the three hyperscalers nudge you toward identity-based auth (Entra, AWS IAM, Google IAM) where production should live. A key is a thing you can lose; an identity is a thing you can scope, audit, and revoke centrally. When you have the choice, choose identity. The most secure secret is the one you never had to store.

B.6 — Verify the Whole Toolkit

Prove the pieces work, end to end. You don’t need every cloud provider — pick what you’ll actually use — but you should have local working before you trust the cloud.

• Local model: ollama run llama3.1:8b answers a question.
• Local endpoint: the curl to http://localhost:11434/v1/chat/completions returns JSON.
• Front end: Open WebUI at http://localhost:3000 chats through Ollama, and a PDF upload answers a grounded question.
• (If GPU) vLLM at :8000/v1/models lists your model.
• SDK portability: the B.4 script runs unchanged against Ollama by editing only base_url and api_key.
• Cloud, safely: at least one cloud call works with the key read from an environment variable, and git status shows your .env is ignored, never staged.

If that last bullet is true — your secret is in the environment and out of git — you’ve learned the most important habit in this appendix. The model that answers is impressive. The key you didn’t leak is what keeps you employed.

When Things Go Wrong

• ollama: command not found after install — open a fresh terminal so PATH refreshes; on Linux confirm the service is up with systemctl status ollama.
• Open WebUI can’t see any models — it can’t reach Ollama. On macOS/Windows use OLLAMA_BASE_URL=http://host.docker.internal:11434; on Linux use --network=host and http://localhost:11434.
• vLLM exits with a CUDA / out-of-memory error — your model is too big for the card. Pick a smaller model or a quantized build, lower the context length, or add --tensor-parallel-size to spread it across more GPUs.
• SDK call hangs or refuses the connection locally — the daemon/server isn’t running, or you have the wrong port (Ollama :11434, vLLM :8000, llama.cpp :8080, LM Studio :1234).
• Cloud call returns 401 / 403 — bad or missing key, or (Bedrock/Vertex) you haven’t requested access to that model or enabled the API. For Azure, check that you’re targeting the deployment name and a valid api_version.
• “I created an AWS account but I can’t sign in.” In order of likelihood: (1) the sign-in page offers Root user vs IAM user — you created the account, so choose Root user and use your signup email; picking “IAM user” is the single most common cause. (2) Activation isn’t finished — AWS wants email confirmation, phone/SMS verification, and a valid payment method, normally minutes but occasionally up to 24 hours, and services stay unavailable until it completes. (3) Your card was declined (debit and prepaid cards frequently are), which suspends the account — look for the email. (4) You’re at an SSO/Identity Center start URL instead of the main console sign-in. Before you spend another hour on this: check whether your assignment actually requires AWS. For Chapter 12’s project it does not — see §B.5.0 and the project’s platform table.
• Bedrock says access denied even though you’re signed in — signing in is only gate two of three. Open the Bedrock console, go to Model access, and request the specific model in the specific region you’re calling; approval is not instant, and some models ask for a use-case statement. A different region is a different request.
• You committed a key by accident — revoke it in the provider console immediately (rewriting git history does not un-leak it; the bots already have it), then issue a new one and move it to an env var.

Up next: Appendix C — Agentic AI, where the model stops just answering and starts acting — with the human-in-the-loop rules that keep that safe. Or head back to the chapters and put a local model to work. See you in the gym.