Proxy LLM traffic¶

Building on the agent example from the previous section, we wish to put a proxy between the agent and the LLM. Doing so has a number of benefits, including:

• It allows us to observe requests made to the LLM.
• The proxy can act as a control point to authorize certain requests but not others.
• The proxy can be made to limit or cap usage to a certain number of tokens.

Install agentgateway¶

Install the agentgateway binary, version 1.5.0:

curl -sL https://agentgateway.dev/install | bash -s -- --version 1.5.0

Verify that the agentgateway binary is in the PATH:

agentgateway --version

Configure the proxy¶

Agentgateway can be configured to call a variety of models and providers.

By default agentgateway listens for LLM requests on port 4000.

Run the command below and review the configuration for agentgateway:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  database:
    url: sqlite://./data.db
llm:
  models:
  - name: trend-pro
    provider: openAI
    params:
      baseUrl: http://localhost:11434/v1
      model: qwen3:8b
      apiKey: ollama-does-not-check-this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  database:
    url: sqlite://./data.db
llm:
  models:
  - name: trend-pro
    provider: gemini
    params:
      model: gemini-3.5-flash-lite
      apiKey: "$GEMINI_API_KEY"

In general, configuring the apiKey at the proxy has an advantage: the key is never exposed to end users, it is maintained by the platform team.

Start the proxy:

agentgateway -f configs/llm-basic.yaml

agentgateway -f configs/llm-basic-gemini.yaml

Open a second terminal, and activate the python virtual environment:

source .venv/bin/activate

In that second terminal, configure the trendwatch agent to point at the proxy when calling the LLM:

export LLM_BASE_URL=http://localhost:4000/v1
export LLM_MODEL=trend-pro
export MCP_URL=stdio:./mcp-servers/trends_server.py

These three values are the same whether you chose Ollama or Gemini. The agent still speaks the OpenAI API; only the base URL and model name changed. Agentgateway maps trend-pro onto the real provider.

Finally, try running the agent once more:

python3 agent/trendwatch.py "what discussions are trending today?"

In agentgateway's logs, you can see that it captures the requests to the LLM along with metadata such as the model that was called, token consumption, and more. What agentgateway captures is configurable, and can even include the user prompt and the response.

Explore the agentgateway UI¶

Agentgateway provides a rich user interface accessible by default at http://localhost:15000/ui.

In the UI's home page, note that the "LLM" section is Enabled.

From the navigation panel, click on "Models", and note how the model named "trend-pro" is configured and maps to the qwen3:8b outgoing model.

Click on "Logs", and note how the calls to the LLM on each term has been captured, along with latencies, requested and outgoing models, and both input and output token usage.

Configure token budgets¶

In the first terminal, press Ctrl+C to terminate agentgateway.

Review the following proxy configuration:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  database:
    url: sqlite://./data.db
llm:
  policies:
    localRateLimit:
    - maxTokens: 5000
      tokensPerFill: 5000
      fillInterval: 300s
      type: tokens
  models:
  - name: trend-pro
    provider: openAI
    params:
      baseUrl: http://localhost:11434/v1
      model: qwen3:8b
      apiKey: ollama-does-not-check-this

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

# yaml-language-server: $schema=https://agentgateway.dev/schema/config
config:
  database:
    url: sqlite://./data.db
llm:
  policies:
    localRateLimit:
    - maxTokens: 5000
      tokensPerFill: 5000
      fillInterval: 300s
      type: tokens
  models:
  - name: trend-pro
    provider: gemini
    params:
      model: gemini-3.5-flash-lite
      apiKey: "$GEMINI_API_KEY"

The above configuration adds a policy to rate-limit requests. The rate limiting is specified in terms of number of tokens used, tokens being the native "metric" of utilization for LLMs.

maxTokens is set artificially low at 5000 tokens over a 5-minute (300 second) interval, to make it easy to test:

Restart agentgateway with the updated configuration file:

agentgateway -f configs/token-budget.yaml

agentgateway -f configs/token-budget-gemini.yaml

From the other terminal, try to run the agent a few times to trigger the rate limit:

for i in {1..3}; do
  python3 agent/trendwatch.py "what discussions are trending today?"
done

The first session should succeed, but as the token utilization exceeds the limit, we start receiving HTTP 429 (Rate Limited) responses. The agentgateway logs also shows entries with http.status=429, indicating that the request was rate-limited.

Agentgateway has many more features and capabilities pertaining to management of utilization, and cost control.

With agentgateway we can:

• Monitor and control usage in terms of money rather than tokens, by configuring a cost catalog for a variety of models.
• Attribute utilization by team and user, giving some teams higher budgets than others.
• Configure budgets per team and user, and in monetary terms instead of tokens.

Summary¶

In this exercise, you have placed agentgateway in between the agent and the LLM, obtained visibility over calls to LLMs, and configured token budgets.

In the next section, we build on this foundation and demonstrate the value of also placing agentgateway in front of the MCP tool servers.