“What should I read next?” seems like a simple question. But answering it well requires knowing what the user is currently reading, what they’ve finished recently, and what’s already on their list. When I built QuietReads, a book tracking app with an AI assistant, I faced a choice: pre-load all this context on every request (expensive), build complex routing logic to fetch the right data (brittle), or find a different approach entirely.
In those post, I talk about why I chose this option, the architectural implications, and the tradeoffs involved.
Conversational Interfaces Are Not Deterministic
A lot of mid-career technologists like me still think deterministically. We reach for decision trees, map out every probable permutation, and write test cases to cover each branch. This works well for forms and structured APIs where you control the inputs.
But QuietReads has a chat interface. Users might ask:
- “What should I read next?”
- “I’m in the mood for something like the last book I finished, but shorter”
- “What were my thoughts on that dystopian novel from last month?”
Each query requires different context. The first needs the user’s want-to-read list. The second needs their recently finished books plus some understanding of “shorter.” The third requires searching through their notes. I couldn’t predict which context any given question would need, and I didn’t want to fetch everything every time.
The traditional approach would be routing logic. For just the first query, you might write something like:
def get_context_for_recommendation(message, user_id):
context = {}
if contains_recommendation_intent(message):
context['want_to_read'] = get_want_to_read_books(user_id)
context['recently_finished'] = get_recently_finished(user_id)
if mentions_specific_book(message):
book = extract_book_reference(message)
context['book_details'] = get_book_details(book)
# ... and this continues for every intent type
return contextThis gets unwieldy fast. Each new question type requires new routing rules. The intent detection functions themselves need maintenance. And you’re constantly guessing what context the model will need.
LLMs allow us to use a different mental model.
Think of LLMs as expert Lego assemblers. You provide a curated set of bricks (tools), an instruction manual (your system prompt), and let the assembler determine which bricks to use and in what order. You don’t hand them every brick in existence. You give them the right pieces for the task and clear guidance on when to use each one.
Tools as Building Blocks
In QuietReads, I define “context tools” that let the AI retrieve user data as needed:
CONTEXT_TOOLS = [
{
"name": "get_user_profile",
"description": (
"Get the user's name and reading preferences. Use this when you need to "
"personalize your response or discuss their reading interests."
),
"input_schema": {"type": "object", "properties": {}}
},
{
"name": "get_want_to_read",
"description": (
"Get books on the user's want-to-read list. IMPORTANT: Always call this "
"BEFORE recommending any books to avoid suggesting books they already have."
),
"input_schema": {
"type": "object",
"properties": {
"limit": {"type": "integer", "description": "Max books to return (default: 10)"}
}
}
}
]
Each tool has a clear description of when to use it. The model reads these descriptions and decides which tools to call based on the user’s question.
The Agentic Loop
When the model decides to use a tool, we handle that request, execute the tool, and feed the results back. This creates a loop (simplified code below):
async def execute(self, system, messages, tools, tool_handlers, max_tokens=2048):
response = self.client.messages.create(
model=self.model,
max_tokens=max_tokens,
system=system,
messages=messages,
tools=tools
)
while response.stop_reason == "tool_use":
tool_results = []
for block in response.content:
if block.type == "tool_use":
tool_name = block.name
tool_input = block.input
# Execute the tool and capture result
result = await tool_handlers[tool_name](tool_input)
tool_results.append({
"type": "tool_result",
"tool_use_id": block.id,
"content": result
})
# Feed results back and get next response
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": tool_results})
response = self.client.messages.create(...)
return responseThe model might call multiple tools, or call the same tool with different parameters, or decide it has enough context after the first call.
The loop continues until the model has everything it needs to answer. In practice, you should also enforce a maximum iteration count as a guardrail against runaway loops or unexpectedly expensive queries. When the model requests multiple tools in a single response, you can execute them in parallel for performance gains.
This code handles the happy path. Production implementations need additional safeguards: error handling when tools fail or timeout, validation of tool inputs before execution, and graceful handling when the model hallucinates a tool name that doesn’t exist.
Client and Server-side Tools
QuietReads uses two categories of tools. Client-side tools are functions I implement: when the model calls get_want_to_read, my code queries the database and returns formatted results.
Server-side tools are capabilities the AI provider offers (like the web_search tool used below).
I enable web search so the assistant can look up recent book releases or author news. Anthropic’s infrastructure handles the search; I just control when and how it’s available:
tools:
web_search:
type: "web_search_20250305"
name: "web_search"
server_side: true
enabled: true
config:
max_uses: 5 # Limit searches per requestWith some prompt engineering, the model can integrate custom database queries with real-time web searches, producing responses that feel coherent to the user.
The system prompt guides how the model synthesizes information from different sources, when to cite web results versus personal reading history, and how to maintain a consistent voice across tool-augmented responses.
Architectural Implications
It is important to recognize where determinism matters and where it doesn’t. Each tool is a testable piece of code. I can unit test get_want_to_read in isolation, verify it returns the right data, and trust it to behave consistently. What I can’t fully predict is which tools the model will call or in what order. In my work, I have found that even cheaper models like the Haiku family of models do a decent job at tool use.
This separation has practical implications. Tool descriptions are instructions the model uses to decide when to call each tool. Writing clear, specific descriptions is as important as the implementation itself. Instead of pre-loading everything a user might need, I provide minimal context upfront and let the model request more, keeping initial requests fast and reducing token costs.
And while the model chooses its tools, I still control the boundaries. QuietReads runs input guardrails before messages reach the assistant and can validate outputs before returning them to the user.
The Tradeoffs
Tool calling introduces real costs that you should weigh against your specific requirements.
- Predictability. With static context, you know exactly what data the model sees on every request. With tool calling, the model decides what to retrieve. This makes cost and performance harder to predict. A simple question might resolve in one API call; a complex one might trigger four tool calls and five round-trips.
- Prompt caching. Static context can benefit significantly from prompt caching, where repeated system prompts are stored and reused. Dynamic tool results change with each request, which can reduce or eliminate caching benefits. Depending on your usage patterns, this could meaningfully impact both latency and cost.
- Quality assurance. Unit testing individual tools is straightforward, but testing the system end-to-end becomes harder. The model might call tools in unexpected combinations, or skip tools you expected it to use. Comprehensive evaluations become essential because tool calling adds non-determinism to the critical path. I’ll write more about evaluation strategies in a future post.
- Refactoring risk. IDE tooling can automatically update function signatures across a codebase. Tool definitions live in JSON objects that describe behavior and parameters in natural language. If you change a tool’s behavior or modify its parameters, automated refactoring won’t catch the JSON definitions, and the mismatch may not surface until production. LLM-based coding agents like Claude Code handle this well, and adding tool-specific checks to code review agents helps catch these issues.
- Latency. Each iteration of the agentic loop requires a round-trip to the API. For QuietReads, this is acceptable. For applications where response time is critical, the additional latency may be a dealbreaker.
That said, tool calling offers real advantages beyond flexibility. Token costs can decrease because the model only retrieves data it actually needs. Direct tool calls avoid the protocol overhead of intermediary layers like MCP servers. And tools create a clean separation of concerns: database migrations, API upgrades, or new data providers can happen without touching the prompt.
This pattern fits QuietReads: read-only tools, flexible latency, and context costs that exceed API overhead. Applications with side effects, strict latency, or predictable context needs may want different approaches.
Navigating a Mindset Shift
Building with tool calling requires accepting the risk of non-deterministic code execution. You cannot predict every code path. Instead of mapping out decision trees, you’re designing capabilities and constraints. You’re giving the model a well-stocked toolbox and clear guidance, then trusting it to assemble the right response.
You control what tools exist, what data they access, what the model knows about when to use them, and what guardrails prevent misuse. The model handles the dynamic orchestration that would otherwise require hundreds of lines of if/else chains.
For those of us who’ve spent years thinking in flowcharts, this shift takes practice. But once it clicks, you start asking different questions: not “what are all the paths a user might take?” but “what capabilities does the model need, and how do I describe when to use them?”
