Production engineering beyond conference demos: RAG, costs, data protection and responsible use of AI tools.
"AI amplifies those who think. It does not replace them."
Cesar Zanis
Code generation is increasingly accessible. The value of engineering extends beyond writing a CRUD application.
Professionals need to understand the business, design solutions and coordinate tools to deliver them. AI amplifies sound judgment; it does not replace it.
The quality of an AI response depends strongly on the quality of the context you provide.
Before opening an AI tool, work through the business problem. Who is the user? What is the real pain point? What happens without this feature?
'Build a registration system' gives the model little useful context. Explaining that a field inspector uses a Honeywell handheld with unreliable internet and must save forms without losing data, even if the battery dies, makes the constraints concrete.
Your experience supplies the context that makes the output useful.
Use reasoning models for architectural exploration: compare alternatives, analyze trade-offs, review ADRs and question decisions. The final decision remains yours.
Use editor-integrated agents for implementation: boilerplate, CRUD, tests and refactoring. They can work with real project files, execute commands and correct errors.
Do not delegate accountability for architecture to AI. It proposes; you decide.
The difference between a generic chatbot and a useful assistant.
RAG (Retrieval-Augmented Generation) retrieves relevant context before generating an answer. Without retrieval, a model relies on its training knowledge. Well-designed RAG supplies your documentation, knowledge base and decision history.
A common mistake is using vector search alone while ignoring lexical search such as BM25 or full-text search. Evaluate hybrid retrieval and reranking against a representative set of questions.
# A minimal working RAG architecture
# 1. Indexing (offline, when content changes)
def index_document(doc: str, doc_id: str):
# Chunking: overlapping blocks to preserve context
chunks = chunk_with_overlap(doc, size=800, overlap=100)
for i, chunk in enumerate(chunks):
embedding = embed(chunk) # via API: text-embedding-004
vector_store.upsert(
id=f"{doc_id}_{i}",
vector=embedding,
metadata={"text": chunk, "doc_id": doc_id}
)
# 2. Retrieval (in real time, per user query)
def retrieve(query: str, top_k: int = 5) -> list[str]:
query_embedding = embed(query)
# Semantic search (vectors)
semantic_results = vector_store.query(query_embedding, top_k=top_k)
# Lexical search (exact words): do not ignore it
lexical_results = full_text_search(query, top_k=top_k)
# Hybrid reranking: RRF (Reciprocal Rank Fusion)
return reciprocal_rank_fusion([semantic_results, lexical_results])
# 3. Generation: context injected into the prompt
def answer(query: str) -> str:
context_chunks = retrieve(query)
context = "\n\n".join(context_chunks)
prompt = f"""Responda baseado APENAS no contexto abaixo.
Se a resposta não estiver no contexto, diga explicitamente.
CONTEXTO:
{context}
PERGUNTA: {query}"""
return llm.generate(prompt)Measure input, output and context costs separately.
Production costs include system instructions, conversation history, retrieved context and generated output. Each grows differently.
Summarize history instead of resending every turn. Route straightforward tasks to smaller models and reserve larger models for tasks that need them. Evaluate response caching where appropriate, and set explicit output limits. Measure quality and cost together.
# Cost controls in production
# 1. Compress history: do not send everything
def compress_history(history: list[dict], max_turns: int = 4) -> str:
if len(history) <= max_turns:
return format_history(history)
# Summarize older turns and keep the last N messages in full
old_turns = history[:-max_turns]
recent_turns = history[-max_turns:]
summary = llm.generate(
f"Resuma em 2-3 frases os pontos principais desta conversa: {old_turns}",
config={"maxOutputTokens": 150} # short summary, small model
)
return f"[Resumo anterior: {summary}]\n{format_history(recent_turns)}"
# 2. Route by complexity: choose a model for each task
def route_to_model(query: str) -> str:
# Fast, inexpensive classification
is_complex = classifier.predict(query) # local or flash model
return "gemini-3.8-flash" if is_complex else "gemini-3.5-flash-lite"
# 3. Explicit output limit
generation_config = {
"maxOutputTokens": 600, # enough for a useful answer
"temperature": 0.3, # adjust temperature to the task and model
}Define the lawful basis and send only necessary data.
Before sending personal data to an AI API, define its purpose and applicable lawful basis, minimize the data, and review retention, access and international transfers. Consent is one lawful basis under Brazil's LGPD, not the only one. Training policies vary by product and contract; for example, OpenAI API data is not used for training by default. Evaluate anonymization, contractual controls and local processing for the situation. Local processing also requires secure access, logs and backups.
# Basic masking before sending data to an external LLM
import re
def anonymize_for_llm(text: str) -> tuple[str, dict]:
"""Remove PII antes de enviar. Retorna texto limpo + mapa para substituição."""
replacements = {}
counter = {"cpf": 0, "email": 0, "phone": 0}
# CPF
def replace_cpf(m):
key = f"[CPF_{counter['cpf']}]"
replacements[key] = m.group(0)
counter["cpf"] += 1
return key
# Email
def replace_email(m):
key = f"[EMAIL_{counter['email']}]"
replacements[key] = m.group(0)
counter["email"] += 1
return key
text = re.sub(r'\d{3}\.\d{3}\.\d{3}-\d{2}', replace_cpf, text)
text = re.sub(r'[\w.-]+@[\w.-]+\.\w+', replace_email, text)
return text, replacements
# Usage
clean_text, pii_map = anonymize_for_llm(user_message)
llm_response = llm.generate(clean_text)
# Never log pii_map: it contains the original data
# Use only if the response needs the original valuesKnow where each approach makes sense.
Local tools such as Ollama, vLLM and llama.cpp run models on infrastructure you control. They can suit classification, extraction and workloads with strict data-handling requirements. Check every dependency and telemetry path rather than assuming data never leaves.
Cloud platforms such as Vertex AI, AWS Bedrock and Azure OpenAI provide managed access to models without operating your own GPUs. Compare capability, privacy, latency and cost for the task.
A hybrid approach can use a local model for routing and a cloud model for harder requests, when the data policy permits it.
Abstract the provider behind an interface when multiple clouds are needed. Dependency Inversion applies here as it does to databases.
# Provider abstraction: switch clouds without changing business logic
from abc import ABC, abstractmethod
class LLMProvider(ABC):
@abstractmethod
def generate(self, prompt: str, config: dict) -> str: ...
# Google Cloud
class VertexProvider(LLMProvider):
def generate(self, prompt: str, config: dict) -> str:
from vertexai.generative_models import GenerativeModel
model = GenerativeModel(config.get("model", "gemini-3.5-flash-lite"))
response = model.generate_content(prompt)
return response.text
# AWS
class BedrockProvider(LLMProvider):
def generate(self, prompt: str, config: dict) -> str:
import boto3
client = boto3.client("bedrock-runtime")
# Claude via Bedrock
response = client.invoke_model(
modelId=config.get("model", "anthropic.claude-3-haiku"),
body={"prompt": prompt, "max_tokens": config.get("max_tokens", 600)}
)
return response["body"].read().decode()
# Local (Ollama)
class OllamaProvider(LLMProvider):
def generate(self, prompt: str, config: dict) -> str:
import requests
r = requests.post("http://localhost:11434/api/generate", json={
"model": config.get("model", "llama3.2"),
"prompt": prompt, "stream": False
})
return r.json()["response"]
# Model routing
def get_provider(task_type: str) -> LLMProvider:
if task_type in ("classify", "extract", "moderate"):
return OllamaProvider() # local execution; protect access and logs
if task_type == "generate":
return VertexProvider() # cloud, large model
return BedrockProvider() # multi-cloud fallbackA practical criterion: if you cannot explain what an agent does in one sentence, you do not yet understand it well enough to operate it in production. Simplicity is a sign of clarity.
Need help designing an AI solution for your business?
Talk to Cesar Zanis