Error: connection refused
Error: timeout exceeded
Info: request completed

No timestamp context. No request ID. No indication of which service, which user, or which downstream dependency was involved. I was reading thousands of lines of text, trying to mentally correlate entries by their position in the file. Like solving a puzzle where half the pieces are from a different box.

That experience changed how I think about logging. Not as an afterthought you sprinkle into your code, but as infrastructure you design up front. The same way you’d design your database schema or your API contracts.

The problem with unstructured logs

Most applications start with console.log or fmt.Println. And honestly, that’s fine for a single process running on your laptop. You can read the output. You know what’s happening because you just wrote the code.

But the moment you have two services talking to each other, unstructured text logs become almost useless. You can’t filter them. You can’t correlate them across services. You can’t aggregate them into dashboards. You end up grepping through gigabytes of text, hoping the timestamp and some keyword will be enough.

The core issue is that unstructured logs are optimized for humans reading a single stream in real time. Production systems don’t work that way. You have dozens of processes, thousands of requests per second, and you’re usually looking at the logs hours after the problem happened.

Structured logging solves this by treating every log entry as a data record with typed fields, not a sentence for a human to read.

What structured logging actually looks like

Instead of this:

User 4521 failed to authenticate: invalid password

You produce this:

{
  "timestamp": "2025-01-15T14:32:01.442Z",
  "level": "warn",
  "message": "authentication failed",
  "user_id": "4521",
  "reason": "invalid_password",
  "ip": "192.168.1.42",
  "service": "auth-api",
  "request_id": "req-a8f3c"
}

Same information. But now every field is queryable. You can ask your log aggregator: “show me all authentication failures in the last hour, grouped by reason.” Or: “show me every log entry with request_id: req-a8f3c across all services.” You couldn’t do either of those with plain text.

The format doesn’t have to be JSON. Some teams use logfmt (the key=value format popular in the Go ecosystem). The point is that the log entry has a predictable structure with named fields.

Setting up structured logging in practice

I’ll show this in two languages because the patterns are the same, but the ergonomics differ.

Go (standard library, slog)

Go added the log/slog package in version 1.21. Before that, most teams used third-party libraries. Now there’s a good option in the standard library.

package main

import (
	"log/slog"
	"os"
)

func main() {
	logger := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{
		Level: slog.LevelInfo,
	}))

	slog.SetDefault(logger)

	slog.Info("server starting",
		"port", 8080,
		"environment", "production",
	)
}

This outputs a JSON line with timestamp, level, message, and your custom fields. No extra dependencies.

Where slog really shines is when you create child loggers with pre-bound fields:

func handleRequest(w http.ResponseWriter, r *http.Request) {
	requestID := r.Header.Get("X-Request-ID")
	if requestID == "" {
		requestID = generateID()
	}

	log := slog.With(
		"request_id", requestID,
		"method", r.Method,
		"path", r.URL.Path,
	)

	log.Info("request received")

	user, err := authenticate(r)
	if err != nil {
		log.Warn("authentication failed", "error", err.Error())
		http.Error(w, "unauthorized", http.StatusUnauthorized)
		return
	}

	log = log.With("user_id", user.ID)
	log.Info("user authenticated")

	// Every subsequent log in this handler carries request_id, method, path, and user_id
}

Every log line from this handler now carries the request ID, the HTTP method, the path, and (once authenticated) the user ID. You don’t pass these fields around manually to every function call; you build them up as context accumulates.

Node.js (pino)

In Node.js, pino has been the go-to structured logger for years. It’s fast (it writes logs asynchronously by default) and outputs JSON.

const pino = require('pino');

const logger = pino({
  level: process.env.LOG_LEVEL || 'info',
});

logger.info({ port: 8080, environment: 'production' }, 'server starting');

The child logger pattern works the same way:

function handleRequest(req, res) {
  const requestId = req.headers['x-request-id'] || generateId();

  const log = logger.child({
    request_id: requestId,
    method: req.method,
    path: req.url,
  });

  log.info('request received');

  try {
    const user = authenticate(req);
    const userLog = log.child({ user_id: user.id });
    userLog.info('user authenticated');
    // use userLog for the rest of this request
  } catch (err) {
    log.warn({ error: err.message }, 'authentication failed');
    res.writeHead(401);
    res.end('unauthorized');
  }
}

The pattern is identical across languages. Create a logger, bind context fields as you learn them, and every log entry automatically carries that context.

Correlation IDs: the thing that makes distributed debugging possible

I mentioned request_id in both examples. This is probably the single most impactful thing you can add to your logging infrastructure. I’m sure about this one.

The idea is simple. When a request enters your system, you generate a unique ID (or read one from an incoming header). Every service that touches this request includes that ID in its logs. When something goes wrong, you search for that one ID and get the full story across every service.

Here’s the thing that trips people up: you have to propagate it. If Service A calls Service B, A needs to include the correlation ID in the outgoing request header. B needs to read it and attach it to its own logger.

// Service A: outgoing call
func callServiceB(ctx context.Context, requestID string) error {
	req, _ := http.NewRequestWithContext(ctx, "GET", "http://service-b/data", nil)
	req.Header.Set("X-Request-ID", requestID)
	resp, err := http.DefaultClient.Do(req)
	// ...
}

// Service B: incoming middleware
func correlationMiddleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		requestID := r.Header.Get("X-Request-ID")
		if requestID == "" {
			requestID = generateID()
		}

		ctx := context.WithValue(r.Context(), "request_id", requestID)
		next.ServeHTTP(w, r.WithContext(ctx))
	})
}

The header name is a convention, not a standard. X-Request-ID is common. Some teams use X-Correlation-ID or X-Trace-ID. Pick one and stick with it everywhere. The name matters less than consistency.

If you’re using a message queue (Kafka, RabbitMQ, SQS), put the correlation ID in the message metadata too. Async boundaries are where correlation breaks down most often, and that’s exactly where you need it most.

Log levels: fewer than you think

I’ve seen logging configs with eight or ten levels. In practice, I use four:

• debug is for development. Verbose, noisy, turned off in production unless you’re actively investigating something.

• info records normal operations. Request received, request completed, job started, job finished. The “everything is fine” level.

• warn means something unexpected happened, but the system handled it. A retry succeeded. A cache miss on something you expected to be cached. A deprecated endpoint got called.

• error means something failed and a human should probably know about it. A database query timed out and the request returned a 500. A downstream service is unreachable. Payment processing failed.

I’ve seen teams debate whether they need fatal or critical as separate levels. I think that’s overthinking it. If your process is about to crash, log it as error with a field like ”fatal”: true and let your alerting system pick it up.

The more important discipline is not *which* level to use but making sure you actually use them consistently. I’ve worked on codebases where error was used for input validation failures. That meant the error rate dashboard was always noisy, and real errors got buried. That’s a people problem, not a tools problem, but you can make it easier by writing down what each level means for your team.

What to log (and what not to)

This is where I see the biggest mistakes.

Log too little and you can’t debug anything. Log too much and you can’t find anything (and your log storage bill gets alarming).

Things I always log:

Incoming requests (method, path, status code, duration). Outgoing calls to other services or databases (target, duration, success or failure). Business events that matter (order created, payment processed, user signed up). Errors and the context around them.

Things I never log:

Request or response bodies in production (too much data, and you will accidentally log passwords or tokens). Personally identifiable information without redaction. Secrets, API keys, or database credentials. Health check requests (they flood your logs with noise and they tell you nothing useful).

The PII point deserves emphasis. I’ve seen production logs that included full email addresses, phone numbers, and once even credit card numbers. Structured logging makes this worse because it’s so easy to just spread an object into your log fields. You need explicit allowlists or redaction for any user-facing data.

// Don't do this
log.info({ user: req.body }, 'user signup');

// Do this
log.info({
  user_id: user.id,
  email_domain: user.email.split('@')[1],
  plan: user.plan,
}, 'user signup');

From logs to observability

Structured logs are one pillar. On their own, they’re a massive improvement over unstructured text. But they’re not the full picture.

The industry generally talks about three signals: logs, metrics, and traces. I think of it differently. Logs tell you *what* happened. Metrics tell you *how much* is happening. Traces tell you *where time was spent*.

You don’t need all three on day one. Start with structured logs and correlation IDs. That alone will solve 80% of your debugging problems. I’m pretty confident about that number based on my own experience, though your mileage will vary.

When you’re ready to add metrics, the pattern is similar to structured logging: named, typed, labeled data points. Request duration as a histogram, labeled by service and endpoint. Error counts, labeled by error type. Queue depth over time. The same fields you put in your logs (service name, endpoint, environment) should appear as labels on your metrics.

Traces are the most complex to set up. They require instrumentation at every network boundary, and the tooling is still maturing. OpenTelemetry has become the standard way to collect all three signals, and most popular frameworks have integrations for it. But I’ll be honest: I’ve only set up tracing on one production system, and it took significantly more effort than I expected. Worth it for complex distributed systems with many hops. Probably overkill for a two-service architecture.

Operational patterns that actually help

A few things I’ve learned the hard way.

First, always include the service name and environment in every log entry. This sounds obvious, but when you’re aggregating logs from twelve services into one system, and someone forgot to tag their logs with the service name, you’ll spend twenty minutes figuring out where a log line came from.

Second, log at the boundaries. When a request enters your service, log it. When it leaves (response sent), log it with the duration and status code. When you call another service, log the call and the result. If you do nothing else, boundary logging with correlation IDs gives you a timeline of every request through your system.

Third, make your log pipeline resilient. Your application should not crash because the log aggregator is down. Write to stdout, let a sidecar or agent handle shipping. If the agent falls behind, you lose some logs. That’s better than losing the service.

Remember that correlation ID propagation I talked about earlier? It becomes even more important here. When you have boundary logging on every service, the correlation ID is the thread that stitches those boundary events into a coherent story. Without it, you have a pile of disconnected entries. With it, you have a timeline.

Fourth, and I could be wrong about this being universally true, but I’ve found that a single log entry at the end of a request with all the context (duration, status, user ID, any errors) is more useful than many small entries scattered throughout. Some teams call this the “request summary log.” It gives you one searchable record per request, which makes aggregation and alerting much simpler.

The mistake I keep seeing

Teams invest in log aggregation tooling but never invest in log quality. They set up Elastic Search or Loki or whatever, pipe everything in, build dashboards. Then when something breaks, they search the logs and find entries like:

Something went wrong
Error occurred
null

The aggregation system is only as good as what you put into it. If your log entries don’t have context, no amount of tooling will help. Spend the time making every log entry useful on its own. Include the operation that failed, the inputs that caused it (redacted if sensitive), and what the system did about it.

// This is useless
slog.Error("database error")

// This is useful
slog.Error("failed to fetch user profile",
    "user_id", userID,
    "query_duration_ms", elapsed.Milliseconds(),
    "error", err.Error(),
    "retry_attempted", true,
    "retry_succeeded", false,
)

The second entry tells you everything. Who was affected, how long the query ran, whether a retry was attempted, and whether it worked. That’s the difference between “something broke” and “I know exactly what broke and for whom.”

Where to start if you have nothing

If you’re starting from scratch or retrofitting an existing system, here’s what I’d do in order:

1. Pick a structured logger for your language. In Go, use log/slog. In Node.js, pino is the safe choice. In Python, the standard logging module with a JSON formatter works fine. Don’t build your own.

2. Add a correlation ID middleware. Generate an ID on the first service that receives a request. Propagate it through headers on every outgoing call.

3. Log at every boundary. Request in, request out, external call made, external call returned.

4. Write to stdout. Let the infrastructure handle shipping logs somewhere searchable.

5. Agree with your team on what each log level means. Write it down. Two sentences per level is enough.

That’s it for the first pass. You can add metrics, traces, and fancier tooling later. But these five steps will transform your ability to debug production issues. I’ve done this exact sequence on three different projects now, and each time the improvement was immediate.

The honest truth is that structured logging isn’t exciting. Nobody’s going to write a blog post about how your JSON log lines are a breakthrough. But the first time you search for a correlation ID and see the entire request path across four services in under a minute, you’ll wonder how you ever worked without it.

The whole thing cascades in about four minutes. Twelve minutes of availability can be gone before someone manually scales down the traffic.

The part worth dwelling on isn’t the outage itself. It’s how invisible the problem is before it happens. Most teams have dashboards for CPU, memory, error rates, request latency. Nobody is watching connection pool utilization. And the failure mode isn’t gradual. It’s a cliff.

This is worth thinking about whenever a team spins up a new service without thinking carefully about how it manages connections to its dependencies. So I want to walk through what actually happens when connection pools go wrong, why the defaults are almost always wrong for production, and what recovery looks like when you’re already in the hole.

What connection pooling actually solves

Opening a new database connection is expensive. For PostgreSQL, each connection spawns a new backend process. There’s a TCP handshake, TLS negotiation if you’re using encryption (you should be), authentication, and some session setup. That easily takes 20 to 50 milliseconds, sometimes more. If every incoming request opens a fresh connection and closes it when done, you’re burning a lot of time and putting real pressure on the database server’s process table.

A connection pool solves this by keeping a set of pre-established connections ready. Your application checks one out, uses it, returns it. The next request grabs the same connection without paying the setup cost.

Simple idea. The complexity is in the details.

The defaults will hurt you

Most connection pool libraries ship with defaults that work fine for development and fall apart under load. This pattern shows up across languages and databases. The pool size is set to something like 10 or 20, the connection timeout is either infinite or very long, and there’s no limit on how long a connection can be held.

Here’s the thing: the right pool size depends on your database, your query profile, your concurrency model, and your infrastructure. There’s a well-known formula from the PostgreSQL wiki that suggests connections = (core_count * 2) + effective_spindle_count for the database side. For SSDs, the spindle count is effectively 1. So a 4-core database server might optimally handle around 9 to 10 concurrent active queries. Not 100. Not 200.

But most teams don’t think about it from the database’s perspective. They think about it from the application side: “I have 50 concurrent requests, so I need 50 connections.” That math ignores the fact that the database itself becomes slower with too many concurrent connections. Context switching, lock contention, shared buffer pressure. More connections past the sweet spot means every query gets slower, which means connections are held longer, which means you need more connections. It’s a death spiral.

The approach that tends to work well for most web applications is to keep the per-instance pool small (something like 5 to 15 connections per application instance) and use an external connection pooler like PgBouncer in front of PostgreSQL when you have many application instances. PgBouncer multiplexes hundreds of application-side connections onto a much smaller number of actual database connections. It’s been around for ages, it’s battle-tested, and it handles the mismatch between “lots of app instances” and “database that works best with fewer connections” really well.

What exhaustion looks like from the inside

Let me walk through a concrete scenario. Here’s the sequence of events in a typical incident.

Picture a marketing campaign that drives a traffic spike around 2x normal peak. Not extreme. The app servers can handle the request volume. But several of the hot endpoints run queries that, under normal load, complete in 5 to 15 milliseconds. Under the increased concurrency, those queries start competing for the same rows and indexes. Lock contention pushes some of them to 200, 500, even 800 milliseconds.

Each slow query holds a connection from the pool for that entire duration. Say the pool size is 20 per instance. With 4 instances, that’s 80 connections total. PostgreSQL’s max_connections is set to 100 (the default). You’re already close to the ceiling.

As queries slow down, connections aren’t returned fast enough. New requests come in and wait for a connection. If the pool’s checkout timeout is set to 30 seconds, which is absurdly long, requests just sit there. Waiting. The client-side timeout on the API gateway is 10 seconds, so users see errors, but the server-side requests are still holding spots in the queue.

Health check endpoints share the same connection pool. When the pool is exhausted, health checks can’t get a connection, time out, and Kubernetes marks the pods as unhealthy. Rolling restarts kick in. New pods come up, try to establish 20 connections each to PostgreSQL, push the total connection count over max_connections, get rejected, and crash.

This is the part that tends to catch people off guard. The recovery mechanism (pod restarts) makes the problem worse. Each restart attempt consumes connections during initialization, adding pressure to a database that’s already at its limit.

The fixes, in order

The fix breaks into roughly three phases: stop the bleeding, tune the pools, then add visibility so you see it coming next time.

Immediate stabilization

First, manually reduce the replica count to 2 instances. Fewer instances mean fewer total connections. The database can breathe again. Queries start completing normally once contention drops.

Then kill the long-running queries manually using pg_terminate_backend. Ideally you’d have a statement timeout configured in PostgreSQL itself from the start; if not, this is the moment you wish you did. Setting statement_timeout to 5 seconds for the application role is a good baseline. If a query runs longer than that, something is wrong and it’s better to fail fast than hold connection hostage.

Pool configuration changes

Reduce the pool size per instance from 20 to 8. This feels counterintuitive at first. Fewer connections per instance? But with 6 instances running (scale up the instance count instead), that’s 48 application-side connections. More than enough for typical throughput, and well within what PostgreSQL handles efficiently.

Set the checkout timeout to 2 seconds. If a request can’t get a connection within 2 seconds, it fails immediately with a clear error. This is important. A short checkout timeout converts invisible queuing into visible errors. You can monitor error rates. You can’t easily monitor “requests silently waiting in a pool queue.”

Add a connection max lifetime. Connections that have been open for more than 30 minutes get recycled. This prevents problems with stale connections, server-side session bloat, and helps after database failovers where old connections might be pointing at the wrong host.

Add idle connection cleanup too. If a connection has been sitting unused for more than 5 minutes, close it. No reason to hold resources you’re not using.

Health check isolation

This one is worth repeating until it sticks. Your health check endpoint should never depend on the same resource pool as your business logic. Move health checks to use a separate, tiny connection pool (2 connections). Some teams just check if the process is alive without hitting the database at all for liveness probes, and only check the database for readiness probes. Either approach works. The point is that your orchestrator’s ability to assess your application’s health shouldn’t compete with actual traffic.

Remember that cascading pod restart problem? It goes away once health checks have their own pool. Even when the main pool is under pressure, readiness probes can still respond, so Kubernetes stops killing pods that are actually making progress.

This isn’t just about databases

I’ve focused on PostgreSQL here because that’s where this pain shows up most vividly. But connection exhaustion follows the same pattern everywhere you maintain persistent connections or limited resource pools.

Redis connection pools behave similarly. HTTP client pools for outbound service calls have the same dynamics. gRPC channels maintain underlying connections that can get saturated. Thread pools in worker-based architectures (think Java or .NET) exhibit identical cliff-edge behavior when saturated.

The pattern is always the same:

1. A resource pool has a fixed size.

2. Under load, consumers hold resources longer than expected.

3. New consumers can’t acquire resources and start queuing.

4. The queue grows faster than resources are released.

5. Timeouts (or lack thereof) determine whether the failure is visible and fast, or invisible and cascading.

I could be wrong about this, but I think most production outages trace back to some form of resource exhaustion rather than the thing people usually blame first (like “the database was slow”). The database was slow because it had too many connections. The service was slow because the HTTP client pool was saturated. The root cause is usually the pool management, not the downstream system.

What to actually monitor

There are a few metrics I now consider non-negotiable for any service that talks to a database or external dependency.

Pool utilization as a percentage: active connections divided by max pool size. Alert when this stays above 80% for more than a minute. Not on a spike. On sustained pressure. Brief spikes to 90% during a burst are normal. Sitting at 85% for five minutes means you’re one slow query away from exhaustion.

Checkout wait time: how long requests spend waiting for a connection. If this number is usually zero and suddenly it’s not, something changed. This is the metric that catches slow-burn pool pressure — wait times creep up, you investigate, you find something like a missing index on a new table, and you fix it before users notice anything.

Connection creation rate: how often the pool is creating new connections. A sudden spike in new connection creation can indicate connection churn (connections being closed and reopened rapidly) or a failover event where all connections got invalidated at once.

Pool error count: connection checkout timeouts, failed connection attempts, connections rejected by the database. These should normally be zero. Any non-zero value is worth investigating.

A note on connection poolers

I mentioned PgBouncer earlier. I want to be clear about when it’s worth adding and when it’s just more complexity.

If you have 2 to 4 application instances with well-tuned pool sizes, you probably don’t need PgBouncer. The total connection count stays manageable, and the extra hop adds latency (small, but it’s there) and another component to operate.

If you have 20 or 50 or 200 instances (common in Kubernetes deployments with aggressive autoscaling), you almost certainly need something between your apps and the database. Without it, autoscaling events can overwhelm PostgreSQL’s connection limit. PgBouncer in transaction mode works well here. It assigns a real database connection only for the duration of a transaction, then returns it to the shared pool. This means 200 application instances can share, say, 50 real database connections.

The gotcha with transaction-mode pooling is that you can’t use session-level features: prepared statements (in some configurations), SET commands, LISTEN/NOTIFY, temp tables. If your application relies on those, you need to think carefully about which pooling mode to use. Teams can burn days debugging weird behavior that traces back to PgBouncer silently swapping the underlying connection mid-sesson.

What’s worth doing up front

The fixes for this kind of outage are all things you can configure from day one. Statement timeouts, reasonable pool sizes, checkout timeouts, separate health check pools, basic pool monitoring. None of it is novel. None of it requires new technology.

The problem is that connection pooling rarely gets treated as something that needs active design. It’s left as a library default. Set it up once, forget about it. And that works right up until it doesn’t, and when it stops working, it stops all at once.

If you’re setting up a new service today, it’s worth spending maybe an hour on pool configuration before writing any business logic. Set the pool size based on the database’s capacity divided by the expected number of instances. Set a checkout timeout of 1 to 3 seconds. Set a statement timeout on the database role. Set up the four metrics listed above. Wire health checks to a separate pool or at least a separate connection.

That hour saves you from a 2 a.m. page six months later. I’m pretty confident about that.

The other thing worth doing, and it’s the kind of thing that’s easy to skip until you’ve watched this pattern play out somewhere, is load testing specifically for pool exhaustion. Not just “can the system handle 10,000 requests per second” but “what happens when a downstream dependency gets slow.” Inject latency into your database calls and watch what the pool does. Watch where the cliff is. You want to find it in a test environment at 3 p.m., not in production at midnight.

Connection pools are one of those things that feel boring until they’re the only thing that matters. Treat them like infrastructure, not like library configuration, and they’ll stay boring. Which is exactly what you want.

Then you deploy it. And it calls the same tool four times in a loop because the LLM hallucinated a retry instruction. Or it silently eats an error and returns a confident, completely wrong answer to your user. Or it runs for 47 minutes burning tokens on a task that should take 10 seconds.

I’ve been building agent-based systems for the past few months, and the gap between “works in my notebook” and “runs in production without waking me up” is enormous. This is my attempt to write down what I’ve actually learned about closing that gap. Some of this I’m confident about. Some of it I’m still figuring out.

The problem nobody talks about in agent tutorials

AI agents are stateful, non-deterministic processes that make decisions at runtime. That sentence sounds obvious, but it has consequences that most tutorials skip.

A traditional API endpoint receives a request, does some work, returns a response. The work is predictable. You can write tests for it. You can set timeouts. You know the blast radius.

An agent is different. It decides what to do next based on LLM output, which means you can’t fully predict the execution path. It might call one tool or five. It might finish in 2 seconds or loop for a minute. It might encounter an error from an external API and decide (on its own) to retry, or to try a completely different approach, or to give up and hallucinate an answer.

This is why durability matters so much for agents. Not durability in the “survives a server restart” sense (though that too), but durability in the broader sense: the agent should behave predictably even when the world around it doesn’t.

Step 1: Put boundaries on everything

Before you think about orchestration patterns or fancy frameworks, the single most useful thing you can do is constrain your agent’s behavior.

I mean this literally. Set hard limits on:

• Maximum number of LLM calls per task (I usually start with 10 and adjust)

• Maximum wall-clock time per agent run

• Maximum tokens spent per run

• Maximum number of tool invocations

Without these, a confused agent will happily burn through your entire monthly API budget in one run. I’ve seen it happen. Not to me, thankfully. Okay, once to me.

Here’s what a simple bounded agent loop looks like in TypeScript:

async function runAgent(task: string, tools: Tool[], options: AgentOptions) {
  const maxSteps = options.maxSteps ?? 10;
  const maxDurationMs = options.maxDurationMs ?? 30_000;
  const startTime = Date.now();
  const messages: Message[] = [{ role: "user", content: task }];
  let steps = 0;

  while (steps < maxSteps) {
    if (Date.now() - startTime > maxDurationMs) {
      return { status: "timeout", steps, messages };
    }

    const response = await callLLM(messages, tools);
    messages.push(response);
    steps++;

    if (response.toolCalls && response.toolCalls.length > 0) {
      for (const call of response.toolCalls) {
        const result = await executeToolWithTimeout(call, 5000);
        messages.push({ role: "tool", content: result, toolCallId: call.id });
      }
    } else {
      return { status: "complete", steps, messages };
    }
  }

  return { status: "max_steps_exceeded", steps, messages };
}

Nothing fancy. But notice the return type always includes status. That's the first principle: every agent run should terminate with an explicit status, not just a response. You need to know whether it finished, timed out, or hit a limit. This is the thing that makes the difference between "it worked" and "I can monitor and alert on it."

Step 2: Make tool execution the reliability boundary

Your agent is only as reliable as its tools. And tools fail. APIs return 500s, databases time out, rate limits kick in.

The pattern I’ve found most useful: wrap every tool in its own error boundary, with its own timeout, and return structured results regardless of success or failure. The LLM is surprisingly good at handling “this tool failed with error X” if you give it that information cleanly. What it’s terrible at is handling a thrown exception that kills the entire agent loop.

async function executeToolWithTimeout(
  call: ToolCall,
  timeoutMs: number
): Promise<string> {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), timeoutMs);

  try {
    const tool = toolRegistry.get(call.name);
    if (!tool) {
      return JSON.stringify({
        error: true,
        message: `Unknown tool: ${call.name}`,
      });
    }

    const result = await tool.execute(call.arguments, {
      signal: controller.signal,
    });
    return JSON.stringify({ error: false, data: result });
  } catch (err) {
    const message =
      err instanceof Error ? err.message : "Tool execution failed";
    return JSON.stringify({ error: true, message });
  } finally {
    clearTimeout(timer);
  }
}

The key insight: never throw from tool execution. Always return a structured result. Let the LLM decide what to do with failures. This is one of those things I’m quite certain about after watching agents in production for a while.

Step 3: Think about durability for long-running agents

Short agents that finish in a few seconds? The pattern above is probably enough. But once agents start running for minutes, or need to survive server restarts, or coordinate with other agents, you need something more. This is where the concept of durable execution comes in. If the process dies after step 3 of 7, you should be able to resume from step 3 instead of starting over.

I think this matters more than most people realize. In serverless environments especially, your function might get killed by the platform after a timeout. Without checkpointing, that’s a complete waste of every token and API call that already happened.

The principle is straightforward even if you don’t use a specific durability framework. After each significant step (LLM call, tool result, decision point), persist the agent’s state somewhere. A database, a queue, a file. Whatever your infrastructure supports. Then build your agent loop to accept a “resume from” parameter.

I’m not going to pretend I’ve nailed this perfectly. My current approach is to store the full message history after each step in Postgres, with a run ID and step number. If the process crashes, a recovery worker picks up incomplete runs and resumes them. It’s not elegant but it works.

Step 4: Parallel agents are powerful and dangerous

The Pragmatic Engineer blog recently covered an interesting trend: developers kicking off multiple AI agents in parallel to work on different parts of a codebase simultaneously. The idea is that instead of one agent doing everything sequentially, you split the work and let multiple agents tackle sub-tasks at the same time.

I’ve been experimenting with this and it’s genuinely useful. But it introduces failure modes that sequential agents don’t have.

The obvious one: what happens when agent 3 out of 5 fails? Do you retry just that one? Do you cancel all of them? Does the output of agent 3 depend on agents 1 and 2?

Here’s the pattern I’ve settled on for parallel agent work:

interface AgentTask {
  id: string;
  prompt: string;
  tools: Tool[];
  dependsOn?: string[];
}

async function runParallelAgents(tasks: AgentTask[]): Promise<Map<string, AgentResult>> {
  const results = new Map<string, AgentResult>();
  const pending = new Map(tasks.map((t) => [t.id, t]));

  while (pending.size > 0) {
    const ready: AgentTask[] = [];

    for (const [id, task] of pending) {
      const depsResolved = (task.dependsOn ?? []).every(
        (dep) => results.has(dep) && results.get(dep)!.status === "complete"
      );
      if (depsResolved) ready.push(task);
    }

    if (ready.length === 0 && pending.size > 0) {
      for (const [id] of pending) {
        results.set(id, { status: "blocked", steps: 0, messages: [] });
        pending.delete(id);
      }
      break;
    }

    const batchResults = await Promise.allSettled(
      ready.map(async (task) => {
        const depContext = (task.dependsOn ?? [])
          .map((dep) => results.get(dep))
          .filter(Boolean);
        const contextualPrompt = buildContextualPrompt(task.prompt, depContext);
        const result = await runAgent(contextualPrompt, task.tools, {
          maxSteps: 10,
          maxDurationMs: 30_000,
        });
        return { id: task.id, result };
      })
    );

    for (const settled of batchResults) {
      if (settled.status === "fulfilled") {
        results.set(settled.value.id, settled.value.result);
        pending.delete(settled.value.id);
      } else {
        const failedTask = ready.find(
          (t) => !results.has(t.id)
        );
        if (failedTask) {
          results.set(failedTask.id, {
            status: "error",
            steps: 0,
            messages: [],
          });
          pending.delete(failedTask.id);
        }
      }
    }
  }

  return results;
}

Notice the dependency graph. Some agents can run in parallel, but others depend on earlier results. The orchestrator resolves dependencies, runs independent tasks concurrently, and handles failures without killing the entire batch.

I’m going to be honest: the error handling here is something I’m still iterating on. The “blocked” status when dependencies can’t be resolved feels like the right thing, but I haven’t tested it under enough real scenarios to be certain.

Step 5: Observe everything, trust nothing

Remember the observability point from step 1? It comes back here, and it’s even more important with agents than with normal services.

For every agent run, I log:

• Total steps taken

• Total tokens consumed (prompt and completion separately)

• Wall-clock duration

• Which tools were called and how many times

• The terminal status (complete, timeout, max_steps, error)

• Whether the agent retried any tool calls

This is how you catch the patterns that kill you. “Hey, the invoice-processing agent has been averaging 8 steps for the past week, but today it’s averaging 14.” That’s your early warning. Something changed in the data, or the LLM is behaving differently, or a downstream API is returning errors that cause retries.

Without these metrics, you’ll find out when your token bill arrives. Or when a user complains. Or at 3 AM.

One thing I keep going back to: the bounded execution from step 1 is what makes observability useful. If an agent can run unbounded, your metrics are meaningless because the variance is infinite. Boundaries give you a normal range to compare against.

Step 6: Test the failure modes, not just the happy path

This is the part most people skip and it’s the part that matters most.

Your tests for an agent system should include:

• What happens when the LLM returns malformed tool calls?

• What happens when a tool times out on every invocation?

• What happens when the agent hits its step limit without completing the task?

• What happens when two parallel agents try to modify the same resource?

• What happens when the LLM decides to call a tool that doesn’t exist?

I write these as integration tests with a mock LLM that returns predefined sequences. It’s not perfect because you can’t predict every weird thing a real LLM will do. But it catches the structural failures: the ones where your orchestration logic breaks, not where the LLM says something dumb.

For the LLM-says-something-dumb cases, I rely on the boundaries from step 1 and the observability from step 5. You can’t test for every hallucination. But you can make sure hallucinations don’t cause unbounded damage.

What I’d do differently starting from scratch

If I were building a new agent system today, I’d start with the boring stuff first. Timeouts, structured tool results, status tracking, logging. Then I’d add the actual agent logic on top.

Most teams do it the other way around. They get the agent working, it’s exciting, it does cool things. Then they spend three months retrofitting all the production-hardening stuff. I’ve done this. It’s painful. The guardrails are much easier to build when you design around them from the start.

I’d also think carefully about whether I actually need agents at all. A lot of problems that people solve with agents can be solved with a well-structured prompt and a single LLM call. Agents add complexity. Every step in an agent loop is a place where things can go wrong. If your task doesn’t require dynamic tool selection or multi-step reasoning, a simpler approach is almost always better.

That said, when you do need agents (and there are real cases where you do), building them with durability in mind from day one will save you more headaches than any framework or library choice.

Where I’m still figuring things out

I don’t have a great answer for agent memory yet. For short tasks, passing the full message history works fine. For agents that run across multiple sessions or need to remember things from days ago, I’m experimenting with summarization and retrieval patterns, but nothing feels solid yet.

I also don’t have strong opinions on agent frameworks. There are a lot of them. Some seem good, some seem like thin wrappers around API calls with a lot of abstraction for abstraction’s sake. I’ve been writing my own orchestration code because it helps me understand the failure modes, but I could be wrong that this is the best use of my time.

And multi-agent coordination where agents communicate with each other, not just run in parallel, is something I’ve read about more than I’ve built. Projects like Wuphf (which uses Git and Markdown files as a shared knowledge base between agents) are interesting because they solve the coordination problem through a shared artifact instead of direct communication. That feels right to me, but I haven’t tested it enough to recommend it.

The honest summary: if you get the basics right (boundaries, structured tool results, observability, explicit status tracking), you can build agent systems that run in production without constant babysitting. The fancy orchestration patterns matter less than you’d think. The boring reliability patterns matter more.

Build the guardrails first. Then let the agents loose inside them.

But when you try to use AI on real production work, things with actual constraints, team conventions, and code that has to survive contact with other engineers, that approach falls apart fast. The output is technically correct but doesn’t fit anywhere. You end up rewriting most of it anyway.

I’ve been reading through some of the most useful practical writing I’ve found on this topic, and I want to walk through three patterns that actually change the output quality in a meaningful way. Not theory. Patterns you can start using today.

Start with context, not prompts

Here’s a pattern I see constantly: engineers write detailed prompts but give the AI no context about where the code will live. No project structure. No team conventions. No architectural decisions. Just a description of the feature they want.

The AI fills in the blanks. And it’s good at that. But it fills them in with generic, statistically average choices, not your choices.

The fix is something called knowledge priming (or context engineering if you want to sound fancy). Before you start a session, you feed the AI the information it needs to make decisions that match your codebase.

This can be as simple as pasting your team’s style guide into the conversation. Or pointing the AI at a representative file from your codebase and saying “write new code that looks like this.” Or, and this is where it gets more structured, maintaining a document that lives in your repo and gets included in every AI session automatically.

The Encoding Team Standards piece from Martin Fowler’s site gets into exactly this. The idea is to make your team’s conventions explicit and machine-readable, so you’re not re-explaining them every session. Things like: how you name variables, how you handle errors, what packages you prefer, what patterns you avoid. Not a vague “we care about clean code.” Specific, concrete rules the AI can actually follow.

This matters more than most people realize. The AI isn’t being sloppy when it ignores your conventions. It genuinely doesn’t know them. Give it the information and the output quality shifts noticeably.

Build a harness before you build a feature

This one changed how I think about AI-assisted development entirely.

The instinct when working with a coding agent is to ask it to build the thing you need. But what usually happens is the agent generates something, you’re not sure if it’s right, you ask for changes, the changes break something else, and you spend more time debugging than you would have writing it yourself.

The pattern that actually works is to build the harness first.

A harness here isn’t a testing framework in the traditional sense. It’s a set of constraints (tests, type contracts, lint rules, example inputs and outputs) that define what correct looks like before any implementation exists. You give the agent something to run against. It can iterate on its own output, catch its own mistakes, and come back to you with something that already passes your criteria.

This is the core of what harness engineering describes. Instead of reviewing AI output by reading it and hoping you catch the bugs, you create an automated feedback mechanism. The agent fails fast, locally, on your constraints, not in production, not in code review.

Here’s what this looks like in practice. Say you’re asking an agent to implement a data transformation function. Before you write the prompt, you write:

- A few unit tests covering the expected behavior

- Type signatures that constrain the inputs and outputs

- Maybe a couple of edge cases you know are tricky

Then you give the agent the tests and tell it to make them pass. Not “build me a function that does X.” Give it something to aim at.

The difference in output quality is real. The agent has a ground truth to orient around. It stops guessing at what “correct” means and starts solving a specific, verifiable problem.

This also makes code review faster. When an agent’s output comes with passing tests, you’re not starting from scratch when evaluating it. You’re asking: are these tests sufficient? That’s a much smaller question.

Create a feedback loop, not a conversation

Most AI-assisted coding sessions look like a conversation. You ask for something, you get something, you give feedback, you get a revision. Back and forth.

That works. But it doesn’t scale, and it doesn’t improve over time. Every session starts from zero. Every mistake is one you have to catch yourself.

The Feedback Flywheel pattern is about turning that conversation into something self-improving. The idea is to capture the feedback you’re giving the AI (the corrections, the style notes, the “no, not like that” moments) and encode them back into the context the AI starts with next time.

So say you’re working with an agent and it keeps generating code with a pattern you don’t use. You correct it. That correction disappears when the session ends. But if you take that correction and add it to your team standards document, it’s part of the context the next session starts with. You stopped correcting the same mistake.

Over time, this compounds. Your AI sessions get progressively less corrective work, because the common mistakes are already ruled out before the session begins. The flywheel is slow at first (encoding one convention at a time is tedious) but it pays back quickly.

The practical steps for this are roughly:

1. Run a session, collect corrections and feedback

2. Group the feedback into categories (naming, patterns, architecture, style)

3. Rewrite the corrections as rules, not commentary: “use X” not “avoid Y because...”

4. Add those rules to a shared context document that every session gets

This is also how teams start sharing AI productivity gains. If one engineer figures out a better prompt structure or catches a common mistake pattern, it shouldn’t stay in their head. It should go into the shared context, where everyone benefits automatically.

A note on what these patterns have in common

Looking at all three, the thread is the same: friction reduction through upfront investment.

Knowledge priming requires writing down your conventions explicitly. Harness engineering requires writing tests before implementation. The feedback flywheel requires capturing corrections and updating shared context. None of these feel productive in the moment. They all slow down the first session.

But they’re the difference between AI assistance that compounds and AI assistance that plateaus.

The engineers I’ve seen get the most out of these tools aren’t the ones with the cleverest prompts. They’re the ones treating the AI’s working environment with the same care they’d treat their own. Good tooling, good constraints, good feedback mechanisms.

The AI isn’t going to ask for any of this. It will work with whatever you give it. The question is whether what you give it is enough for it to do good work.

Where to start

If you’re not doing any of this yet, pick one:

• Easiest: Write a short conventions document for your project. Three to five specific rules the AI should follow. Paste it at the start of every session for a week and notice what changes.

• Higher impact: Before your next feature task, write two or three tests that describe the expected behavior. Use those as the prompt. See if you spend less time revising.

• Long game: After your next AI session, write down the corrections you made. Find the most common one. Turn it into a rule in your conventions document.

None of this requires new tools. It’s a shift in how you set up the work before the AI touches it.

That’s the whole idea. AI coding assistants are powerful, but they’re not magic. They’re tools that reflect the quality of their inputs. Make the inputs better and the outputs follow.

Kind is an excellent tool for setting up a local Kubernetes environment. It offers:

• Run Kubernetes clusters inside Docker containers.

• Quick and simple setup for local development.

• Perfect for both beginners and experienced developers experimenting with Kubernetes features.

Who Is This Article For?

Kubernetes Beginners: Developers getting started with Kubernetes who want a practical, hands-on introduction.

Experienced Developers: Those who prefer a “deploy first” approach—setting up containers and Kubernetes clusters locally before moving to cloud infrastructure.

What You’ll Learn

This article breaks down Kubernetes core concepts step by step. In each section, we’ll dive deeper into the fundamentals, explaining key Kubernetes components and how they interact with each other through practical examples.

Base Project

To keep things practical and focused on Kubernetes concepts, we’ll use a simple Go application. You can find the complete code here: gst-app

Containerization and Kind: Building and Managing Our Kubernetes Environment

Containerization has transformed how applications are built, shipped, and deployed. By isolating applications and their dependencies into lightweight, self-contained packages, containers ensure consistent behavior across different environments—from development to production. Let’s explore containerization fundamentals and how we use Kind (Kubernetes in Docker) to set up our Kubernetes environment.

Containerization: A Modern Approach to Application Deployment

What is Containerization?

Containerization involves packaging an application and its dependencies into a “container”—a lightweight, portable, self-sufficient environment that runs consistently across various infrastructures.

Key Benefits:

• Isolation: Containers provide isolated environments, preventing conflicts between applications running on the same host.

• Portability: Containers run on any system supporting the container runtime, ensuring consistent deployment across development, testing, and production.

• Scalability: Containers can easily scale up or down, making them ideal for dynamic, cloud-native applications.

Example: Dockerfile Configuration

In our project, dockerfile.todo defines the Docker image for the todo-api service:

FROM golang:1.23.0 AS build_todo-api

ENV CGO_ENABLED=0 GOOS=linux GOARCH=amd64
WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .

RUN go build -o todo-api ./main.go 

FROM alpine:3.18
RUN apk --no-cache add postgresql-client
RUN addgroup -g 1000 -S todo && \
    adduser -u 1000 -h /app -G todo -S todo

WORKDIR /app
COPY --from=build_todo-api --chown=todo:todo /app/todo-api /app/todo-api
USER todo
EXPOSE 8000

CMD [”./todo-api”]

LABEL org.opencontainers.image.title=”todo-api” \
      org.opencontainers.image.authors=”Diêgo <diegomagalhaes.contact@gmail.com>” \
      org.opencontainers.image.source=”https://github.com/diegom7s-dev/gst-app” \
      org.opencontainers.image.version=”1.0.0”

This Dockerfile uses a two-stage build:

1. Build Stage (golang:1.23.0 AS build_todo-api): Compiles the application in a clean Go environment, ensuring the final image contains only necessary binaries.

2. Runtime Stage (FROM alpine:3.18): Copies the compiled binary to a minimal Alpine Linux image, providing a lightweight runtime environment with only essential dependencies like postgresql-client.

By separating build and runtime stages, we optimize the image for both size and security—following containerization best practices.

Kind: Simulating a Kubernetes Cluster in Docker

What is Kind?

Kind (Kubernetes in Docker) is a tool for running local Kubernetes clusters using Docker containers as nodes. It’s excellent for local development and testing, allowing developers to create multi-node clusters without needing multiple physical or virtual machines.

Example: Kind Configuration

The kind.config.yaml file defines our Kind cluster configuration:

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
  extraPortMappings:
  # Todo-Api
  - containerPort: 8000
    hostPort: 8000
  # Postgres
  - containerPort: 5432
    hostPort: 5432

This configuration creates a single-node Kind cluster with a control plane role. It maps ports 8000 and 5432 from the host to the container, allowing us to access services running inside the cluster (the todo-api on port 8000 and PostgreSQL on port 5432) directly from our local machine.

Integrating Containerization with Kind: Building and Running the Service

Makefile Setup for Automation

Our project’s Makefile automates various tasks related to building and deploying the todo-api service using Docker and Kind:

# Define dependencies
GOLANG          := golang:1.22.2
ALPINE          := alpine:3.18
KIND            := kindest/node:v1.27.3
POSTGRES        := postgres:15.4

# Building containers
service:
    docker build \
        -f infra/docker/dockerfile.todo \
        -t $(SERVICE_IMAGE) \
        --build-arg BUILD_REF=$(VERSION) \
        --build-arg BUILD_DATE=`date -u +”%Y-%m-%dT%H:%M:%SZ”` \
        .

# Running from within k8s/kind
dev-up:
    kind create cluster \
        --image $(KIND) \
        --name $(KIND_CLUSTER) \
        --config infra/k8s/dev/kind/kind.config.yaml

    kubectl config use-context kind-$(KIND_CLUSTER)
    kubectl wait --timeout=120s --namespace=local-path-storage --for=condition=Available deployment/local-path-provisioner
    kind load docker-image $(POSTGRES) --name $(KIND_CLUSTER)

Key Makefile Targets:

• service: Builds the Docker image for the todo-api service using the Dockerfile at infra/docker/dockerfile.todo.

• dev-up: Creates a Kind cluster using the specified configuration file and loads necessary Docker images into the cluster.

By leveraging Docker and Kind, our setup ensures a streamlined development workflow that mirrors a production environment (within limitations). This allows us to build, deploy, and test our Go application in a local Kubernetes cluster, providing a high-fidelity environment for development and testing.

Essential Kubernetes Components: What They Are and How to Use Them

Understanding Kubernetes core components is fundamental to effectively deploying and managing applications. Let’s explore the key components that form the foundation of Kubernetes, using our Go application as a practical example.

1. Nodes: The Worker Machines in Kubernetes Clusters

What are Nodes?

Nodes are the worker machines in Kubernetes clusters. They’re responsible for running containerized applications and providing the computational resources needed to keep your applications running smoothly. Nodes can be physical servers or virtual machines, depending on your cluster configuration.

Architectural Role:

• Runtime Environment: Nodes serve as the execution environment for your Pods. Each node runs at least a kubelet (an agent responsible for communicating with the Kubernetes control plane), a container runtime (like Docker or containerd), and kube-proxy (which maintains network rules on nodes).

• Resource Management: Nodes provide CPU, memory, storage, and network resources for running containers. Kubernetes manages these resources efficiently, ensuring each Pod receives the necessary resources as specified in its configuration.

Node Components:

• Kubelet: An agent running on each node that ensures containers are running in a Pod. It continuously monitors Pod status and communicates with the Kubernetes API server to maintain the desired state.

• Container Runtime: The software responsible for running containers. Popular runtimes include Docker, containerd, and CRI-O. Kubernetes supports any runtime implementing the Kubernetes Container Runtime Interface (CRI).

• Kube-proxy: A network proxy running on each node that manages network communication between Pods across different nodes. It implements Kubernetes networking services on each node, ensuring Pods can communicate with each other and external services.

Example in Our Project:

In our project, nodes are represented by Docker containers running Kubernetes when using Kind. Each node in a Kind cluster is a Docker container, allowing us to simulate a multi-node Kubernetes cluster locally.

While we don’t have a specific YAML manifest to define nodes (since nodes are managed by the control plane), we rely on them to provide the necessary environment for our Pods and services. For example, when deploying the PostgreSQL database or the todo-api application, Kubernetes schedules these Pods on available nodes, utilizing their computational resources.

Key Concepts Related to Nodes:

• Node Affinity and Anti-Affinity: Kubernetes provides mechanisms to control how Pods are scheduled on nodes. Node affinity allows you to define rules that attract Pods to certain nodes, while anti-affinity ensures Pods are distributed across nodes for improved fault tolerance.

• Taints and Tolerations: Used to prevent certain Pods from being scheduled on specific nodes. For example, a node can be tainted to allow only specific workloads, like those requiring GPUs, ensuring only compatible Pods are scheduled there.

Understanding Node Management:

• Node Status: Each node maintains a status providing essential information like node health, capacity (CPU, memory, etc.), and conditions (e.g., Ready, DiskPressure, MemoryPressure).

• Node Maintenance: Nodes can be marked as unschedulable when needing maintenance, preventing new Pods from being scheduled while allowing existing Pods to continue running or be rescheduled.

2. Pods: The Fundamental Building Block of Kubernetes

What are Pods?

Pods are the smallest deployable units in Kubernetes, representing a single instance of a running process. A Pod can encapsulate one or more containers that share the same network namespace and storage. Containers within a Pod can communicate using localhost and share storage volumes.

Architectural Role:

• Ephemeral Nature: Pods are designed to be ephemeral. When a Pod fails, Kubernetes automatically creates a new Pod to replace it rather than repairing the existing one.

• Container Co-location: Containers that need to share resources (like storage or networking) or must always be deployed together are grouped in a single Pod.

Example in Our Project:

In our project, the StatefulSet configuration in dev-database.yaml defines a Pod template for running a PostgreSQL container:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
  namespace: simple-go-todo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: database
  template:
    metadata:
      labels:
        app: database
    spec:
      containers:
      - name: postgres
        image: ‘postgres:15.4’
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data

This configuration ensures a Pod running a PostgreSQL database is created and maintained, with persistent storage mounted at /var/lib/postgresql/data.

3. Deployments: Managing Your Application’s Desired State

What are Deployments?

Deployments are abstractions that manage Pods and ReplicaSets. They provide declarative updates, ensuring the specified number of Pods is always running, and handle tasks like scaling, rolling updates, and rollbacks.

Architectural Role:

• Scalability and Resilience: Deployments enable horizontal scaling of applications (increasing the number of replicas) to handle increased traffic or workload.

• Rolling Updates and Rollbacks: Support zero-downtime updates by incrementally updating Pods with new application versions and can roll back to a previous version if needed.

Example in Our Project:

The base-service.yaml file specifies a Deployment for our todo application:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: todo
  namespace: simple-go-todo
spec:
  replicas: 1
  selector:
    matchLabels:
      app: todo
  template:
    metadata:
      labels:
        app: todo
    spec:
      containers:
      - name: todo-api
        image: service-image

This Deployment manages the lifecycle of the todo-api Pod, ensuring one instance is always running and can be scaled as needed.

4. Services: Stable Networking for Your Pods

What are Services?

Services provide stable network endpoints for accessing Pods within a Kubernetes cluster. They abstract network access to Pods, enabling communication within the cluster and with external clients.

Service Types:

• ClusterIP: Exposes the Service on an internal cluster IP, accessible only within the cluster

• NodePort: Exposes the Service on a static port on each node’s IP

• LoadBalancer: Provisions an external IP to load balance traffic across nodes

• ExternalName: Maps a Service to an external DNS name

Architectural Role:

• Decoupling: Services decouple clients from the underlying Pod IP addresses, which can change if Pods are recreated or rescheduled

• Service Discovery: They provide a consistent interface for service discovery, allowing other applications to reliably discover and communicate with Pods

Example in Our Project:

The dev-todo-patch-service.yaml creates a Service for the todo-api:

apiVersion: v1
kind: Service
metadata:
  name: todo-api
  namespace: simple-go-todo
spec:
  type: ClusterIP
  ports:
  - name: todo-api
    port: 8000
    targetPort: todo-api

This Service enables internal cluster communication to access the todo-api on a stable IP and port.

5. ConfigMaps and Secrets: Managing Configuration and Sensitive Data

What are ConfigMaps and Secrets?

• ConfigMaps: Store non-sensitive configuration data in key-value pairs

• Secrets: Store sensitive data like passwords, OAuth tokens, and SSH keys, base64-encoded

Architectural Role:

• Separation of Configuration and Code: Enable separating configuration from application code, making applications portable and easier to manage

• Secure and Flexible Management: Secrets ensure sensitive data is managed securely, while ConfigMaps provide a flexible way to manage configurations without hardcoding values

Example in Our Project:

We use a ConfigMap to configure PostgreSQL settings in dev-database.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: pghbaconf
  namespace: simple-go-todo
data:
  pg_hba.conf: |
    local   all             all                                     trust
    # IPv4 local connections:
    host    all             all             0.0.0.0/0               trust
    # IPv6 local connections:
    host    all             all             ::1/128                 trust
    # Allow replication connections from localhost, by a user with the
    # replication privilege.
    local   replication     all                                     trust
    host    replication     all             0.0.0.0/0               trust
    host    replication     all             ::1/128                 trust

This ConfigMap stores PostgreSQL’s access control configuration, mounted as a file in the Pod.

6. StatefulSets: Managing Stateful Applications

What are StatefulSets?

StatefulSets manage the deployment and scaling of a set of Pods, providing guarantees about the ordering and uniqueness of these Pods.

Architectural Role:

• Stateful Application Management: Ideal for managing stateful applications where each Pod must have a unique identity and stable persistent storage

• Stable Network Identity and Storage: Ensures each Pod has a unique, stable network identity and can maintain persistent storage across restarts

Example in Our Project:

The dev-database.yaml file uses a StatefulSet to deploy a PostgreSQL instance:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: database
  namespace: simple-go-todo
spec:
  selector:
    matchLabels:
      app: database
  replicas: 1
  template:
    metadata:
      labels:
        app: database

This configuration provides stable identity and persistent storage for our PostgreSQL database.

Kubernetes’ Declarative Model

Kubernetes uses a declarative model where you define your application’s desired state, and Kubernetes continuously works to maintain that state. By defining your application components as YAML manifests, you can easily manage and scale your applications in a Kubernetes cluster. This approach contrasts with imperative models where each step is executed manually, offering a more scalable and resilient way to manage applications.

Understanding these Kubernetes objects and their architecture is crucial to leveraging Kubernetes’ full potential. In our project, we apply these concepts to deploy a Go application, providing a practical example of how each component fits into the overall architecture.

Putting It Into Practice: Running the Project with Makefile Commands

In this final section, we’ll walk through the step-by-step process of building, deploying, and running our Go application using the commands defined in the Makefile. This will provide a comprehensive understanding of how each command contributes to the overall deployment process and ensure everything works correctly in your Kubernetes environment.

The Makefile simplifies the workflow by automating repetitive tasks. Let’s break down the main commands and what happens when you execute each one.

1. Installing Dependencies

The first step in setting up our environment is installing all necessary dependencies. The Makefile provides a target to install these dependencies using Homebrew (feel free to adapt this for your preferred package manager):

dev-brew:
    brew update
    brew list kind || brew install kind
    brew list kubectl || brew install kubectl
    brew list kustomize || brew install kustomize
    brew list pgcli || brew install pgcli

This step ensures all necessary tools are available on your machine to interact with the Kubernetes cluster and manage configurations.

2. Pulling Docker Images

Before building our custom Docker image, we need to ensure we have the necessary base images:

dev-docker:
    docker pull $(GOLANG)
    docker pull $(ALPINE)
    docker pull $(KIND)
    docker pull $(POSTGRES)

This script pulls the specified Docker images for Go, Alpine, Kind node, and PostgreSQL. These images are the foundation for building our custom application image and running our local Kubernetes cluster.

3. Building the Docker Image

The Makefile includes a command to build the Docker image for our todo-api service:

service:
    docker build \
        -f infra/docker/dockerfile.todo \
        -t $(SERVICE_IMAGE) \
        --build-arg BUILD_REF=$(VERSION) \
        --build-arg BUILD_DATE=`date -u +”%Y-%m-%dT%H:%M:%SZ”` \
        .

Docker Image Build: This command builds the Docker image using the Dockerfile at infra/docker/dockerfile.todo. It tags the image with the version specified in the VERSION variable (like todo-api:0.0.1).

Build Arguments: BUILD_REF and BUILD_DATE are passed as build arguments to incorporate versioning information and build metadata into the image.

The resulting Docker image contains the compiled Go application, ready to be deployed to our Kubernetes cluster.

4. Creating the Kind Cluster

To simulate a Kubernetes environment locally, we use Kind to create a new cluster:

dev-up:
    kind create cluster \
        --image $(KIND) \
        --name $(KIND_CLUSTER) \
        --config infra/k8s/dev/kind/kind.config.yaml

    kubectl config use-context kind-$(KIND_CLUSTER)
    kubectl wait --timeout=120s --namespace=local-path-storage --for=condition=Available deployment/local-path-provisioner
    kind load docker-image $(POSTGRES) --name $(KIND_CLUSTER)

• Creating the Kind Cluster: The kind create cluster command creates a new Kubernetes cluster named sgt-kind-cluster using the specified Kind node image (kindest/node:v1.27.3) and configuration file (kind.config.yaml).

• Setting Kubernetes Context: kubectl config use-context switches the current Kubernetes context to the new Kind cluster, allowing subsequent kubectl commands to interact with it.

• Waiting for Storage Provisioner: The kubectl wait command waits until the local-path-provisioner deployment is available, ensuring the cluster is ready to provision storage volumes.

• Loading Docker Image into Cluster: kind load docker-image loads the PostgreSQL Docker image into the Kind cluster, making it available for our application.

5. Deploying the Application to Kubernetes

With the cluster configured and images loaded, we can now deploy our application and its dependencies.

Using Kustomize to Manage Kubernetes Configurations

What is Kustomize?

Kustomize is a Kubernetes-native tool that allows you to customize Kubernetes resource configurations without modifying the original YAML files. It’s especially useful for managing different environments (like development, testing, and production) from a common base of configuration files. Using Kustomize, we can automatically generate customized manifests for our cluster by applying specific overlays that adjust configurations as needed.

dev-apply:
    kustomize build infra/k8s/dev/database | kubectl apply -f -
    kubectl rollout status --namespace=$(NAMESPACE) --watch --timeout=120s sts/database
    
    kustomize build infra/k8s/dev/service | kubectl apply -f -
    kubectl wait pods --namespace=$(NAMESPACE) --selector app=$(APP) --timeout=120s --for=condition=Ready

• Apply Database Configuration: kustomize build generates Kubernetes manifests for the database configuration from base YAML files. kubectl apply -f - applies these configurations to the cluster, creating necessary resources (e.g., StatefulSet for PostgreSQL)

• Wait for Database Deployment: The kubectl rollout status command waits for the PostgreSQL StatefulSet to be fully deployed and running before proceeding

• Apply Service Configuration: The process repeats for the todo-api service, ensuring the service and its dependencies are deployed to the cluster

• Wait for Pods to Be Ready: kubectl wait pods ensures all Pods associated with the todo-api application are running and ready before completing the deployment process

6. Testing Application Endpoints

Finally, we can use the Makefile to test our REST API endpoints and verify everything is working as expected:

test_all: create get_all get_one update delete

By following these steps, you can successfully build, deploy, and test your Go application in a local Kubernetes cluster using Docker and Kind. The Makefile automates much of this process, making it easier to manage and reducing the risk of errors.

Conclusion

By combining containerization, Kubernetes, and Kind, we can create a powerful and flexible local development environment that closely resembles a production setup (within limitations). This approach enables efficient development, testing, and iteration, ensuring your applications are robust, scalable, and ready for deployment in real-world environments.