All writing

Logging Isn't About Printing — It's About Debugging at Scale

One of the most painful moments while building scalable systems is this: something breaks in production, and you don't know why.

So you SSH into the server (or open your logging dashboard) and start digging through logs. And that's when reality hits. The logs are cluttered. Everything is INFO.

  • Every request prints five lines.
  • Every database call prints two more.
  • Every middleware prints something.

Now you're scrolling through thousands of lines just to find one real error. This is where most systems fail — not because logging is missing, but because logging is misused.

The Problem: Misusing Log Levels

Many developers use INFO for everything. But log levels exist for a reason:

  • Debug → Detailed internal state, verbose information
  • Info → Important business events
  • Warn → Suspicious behavior
  • Error → Something failed

When debug-level logs are printed at INFO level in production, you end up flooding your logs, increasing storage costs, and slowing down aggregation systems. In production, noise is the enemy of debugging.

Logging Should Be Intentional

Production logs are not for developers browsing terminal output. They are intended for an entire observability pipeline:

  • Machine-parsed & aggregated
  • Indexed & queried
  • Used for alerts and dashboards

Why Structured Logging Matters

Many repositories use Zap for logging. That's good. But often, they don't actually use its core strengths, like structured JSON logs and performance-optimized encoding.

Instead of string concatenation:

go
logger.Info("User login failed for " + email)

Use structured fields:

go
logger.Error("User login failed",
    zap.String("email", email),
    zap.String("reason", "invalid password"),
)

Now your logs are searchable and aggregatable. Machines can filter by email, reason, or level. That's real logging.

The Right Way to Think About Logging

In development: Use Debug level, readable console output, and log generously.

In production: Default to Info or higher, keep Debug disabled, and output structured JSON to stdout.

If you really need debug logs in production, use a runtime-configurable level (like Zap's AtomicLevel) — don't permanently pollute your logs.

Final Thought

Logging is not about printing messages. It's about reducing the time between "Something is broken" and "We know exactly why."

The next time you add logging to your system, don't just add logs. Design them. Because in scalable systems, good logging isn't a feature — it's infrastructure.