78:T451,{ "timestamp": "2024-01-15T14:32:07.123456Z", "level": "ERROR", "logger": "payment.processor", "message": "Payment processing failed", "service": { "name": "payment-service", "version": "2.14.3", "environment": "production", "instance_id": "payment-prod-us-east-1a-7f8c9d" }, "trace": { "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "span_id": "00f067aa0ba902b7", "parent_span_id": "83d9c3e5a8d2f71c" }, "event": { "type": "payment_failure", "user_id": "usr_a1b2c3d4e5", "amount_cents": 9999, "currency": "USD", "failure_reason": "insufficient_funds", "payment_method": "credit_card", "card_last_four": "4242" }, "context": { "request_id": "req_x1y2z3", "session_id": "sess_m9n8o7", "user_agent": "Mozilla/5.0 (iPhone; CPU iPhone OS 17_0)", "client_ip": "192.168.1.100", "geo_country": "US" }, "error": { "type": "InsufficientFundsException", "message": "Account balance 45.00 insufficient for charge 99.99", "stack_trace": "at PaymentProcessor.charge(PaymentProcessor.java:142) ..." } }79:T447,**Anatomy of a Production Log Entry** The example above demonstrates enterprise-grade structured logging. Let's analyze each section: **Core Fields** (`timestamp`, `level`, `logger`, `message`): Universal across all log entries. The timestamp uses ISO 8601 with microsecond precision—critical for ordering events in distributed systems. **Service Metadata**: Identifies *where* the log originated. In a microservices architecture with hundreds of services, this is essential for filtering and routing. **Distributed Tracing Context**: The `trace_id` links this log to all other logs from the same request across services. This enables reconstructing full request flows. **Event Specifics**: The `event` object contains domain-specific data. Note that `amount_cents` is an integer (not a float)—this prevents floating-point precision issues in financial calculations. **Request Context**: Client information for security auditing, debugging, and analytics. **Error Details**: Structured error information enables automated error categorization and alerting on specific exception types.7a:T663,import structlog import logging # Configure structlog for JSON output structlog.configure( processors=[ structlog.stdlib.filter_by_level, structlog.stdlib.add_logger_name, structlog.stdlib.add_log_level, structlog.stdlib.PositionalArgumentsFormatter(), structlog.processors.TimeStamper(fmt="iso", utc=True), structlog.processors.StackInfoRenderer(), structlog.processors.format_exc_info, structlog.processors.UnicodeDecoder(), structlog.processors.JSONRenderer() # Output as JSON ], wrapper_class=structlog.stdlib.BoundLogger, context_class=dict, logger_factory=structlog.stdlib.LoggerFactory(), cache_logger_on_first_use=True, ) # Usage in application code logger = structlog.get_logger() # Bind context that persists across log calls logger = logger.bind( service="payment-service", version="2.14.3", environment="production" ) # In request handler - bind request-specific context logger = logger.bind( trace_id=request.headers.get("X-Trace-Id"), user_id=authenticated_user.id, request_id=generate_request_id() ) # Log with event-specific fields logger.info( "payment_processed", amount_cents=9999, currency="USD", payment_method="credit_card", latency_ms=142 ) # Output: # {"timestamp":"2024-01-15T14:32:07.123456Z","level":"info","logger":"payment.handler", # "service":"payment-service","version":"2.14.3","environment":"production", # "trace_id":"4bf92f...","user_id":"usr_a1b2c3","request_id":"req_x1y2z3", # "event":"payment_processed","amount_cents":9999,"currency":"USD",...}7b:T681, trace_id span_id user_id {"service":"payment-service","version":"2.14.3"}

// Application code import org.slf4j.Logger; import org.slf4j.LoggerFactory; import org.slf4j.MDC; import net.logstash.logback.marker.Markers; public class PaymentService { private static final Logger logger = LoggerFactory.getLogger(PaymentService.class); public void processPayment(PaymentRequest request) { // Set request context in MDC (automatically included in all logs) MDC.put("trace_id", request.getTraceId()); MDC.put("user_id", request.getUserId()); try { // Log with structured fields using Markers logger.info( Markers.append("event", "payment_initiated") .and(Markers.append("amount_cents", request.getAmountCents())) .and(Markers.append("currency", request.getCurrency())), "Processing payment" ); // ... payment logic ... } finally { MDC.clear(); // Clean up thread-local context } } }70:{"json":{"id":"seed_de9b2d1d4fcfac8ff250def941664a28","slug":"logging-at-scale","category":"System Design (HLD)","topic":"Logging at Scale","title":"Logging at Scale: Production-Grade Observability","description":"Master the art and science of logging for distributed systems at scale. Learn structured logging, log levels, aggregation architectures, retention strategies, and correlation patterns that power observability at companies like Google, Netflix, and Amazon.","difficulty":"INTERMEDIATE","estimatedMinutes":60,"pages":[{"id":"seed_c90c24a78b00b476f3f5e6547004f307","title":"Structured Logging","order":0,"content":[{"id":"opening-hook","title":"Beyond printf: The Revolution in Production Logging","content":[{"data":{"content":"In the early days of computing, logging was simple: `printf(\"Something happened at %s\", timestamp)`. This worked when systems were monolithic, traffic was predictable, and a single engineer could mentally trace execution through log files. Those days are long gone.\n\nModern distributed systems generate **petabytes of logs daily**. Netflix produces over 400 billion logging events per day. Google's distributed systems emit logs from millions of machines simultaneously. At this scale, unstructured logs—human-readable text strings—become essentially useless.\n\n**Structured logging** represents a fundamental paradigm shift: instead of logging text for humans to read, we log **machine-parseable data structures** that humans can query. This distinction enables everything modern observability depends on: full-text search, metric aggregation, anomaly detection, and cross-service correlation."},"type":"text"},{"data":{"icon":"Target","title":"What You Will Learn","content":"By the end of this page, you will understand why structured logging is non-negotiable for production systems. You'll master JSON logging formats, schema design patterns, field typing conventions, and parsing strategies that enable your logs to become a queryable source of truth rather than an unreadable stream of text.","variant":"info"},"type":"callout"}]},{"id":"problem-unstructured-logs","title":"The Problem with Unstructured Logs","content":[{"data":{"content":"To appreciate structured logging, we must first understand why unstructured logging fails. Consider a typical unstructured log line:\n\n```\n2024-01-15 14:32:07 ERROR Payment processing failed for user john@example.com, amount $99.99, reason: insufficient funds\n```\n\nThis looks perfectly readable. An engineer can glance at it and understand what happened. But consider the challenges at scale:"},"type":"text"},{"data":{"items":["**Parsing Fragility** — To extract the user email, you need regex `user (\\S+@\\S+)`. But what if another developer logs `for customer john@example.com`? Your parser breaks. Every format variation requires parser updates.","**Field Ambiguity** — Is `$99.99` the transaction amount or the account balance? Without explicit field names, meaning depends on position, which changes across log versions.","**Aggregation Impossibility** — How many payment failures occurred in the last hour? You can't `SUM(amount)` or `GROUP BY reason` when those values are embedded in prose.","**Cross-Service Correlation** — When a request spans 15 microservices, correlating logs requires each service to embed request IDs in the same format. Prose makes this nearly impossible.","**Internationalization Hazards** — Log messages translated for different regions have entirely different formats, making global aggregation fail.","**Indexing Overhead** — Full-text indexing of prose requires expensive tokenization. Structured fields enable efficient inverted indexes on specific attributes."],"title":{"icon":"AlertCircle","text":"Unstructured Logging Pain Points","type":"icon","color":"destructive","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"rows":[["Example","Payment failed for user john@example.com, amount $99.99","{\"event\":\"payment_failed\",\"user\":\"john@example.com\",\"amount\":99.99}"],["Field Extraction","Regex parsing (fragile)","Direct key access (reliable)"],["Schema Evolution","Breaks parsers silently","Explicit field versioning"],["Aggregation","Manual text processing","Native database aggregations"],["Storage Efficiency","High (embedded labels)","Lower (compressed key-value)"],["Query Speed","O(n) text scanning","O(1) indexed field lookup"]],"caption":"Unstructured vs Structured: A Direct Comparison","headers":["Dimension","Unstructured Log","Structured Log (JSON)"]},"type":"table"},{"data":{"icon":"AlertCircle","title":"The Hidden Cost of \"Readable\" Logs","content":"Engineers often argue that unstructured logs are more \"readable.\" This is only true when reading individual lines. At scale, no human reads logs line-by-line. We query, filter, and aggregate. Optimizing for line-by-line readability actively harms queryability—the property that actually matters.","variant":"warning"},"type":"callout"}],"titleType":"number","titleNumber":1},{"id":"json-logging-standard","title":"JSON Logging: The Industry Standard","content":[{"data":{"content":"JSON (JavaScript Object Notation) has become the de facto standard for structured logging. Its ubiquity stems from several properties that make it ideal for logging infrastructure:\n\n- **Universal Parser Support**: Every language has battle-tested JSON parsers\n- **Human Semi-Readability**: Unlike binary formats, JSON remains inspectable\n- **Hierarchical Structure**: Supports nested objects for complex data\n- **Type Flexibility**: Handles strings, numbers, booleans, arrays, and objects\n- **Tooling Ecosystem**: Elasticsearch, Splunk, Loki all have native JSON support"},"type":"text"},{"data":{"code":"$78","title":"Production-Grade Structured Log Entry","language":"json"},"type":"code"},{"data":{"content":"$79"},"type":"text"},{"data":{"icon":"Lightbulb","title":"The Single-Line Imperative","content":"In production, each JSON log entry MUST be a single line—no pretty-printing. Log aggregators parse line-by-line; multi-line logs break parsing. Stack traces should be escaped newlines within a string field, not literal newlines in the JSON.","variant":"tip"},"type":"callout"}],"titleType":"number","titleNumber":2},{"id":"schema-design-principles","title":"Schema Design Principles","content":[{"data":{"content":"Structured logging without consistent schema design descends into chaos. If each team invents their own field names and structures, aggregation becomes impossible. Schema design establishes the **contract** between log producers and log consumers.\n\n**The Two-Level Schema Pattern**\n\nProduction logging schemas typically follow a two-level pattern:\n\n1. **Common Envelope**: Fields present in *every* log entry (timestamp, level, service, trace context)\n2. **Domain Payload**: Event-specific fields that vary by log type\n\nThis pattern enables consistent querying across all logs while allowing flexibility for domain-specific data."},"type":"text"},{"data":{"code":"// Common Envelope (same for all logs)\n{\n \"timestamp\": \"2024-01-15T14:32:07.123456Z\",\n \"level\": \"INFO\",\n \"service\": \"order-service\",\n \"trace_id\": \"4bf92f3577b34da6a3ce929d0e0e4736\",\n \"span_id\": \"00f067aa0ba902b7\",\n \n // Domain Payload (varies by event type)\n \"event_type\": \"order_placed\",\n \"payload\": {\n \"order_id\": \"ord_x1y2z3\",\n \"customer_id\": \"cust_a1b2c3\",\n \"total_cents\": 15999,\n \"items_count\": 3\n }\n}","title":"Schema Design: Envelope + Payload Pattern","language":"json"},"type":"code"},{"data":{"items":["**Use snake_case consistently** — Field names like `user_id`, `order_total`. Camel case works too, but pick one and enforce it organization-wide.","**Namespace domain fields** — Instead of flat `user_id`, `order_id`, use `user.id`, `order.id` or `event.user_id`. Prevents collisions.","**Version your schema** — Include `schema_version: \"1.2\"` when making breaking changes. Consumers can handle multiple versions.","**Prefer flat over deeply nested** — Each nesting level adds query complexity. Limit depth to 3 levels maximum.","**Use consistent naming for common concepts** — If you call it `user_id` in one service, don't call it `userId`, `uid`, or `customer_id` elsewhere.","**Document every field** — Maintain a schema registry with field definitions, types, and examples."],"title":{"icon":"CheckCircle","text":"Schema Design Best Practices","type":"icon","color":"primary","variant":"outline-circle"},"variant":"glass"},"type":"list"},{"data":{"rows":[["timestamp","ISO 8601 string","Yes","When the event occurred (UTC, microsecond precision)"],["level","enum string","Yes","Severity: TRACE, DEBUG, INFO, WARN, ERROR, FATAL"],["logger","string","Yes","Fully qualified logger name (e.g., com.company.service.Class)"],["message","string","Yes","Human-readable summary (for rare manual inspection)"],["service","string","Yes","Originating service/application name"],["version","string","Recommended","Service version (semver)"],["environment","enum string","Yes","dev, staging, production"],["instance_id","string","Recommended","Unique identifier for the process/container"],["trace_id","string","Recommended","Distributed trace identifier (W3C format)"],["span_id","string","Recommended","Current span within the trace"]],"caption":"Common Envelope Fields: The Universal Schema","headers":["Field","Type","Required","Description"]},"type":"table"},{"data":{"icon":"Info","title":"The Cost of Schema Inconsistency","content":"Teams that allow ad-hoc field naming discover, during their first production incident, that they cannot join logs across services. The payment-service logged `customerId`, auth-service logged `userId`, and order-service logged `cust_id`—all referring to the same entity. Schema governance prevents this.","variant":"note"},"type":"callout"}],"titleType":"number","titleNumber":3},{"id":"field-typing-conventions","title":"Field Typing and Conventions","content":[{"data":{"content":"Structured logging's power depends on **consistent field typing**. Log aggregation systems like Elasticsearch create indexes based on field types. If the same field contains a string in one log and a number in another, indexing fails catastrophically.\n\n**The Type Mapping Problem**\n\nConsider this scenario: Your payment service logs `{\"amount\": 99.99}` (float). Your refund service logs `{\"amount\": \"99.99\"}` (string). When Elasticsearch indexes the first log, it creates a `float` mapping. The second log's string value triggers a mapping conflict error—the log is rejected or the field is ignored.\n\nThis problem is insidious because it manifests at *aggregation time*, not at logging time. Your services run fine for months, then a dashboard breaks because 5% of logs have type mismatches."},"type":"text"},{"data":{"rows":[["Identifiers","string","{\"user_id\": \"usr_a1b2c3\"}","IDs may contain letters/hyphens, should never be treated as numbers"],["Monetary Values","integer (cents)","{\"amount_cents\": 9999}","Avoids floating-point precision errors; divide by 100 for display"],["Timestamps","ISO 8601 string","{\"created_at\": \"2024-01-15T14:32:07Z\"}","Universal parsing, timezone-aware, sortable as string"],["Durations","integer (milliseconds)","{\"latency_ms\": 142}","Consistent unit; convert to seconds for display if needed"],["Booleans","boolean","{\"is_premium\": true}","Never use strings like \"true\" or \"false\""],["Enumerations","string","{\"status\": \"completed\"}","Lowercase, underscored values from fixed set"],["Counts","integer","{\"retry_count\": 3}","Always non-negative integers"],["Bytes/Sizes","integer","{\"payload_bytes\": 1048576}","Use bytes as base unit; convert for display"],["Percentages","float (0-1) or integer (0-100)","{\"cpu_usage\": 0.75}","Document which convention you use; be consistent"],["IP Addresses","string","{\"client_ip\": \"192.168.1.1\"}","Enables CIDR queries in some systems"]],"caption":"Field Typing Conventions for Structured Logging","headers":["Data Category","Type","Example Field","Rationale"]},"type":"table"},{"data":{"items":["**Numeric IDs as numbers** — `{\"order_id\": 12345}` breaks when IDs exceed JavaScript's safe integer range or when your ID format changes to include letters.","**Booleans as strings** — `{\"success\": \"true\"}` requires string comparison instead of boolean logic in queries.","**Nullable vs absent fields** — Decide whether missing fields mean `null` or undefined. Be consistent—some systems treat these differently.","**Arrays with mixed types** — `{\"tags\": [\"premium\", 42, true]}` causes indexing issues. Array elements should share a type.","**Floating-point money** — `{\"amount\": 99.99}` can produce `99.98999999` due to floating-point representation. Use integer cents.","**Unix timestamps as integers** — `{\"time\": 1705329127}` loses timezone context and requires manual conversion. ISO 8601 is universally parseable."],"title":{"icon":"AlertTriangle","text":"Common Typing Mistakes to Avoid","type":"icon","color":"accent","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"code":"interface LogEvent {\n // Common envelope - always present\n timestamp: string; // ISO 8601\n level: 'TRACE' | 'DEBUG' | 'INFO' | 'WARN' | 'ERROR' | 'FATAL';\n service: string;\n version: string;\n trace_id?: string;\n span_id?: string;\n}\n\ninterface PaymentFailedEvent extends LogEvent {\n event_type: 'payment_failed';\n payload: {\n user_id: string; // String, not number\n amount_cents: number; // Integer cents\n currency: string; // ISO 4217 code\n failure_reason: 'insufficient_funds' | 'card_declined' | 'fraud_suspected';\n payment_method: 'credit_card' | 'bank_transfer' | 'crypto';\n };\n}\n\n// Type system prevents logging wrong types\nfunction logPaymentFailure(event: PaymentFailedEvent): void {\n console.log(JSON.stringify(event));\n}","title":"Type-Safe Logger Interface (TypeScript Example)","language":"typescript"},"type":"code"},{"data":{"icon":"Lightbulb","title":"Enforce Types at the Logger Level","content":"Don't rely on developers remembering type conventions. Use typed logging interfaces (TypeScript, Java generics, Protocol Buffers) that make it impossible to log wrong types. Catch type errors at compile time, not during a production incident.","variant":"tip"},"type":"callout"}],"titleType":"number","titleNumber":4},{"id":"parsing-processing-pipelines","title":"Parsing and Processing Pipelines","content":[{"data":{"content":"Structured logs are only valuable if your infrastructure can parse and process them efficiently. This section covers the journey from application log emission to queryable storage.\n\n**The Log Processing Pipeline**\n\nModern log processing follows a consistent architecture regardless of specific tooling:\n\n1. **Emission**: Application writes structured JSON to stdout/file\n2. **Collection**: Agent (Fluentd, Filebeat, Vector) reads logs\n3. **Parsing**: Agent validates JSON, extracts fields\n4. **Enrichment**: Add metadata (hostname, container ID, Kubernetes labels)\n5. **Routing**: Direct logs to appropriate destinations based on level/type\n6. **Storage**: Write to search engine (Elasticsearch) or object store (S3)\n7. **Indexing**: Create inverted indexes for fast querying"},"type":"text"},{"data":{"code":"[SERVICE]\n Flush 1\n Log_Level info\n Parsers_File parsers.conf\n\n[INPUT]\n Name tail\n Path /var/log/containers/*.log\n Parser docker\n Tag kube.*\n Mem_Buf_Limit 50MB\n Skip_Long_Lines On\n\n[FILTER]\n Name parser\n Match kube.*\n Key_Name log\n Parser json\n Reserve_Data On\n\n[FILTER]\n Name kubernetes\n Match kube.*\n Kube_URL https://kubernetes.default.svc:443\n Merge_Log On\n K8S-Logging.Parser On\n\n[FILTER]\n Name modify\n Match *\n # Add processing metadata\n Add pipeline_version 1.2.3\n Add processed_at ${TIMESTAMP}\n\n[OUTPUT]\n Name es\n Match *\n Host elasticsearch.logging.svc\n Port 9200\n Index logs-%Y.%m.%d\n Type _doc","title":"Fluent Bit Configuration: Parsing + Enrichment","language":"yaml"},"type":"code"},{"data":{"items":["**Parse at the edge** — Validate JSON and extract fields as early as possible. Invalid logs should be flagged immediately, not silently dropped at the storage layer.","**Fail gracefully** — Malformed logs shouldn't crash the pipeline. Route unparseable logs to a dead-letter queue for investigation.","**Enrich immutably** — Add metadata (timestamps, hostnames) without modifying original fields. Debugging is easier when you can see exactly what the application logged.","**Buffer for reliability** — Use disk-backed buffers between pipeline stages. Network failures shouldn't cause log loss.","**Sample strategically** — For extremely high-volume logs (access logs at 1M+ RPS), consider sampling DEBUG/TRACE levels while keeping all ERROR logs.","**Normalize timestamps** — Convert all timestamps to UTC during processing. Timezone differences silently break queries spanning regions."],"title":{"icon":"Layers","text":"Pipeline Design Principles","type":"icon","color":"primary","variant":"outline-circle"},"variant":"glass"},"type":"list"},{"data":{"left":[{"data":{"items":["Parsing cost zero at collection time","Schema enforced by application code","Type consistency guaranteed","Pipeline configuration simpler","No regex maintenance burden"],"title":{"icon":"CheckCircle","text":"Pre-Parsed (JSON from apps)","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"}],"right":[{"data":{"items":["CPU-intensive regex matching","Parser breakage on format changes","Silent field extraction failures","Complex pipeline configuration","Ongoing regex maintenance overhead"],"title":{"icon":"XCircle","text":"Post-Parsed (Regex on unstructured)","type":"icon","color":"destructive","variant":"outline-circle"},"variant":"accent"},"type":"list"}]},"type":"two-column"},{"data":{"icon":"AlertCircle","title":"The Cost of Post-Parsing","content":"Organizations that emit unstructured logs and parse them centrally suffer from chronic pipeline instability. A single application changing log format can break parsing for all services. Invariably, someone asks: 'Why didn't we just log JSON from the start?' The answer is: you should have.","variant":"error"},"type":"callout"}],"titleType":"number","titleNumber":5},{"id":"performance-considerations","title":"Performance Considerations","content":[{"data":{"content":"Structured logging introduces overhead compared to simple string logging. Understanding and mitigating this overhead is crucial for performance-sensitive applications.\n\n**Sources of Structured Logging Overhead**\n\n1. **Object Allocation**: Creating log event objects consumes heap memory\n2. **Serialization**: Converting objects to JSON is CPU-intensive\n3. **Field Computation**: Calculating derived fields (stack traces, context) has cost\n4. **I/O**: Writing larger payloads takes more time\n5. **Garbage Collection**: Higher object churn increases GC pressure"},"type":"text"},{"data":{"rows":[["String concatenation","2,500,000","0.8μs","48 bytes/log"],["StringBuilder pattern","3,100,000","0.6μs","128 bytes/log"],["Naive JSON (allocating)","450,000","4.2μs","1,024 bytes/log"],["Optimized JSON (pooled)","1,800,000","1.1μs","96 bytes/log"],["Async JSON (buffered)","2,400,000","0.9μs","64 bytes/log (amortized)"]],"caption":"Structured Logging Performance: Benchmark Results","headers":["Logging Approach","Ops/Second","Latency p99","Memory Allocation"]},"type":"table"},{"data":{"code":"// ANTI-PATTERN: Allocates on every log call\nlogger.info(new LogEvent()\n .setTimestamp(Instant.now().toString())\n .setLevel(\"INFO\")\n .setUserId(userId)\n .setAction(\"login\")\n .toJson());\n\n// OPTIMIZED: Reusable builders, lazy evaluation\nimport com.yourcompany.logging.StructuredLogger;\n\nlogger.info(log -> log\n .event(\"user_login\")\n .field(\"user_id\", userId)\n .field(\"session_id\", () -> expensiveSessionLookup()) // Only called if logging\n .field(\"user_agent\", request::getUserAgent)); // Method reference, no lambda allocation\n\n// Implementation uses:\n// 1. Thread-local StringBuilder pool for JSON building\n// 2. Lazy field evaluation (suppliers only invoked if log level enabled)\n// 3. Async writes with bounded queue\n// 4. Pre-allocated field name constants to avoid String allocation","title":"High-Performance Structured Logging (Java)","language":"java"},"type":"code"},{"data":{"items":["**Lazy field evaluation** — Wrap expensive computations in lambdas/suppliers. Only evaluate if the log level is enabled. Logging `log.debug(() -> expensiveToString())` costs nothing in production if DEBUG is disabled.","**Object pooling** — Reuse log event objects and StringBuilders via thread-local pools. Eliminates allocation overhead for high-frequency logs.","**Async logging** — Write to a bounded queue; background thread handles actual I/O. Removes I/O latency from the hot path. But: risk of log loss if process crashes!","**Batched writes** — Aggregate multiple log entries before writing. Amortizes I/O overhead across many events.","**Field selection by log level** — Include full context for ERROR logs, minimal context for DEBUG. This matches access patterns—errors need context, debug is for tracing.","**Pre-compute static fields** — Service name, version, hostname don't change. Compute once at startup, include by reference."],"title":{"icon":"Zap","text":"Performance Optimization Techniques","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"icon":"AlertTriangle","title":"Async Logging: The Trade-Off","content":"Asynchronous logging dramatically improves performance but introduces a risk: if the application crashes, logs still in the async queue are lost. For critical audit logs (financial transactions, security events), use synchronous writes. For debug/info logs, async is usually acceptable.","variant":"warning"},"type":"callout"}],"titleType":"number","titleNumber":6},{"id":"framework-integration","title":"Framework Integration Patterns","content":[{"data":{"content":"Adopting structured logging in existing applications requires integrating with your language's logging framework. Here are patterns for major ecosystems:\n\n**The Adapter Pattern**\n\nMost applications use a logging facade (SLF4J, Python logging, Winston) with pluggable backends. Structured logging is typically implemented as:\n\n1. A custom log formatter/encoder at the backend layer\n2. A context mechanism for attaching request-scoped fields\n3. MDC (Mapped Diagnostic Context) for thread-local data propagation"},"type":"text"},{"data":{"code":"$7a","title":"Python Structured Logging with structlog","language":"python"},"type":"code"},{"data":{"code":"$7b","title":"Java Structured Logging with Logback + Logstash Encoder","language":"java"},"type":"code"},{"data":{"icon":"Info","title":"MDC and Async: A Dangerous Combination","content":"MDC (Mapped Diagnostic Context) uses thread-local storage. When using async processing (CompletableFuture, reactive streams), context is lost when execution moves to another thread. Use context-propagating schedulers or explicit context passing in async code.","variant":"note"},"type":"callout"}],"titleType":"number","titleNumber":7},{"id":"summary-next","title":"Summary: Structured Logging Mastery","content":[{"data":{"content":"Structured logging is the foundation of production observability. Without it, logs are archaeological artifacts requiring manual excavation. With it, logs become a queryable database of system behavior.\n\n**Key takeaways from this page:**"},"type":"text"},{"data":{"type":"ordered","items":["**Unstructured logs fail at scale** — Regex parsing is fragile, aggregation is impossible, and cross-service correlation breaks down.","**JSON is the industry standard** — Universal parser support, semi-human-readability, and rich tooling make JSON the default choice.","**Schema design is non-negotiable** — The common envelope + domain payload pattern enables consistent querying across services.","**Type consistency prevents disasters** — Field type mismatches break indexing silently. Use typed interfaces to enforce conventions.","**Parse at the edge, not centrally** — Applications should emit valid structured JSON; central parsing creates brittleness.","**Performance is solvable** — Lazy evaluation, object pooling, and async writing mitigate structured logging overhead.","**Framework integration exists** — Every major language has structured logging libraries. Adoption is configuration, not code overhaul."],"title":{"icon":"CheckCircle","text":"Key Takeaways","type":"icon","color":"primary","variant":"outline-circle"},"variant":"accent"},"type":"list"},{"data":{"content":"**What's next:**\n\nNow that you understand *how* to structure logs, the next page explores *what* to log. Log levels (DEBUG, INFO, WARN, ERROR) seem simple but are frequently misused. We'll establish rigorous criteria for when each level applies and how improper leveling corrupts observability."},"type":"text"},{"data":{"icon":"CheckCircle","title":"Page Complete","content":"You now understand structured logging—the first pillar of production-grade logging systems. You can design schemas, choose appropriate field types, build parsing pipelines, and integrate structured logging into existing applications. Next, we'll master log levels to ensure your logs contain the right information at the right verbosity.","variant":"success"},"type":"callout"}],"titleType":"number","titleNumber":8}],"isLocked":false},{"id":"seed_b3528c76292099e10922bc593f7e2317","title":"Log Levels and When to Use","order":1,"content":null,"isLocked":true},{"id":"seed_c1c42adee5d964431128ecb1a249b163","title":"Log Aggregation (ELK, OpenSearch)","order":2,"content":null,"isLocked":true},{"id":"seed_a281ba0c063e86763505177b3803ab5e","title":"Log Retention and Cost","order":3,"content":null,"isLocked":true},{"id":"seed_4706b6c901f3cdddebb02850bc753605","title":"Correlation IDs","order":4,"content":null,"isLocked":true}]}}