The Hidden Cost of Logging in Microservices
In distributed systems, observability is essential. However, the pricing models of modern APM and SIEM providers (like Datadog, Splunk, and New Relic) heavily penalize unoptimized logging. Adding a single debug log to a service handling 5,000 requests per second can inadvertently cost tens of thousands of dollars per month.
Why is Logging So Expensive?
- Volume-based Ingestion: Providers charge based on the total bytes sent (typically $0.10 to $1.50 per GB).
- Indexing & Retention: Parsing and indexing logs for search queries (like Datadog's "Retained Events") is computationally expensive and billed per million events.
- Data Egress: If your log shipper sends data out of your AWS/GCP VPC to a SaaS provider, you will also pay Cloud Egress fees.
Strategies to Reduce Log Cost
To prevent cost shock while maintaining observability, implement the following patterns:
- Dynamic Log Levels: Run services in
INFOorWARNmode in production, but expose an API endpoint to dynamically flip a single node toDEBUGmode for 15 minutes when troubleshooting. - Sampling: Only log 1% of successful HTTP 200 requests, but log 100% of HTTP 5xx errors.
- Metrics over Logs: If you are logging to count occurrences (e.g., "User logged in"), use StatsD/Prometheus metrics instead. Metrics are aggregated in memory and are orders of magnitude cheaper than string-based logs.