Loki

After having very positive experiences, we decided to replace our ELK stacks with Loki, primarily for the following reasons:

• Loki is significantly more cost-effective than the storage requirements of Elasticsearch.
• The PromQL-like query language, familiar to users of Prometheus, makes it easier for DevOps and SRE teams who already use Prometheus for monitoring to work with logs.
• Loki's native integration with Kubernetes simplifies the setup and configuration process.
• Loki typically requires less maintenance and overhead compared to ELK.

Archiving indexed log data with a system like Elasticsearch can be expensive, and archiving it as simple text files makes it hard to query. Loki solves this issue by adding a reference database based on Kubernetes labels to each log line, similar to Prometheus, but storing the log data inside a simple blob storage like AWS S3. This allows the user to query the data by predefined labels and keeps the costs for indexing low.

Another benefit is that Loki does not have an endpoint for mutating log data, making the data immutable from a potentially compromised system.

At AOE, we use Loki for long-term log archiving in several Kubernetes clusters.