3.0 KiB
Executable File
Monitoring and Logging
- Monitoring and Logging Tools Overview
- Kibana
- Prometheus
- Grafana
- Other Popular Tools
- Best Practices for Monitoring and Logging
- Simple Summary
Monitoring and Logging Tools Overview
Modern systems, especially cloud and distributed applications, require tools to observe performance, detect issues, and analyse logs. Tools like Kibana, Grafana, and Prometheus are widely used for these purposes.
Kibana
Purpose
Log visualisation and analysis.
How it works
Kibana is part of the Elastic Stack (formerly ELK Stack) and is used to visualise data stored in Elasticsearch.
Key Features
- Search and analyse large log datasets
- Interactive dashboards
- Log filtering and querying
- Security and anomaly detection features
Common Use Cases
- Viewing application logs
- Debugging errors in production systems
- Security monitoring
Example Setup
- Install Elasticsearch
- Send logs using Logstash or Filebeat
- Use Kibana to visualise logs
Prometheus
Purpose
Metrics collection and monitoring.
Prometheus is designed to collect numeric metrics over time from systems and applications.
Key Features
- Time-series database
- Powerful query language (PromQL)
- Built-in alerting
- Pull-based metrics collection
Common Use Cases
- Monitoring servers and containers
- Tracking CPU, memory, and request latency
- Infrastructure monitoring in Kubernetes
Example Setup
- Install Prometheus
- Configure targets to scrape metrics
- Expose metrics via a /metrics endpoint
- Query metrics using PromQL
Grafana
Purpose
Visualisation and dashboards.
Grafana is commonly used with Prometheus but can connect to many different data sources.
Key Features
- Highly customisable dashboards
- Supports many data sources (Prometheus, Elasticsearch, databases)
- Alerting and notifications
- Real-time visual monitoring
Common Use Cases
- Infrastructure monitoring dashboards
- Business metrics visualisation
- Combining logs, metrics, and traces
Example Setup
- Install Grafana
- Connect a data source (Prometheus, Elasticsearch, etc.)
- Build dashboards using panels and queries
Other Popular Tools
Logstash
- Log processing pipeline
- Collects, transforms, and sends logs to Elasticsearch
Filebeat
- Lightweight log shipper
- Sends logs from servers to Elasticsearch
Loki
- Log aggregation system designed by Grafana Labs
- Integrates well with Grafana dashboards
Best Practices for Monitoring and Logging
Monitor Key Metrics
- CPU usage
- Memory usage
- Request latency
- Error rates
Centralise Logs
Send logs from all services to a single platform.
Use Alerts
Configure alerts to notify you of abnormal behavior.
Combine Logs and Metrics
- Metrics tell you that something is wrong
- Logs help you understand why it is wrong
Create Meaningful Dashboards
Focus on actionable information instead of displaying excessive data.
Simple Summary
- Prometheus collects metrics
- Grafana visualises metrics
- Kibana analyses logs