120 lines
3.0 KiB
Org Mode
120 lines
3.0 KiB
Org Mode
#+TITLE: Monitoring and Logging
|
|
#+OPTIONS: num:nil
|
|
#+DATE: <2026-01-18 Sun 23:00>
|
|
#+filetags: :learning:notes:
|
|
#+WIP:
|
|
#+COMMENTS: t
|
|
#+SLUG: monitoring-and-logging
|
|
|
|
* Monitoring and Logging Tools Overview
|
|
|
|
Modern systems, especially cloud and distributed applications, require tools to observe performance, detect issues, and analyse logs. Tools like Kibana, Grafana, and Prometheus are widely used for these purposes.
|
|
|
|
* Kibana
|
|
** Purpose
|
|
Log visualisation and analysis.
|
|
|
|
** How it works
|
|
Kibana is part of the Elastic Stack (formerly ELK Stack) and is used to visualise data stored in Elasticsearch.
|
|
|
|
** Key Features
|
|
- Search and analyse large log datasets
|
|
- Interactive dashboards
|
|
- Log filtering and querying
|
|
- Security and anomaly detection features
|
|
|
|
** Common Use Cases
|
|
- Viewing application logs
|
|
- Debugging errors in production systems
|
|
- Security monitoring
|
|
|
|
** Example Setup
|
|
1. Install Elasticsearch
|
|
2. Send logs using Logstash or Filebeat
|
|
3. Use Kibana to visualise logs
|
|
|
|
* Prometheus
|
|
** Purpose
|
|
Metrics collection and monitoring.
|
|
|
|
Prometheus is designed to collect numeric metrics over time from systems and applications.
|
|
|
|
** Key Features
|
|
- Time-series database
|
|
- Powerful query language (PromQL)
|
|
- Built-in alerting
|
|
- Pull-based metrics collection
|
|
|
|
** Common Use Cases
|
|
- Monitoring servers and containers
|
|
- Tracking CPU, memory, and request latency
|
|
- Infrastructure monitoring in Kubernetes
|
|
|
|
** Example Setup
|
|
1. Install Prometheus
|
|
2. Configure targets to scrape metrics
|
|
3. Expose metrics via a /metrics endpoint
|
|
4. Query metrics using PromQL
|
|
|
|
* Grafana
|
|
** Purpose
|
|
Visualisation and dashboards.
|
|
|
|
Grafana is commonly used with Prometheus but can connect to many different data sources.
|
|
|
|
** Key Features
|
|
- Highly customisable dashboards
|
|
- Supports many data sources (Prometheus, Elasticsearch, databases)
|
|
- Alerting and notifications
|
|
- Real-time visual monitoring
|
|
|
|
** Common Use Cases
|
|
- Infrastructure monitoring dashboards
|
|
- Business metrics visualisation
|
|
- Combining logs, metrics, and traces
|
|
|
|
** Example Setup
|
|
1. Install Grafana
|
|
2. Connect a data source (Prometheus, Elasticsearch, etc.)
|
|
3. Build dashboards using panels and queries
|
|
|
|
* Other Popular Tools
|
|
|
|
** Logstash
|
|
- Log processing pipeline
|
|
- Collects, transforms, and sends logs to Elasticsearch
|
|
|
|
** Filebeat
|
|
- Lightweight log shipper
|
|
- Sends logs from servers to Elasticsearch
|
|
|
|
** Loki
|
|
- Log aggregation system designed by Grafana Labs
|
|
- Integrates well with Grafana dashboards
|
|
|
|
* Best Practices for Monitoring and Logging
|
|
|
|
** Monitor Key Metrics
|
|
- CPU usage
|
|
- Memory usage
|
|
- Request latency
|
|
- Error rates
|
|
|
|
** Centralise Logs
|
|
Send logs from all services to a single platform.
|
|
|
|
** Use Alerts
|
|
Configure alerts to notify you of abnormal behavior.
|
|
|
|
** Combine Logs and Metrics
|
|
- Metrics tell you that something is wrong
|
|
- Logs help you understand why it is wrong
|
|
|
|
** Create Meaningful Dashboards
|
|
Focus on actionable information instead of displaying excessive data.
|
|
|
|
* Simple Summary
|
|
- Prometheus collects metrics
|
|
- Grafana visualises metrics
|
|
- Kibana analyses logs
|