Metrics

We'll use Prometheus as our backend for metrics

wget https://github.com/prometheus/prometheus/releases/download/v3.5.0/prometheus-3.5.0.linux-amd64.tar.gz
tar xvf prometheus-3.5.0.linux-amd64.tar.gz
chmod +x prometheus-3.5.0.linux-amd64/prometheus
chmod +x prometheus-3.5.0.linux-amd64/promtool
sudo mv prometheus-3.5.0.linux-amd64/prometheus /usr/local/bin
sudo mv prometheus-3.5.0.linux-amd64/prometheus /usr/local/bin
sudo mkdir /etc/prometheus
sudo mv prometheus-3.5.0.linux-amd64/prometheus.yml /etc/prometheus/
sudo mkdir /var/lib/prometheus
rm -rf prometheus-3.5.0.linux-amd64

Create prometheus user

sudo useradd --no-create-home --shell /bin/false prometheus
sudo chown -R prometheus:prometheus /etc/prometheus/
sudo chown -R prometheus:prometheus /var/lib/prometheus

Create service file for Prometheus in /etc/systemd/system/prometheus.service

sudo nano /etc/systemd/system/prometheus.service

[Unit]
Description=Prometheus Server
Documentation=https://prometheus.io/docs/introduction/overview/
After=network-online.target

[Service]
User=prometheus
Restart=on-failure

ExecStart=/usr/local/bin/prometheus \
  --config.file=/etc/prometheus/prometheus.yml \
  --web.enable-remote-write-receiver \
  --storage.tsdb.path=/var/lib/prometheus/data

[Install]
WantedBy=multi-user.target

Start and enable service

sudo systemctl daemon-reload
sudo systemctl enable prometheus
sudo systemctl start prometheus
sudo systemctl status prometheus

Add Prometheus as a datasource in Grafana

Remember to save and test

In the Grafana Drilldown feature we should see some metrics already since Prometheus scrapes metrics about it self