Server monitoring with node_exporter, Prometheus, and Grafana

11 May 2020 monitoring raspberry-pi cloud node-exporter prometheus grafana

Our servers are up, but are they healthy? We can set up a monitoring stack to answer that question using node_exporter, Prometheus, and Grafana.

The three services mentioned above each perform a different role in our monitoring solution:

node_exporter: expose metrics from both the Raspberry Pi and the cloud server
Prometheus: aggregate the metrics from our servers
Grafana: visualize all the data

Install node_exporter on the Raspberry Pi #

We’re going to run node_exporter directly on the server (not a docker container) so that it has easier access to the hardware to expose the server metrics. Download and install the raspberry pi version like so:

curl -SL \
  https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-armv7.tar.gz \
  > node_exporter.tar.gz
tar -zxvf node_exporter.tar.gz
sudo ln -s \
  $(pwd)/node_exporter-0.18.1.linux-armv7/node_exporter \
  /usr/local/bin/node_exporter

Next, create a service to manage node_exporter:

# Create a service for node_exporter
cat << EOF | sudo tee /etc/systemd/system/node_exporter.service > /dev/null
[Unit]
Description=NodeExporter

[Service]
TimeoutStartSec=0
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
EOF

# Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Now from your Raspberry pi you should be able to curl localhost:9100/metrics to get your server stats.

Install node_exporter in the cloud #

Download and install node_exporter on your cloud server:

curl -SL \
  https://github.com/prometheus/node_exporter/releases/download/v0.18.1/node_exporter-0.18.1.linux-amd64.tar.gz \
  > node_exporter.tar.gz
tar -zxvf node_exporter.tar.gz
sudo ln -s \
  $(pwd)/node_exporter-0.18.1.linux-amd64/node_exporter \
  /usr/local/bin/node_exporter

Create a service for node_exporter. This service is a little bit different than the one we run on the Raspberry Pi. We explicitly wait for our tinc vpn to come up and then bind only to that interface so the metrics will only be available locally.

# Create a service for node_exporter
cat << EOF | sudo tee /etc/systemd/system/node_exporter.service > /dev/null
[Unit]
Description=NodeExporter
Wants=sys-devices-virtual-net-tun0.device
After=sys-devices-virtual-net-tun0.device

[Service]
TimeoutStartSec=0
ExecStart=/usr/local/bin/node_exporter --web.listen-address=10.0.0.1:9100

[Install]
WantedBy=multi-user.target
EOF

# Enable and start the service
sudo systemctl daemon-reload
sudo systemctl enable node_exporter
sudo systemctl start node_exporter

Configure and run Prometheus #

We’ll pull the example config file from GitHub and add the configuration for the Raspberry Pi and the cloud. For the Raspberry Pi, Prometheus will be running in a docker container but will need to connect to the host machine to access node_exporter. We can specify the docker0 gateway address to do just that (172.17.0.1).

curl -SL \
  https://raw.githubusercontent.com/prometheus/prometheus/master/documentation/examples/prometheus.yml \
  > prometheus.yml

cat << EOF >> prometheus.yml

  - job_name: 'interwebs.rpi'
    static_configs:
    - targets: ['172.17.0.1:9100']

  - job_name: 'interwebs.cloud'
    static_configs:
    - targets: ['10.0.0.1:9100']
EOF

The Grafana container will need to talk to the Prometheus container, so we’ll set up a new docker network that will let them communicate before we start the containers

# Create the docker network
docker network create --driver bridge metrics

docker run \
  -v $(pwd)/prometheus.yml:/etc/prometheus/prometheus.yml \
  -p 9090:9090 \
  --name prometheus \
  --network metrics \
  -d \
  prom/prometheus

Prometheus should now be available on your pi on port 9090: http://interwebs.local:9090

Configure and run Grafana #

We’ll start the Grafana container and then configure it through the web interface.

mkdir grafana
docker run \
  -v $(pwd)/grafana:/var/lib/grafana \
  -p 3000:3000 \
  --user $(id -u) \
  --name grafana \
  --network metrics \
  -d \
  grafana/grafana

Connect to grafana on your pi on port 3000: http://interwebs.local:3000 and login with the default user and password: admin/admin

Set up a new data source and specify your prometheus server: http://prometheus:9090. At this point I chose to import a dashboard: Node Exporter Full dashboard for Grafana | Grafana Labs. On the import page you can paste in the code 1860 and you’re good to go.

This dashboard exposes pretty much all the data from node_exporter. You’ll get a nice overview of both your cloud server and Raspberry Pi: system load, cpu, memory, disk, etc…

← Home