Linux: Prometheus-Node-Exporter

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting.  Prometheus also has node exporters including windows to export OS specific metrics.

Step 1: Install Node Exporter

Install the Prometheus Node Exporter software pacakge

# apt install -y prometheus-node-exporter

Enable and start the node exporter:

# systemctl enable prometheus-node-exporter
# systemctl start prometheus-node-exporter

Step 2: Enable TLS/SSL and Authentication

Generate host certificates for this specific service and place them in the system SSL repository

# mkdir /etc/prometheus/ssl

# cp <service-key> /etc/prometheus/ssl/prometheus-node-exporter.key
# cp <service-cert> /etc/prometheus/ssl/prometheus-node-exporter.pem

# chown root:prometheus /etc/prometheus/ssl/prometheus-node-exporter.key
# chown root:prometheus /etc/prometheus/ssl/prometheus-node-exporter.pem

# chmod 640 /etc/prometheus/ssl/prometheus-node-exporter.key
# chmod 644 /etc/prometheus/ssl/prometheus-node-exporter.pem

Create the following python script to generate an hashed password gen-pass.py

import getpass
import bcrypt

password = getpass.getpass("password: ")
hashed_password = bcrypt.hashpw(password.encode("utf-8"), bcrypt.gensalt())
print(hashed_password.decode())

Run the script and it will prompt for a password:

# python3 gen-pass.py
password:
$2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay

Create a web configuration file and define the encrypted service /etc/prometheus/web-config.yml

tls_server_config:
  cert_file: /etc/ssl/certs/prometheus-node-exporter.pem
  key_file: /etc/ssl/private/prometheus-node-exporter.key

  # Minimum TLS version that is acceptable.
  min_version: "TLS12"

basic_auth_users:
  prometheus: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay

Edit the Prometheus node exporter system default configuration file and define the web service /etc/default/prometheus-node-exporter

ARGS="--web.config=/etc/prometheus/web-config.yml"

Restart the node exporter:

# systemctl restart prometheus-node-exporter

Step 3: Enable Extra metrics

Some metrics are disabled by default, but you can force enable them in the system default configuration file /etc/default/prometheus-node-exporter

ARGS="--web.config=/etc/prometheus/web-config.yml --web.max-requests=5 --collector.ntp --collector.tcpstat"

Restart the node exporter:

# systemctl restart prometheus-node-exporter

Step 4: Collection

From Prometheus server define the new node collection policy by editing  /etc/prometheus/prometheus.yml

 - job_name: node-prometheus
   scrape_interval: 15s
   scrape_timeout: 10s
   scheme: https
   tls_config:
     ca_file: '/etc/ssl/certs/ca-certificates.crt'
   basic_auth:
      username: prometheus
      password: $2b$12$hNf2lSsxfm0.i4a.1kVpSOVyBCfIB51VRjgBUyv6kdnyTlgWj81Ay
    static_configs:
      - targets: ['host-01:9100', 'host-02:9100' ]
    relabel_configs:
      - source_labels: ['__address__']
        separator: ':'
        regex: '(.*):.*'
        target_label: 'instance'
        replacement: '${1}'

Reload Prometheus so it polls the new changes:

# systemctl reload prometheus