Prometheus

Prometheus is a free software application used for event monitoring and alerting. It records real-time metrics in a time series database (allowing for high dimensionality) built using a HTTP pull model, with flexible queries and real-time alerting.

Step 1: Install Prometheus

Install the Prometheus software pacakge

# apt install prometheus

In the latest versions of Debian, noticed that the Prometheus system default configuration file needs parameters else Prometheus won’t start. Edit /etc/default/prometheus and define startup settings. I included as extra the new max retention setting of 60 days.

ARGS="--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/consoles_libraries --storage.tsdb.retention.time=14d --storage.tsdb.wal-compression"

Prometheus should be able to start now

# systemctl enable prometheus
# systemctl start prometheus

Step 2: Enable TLS/SSL

Generate host certificates for this specific service and place them in the system SSL repository

# mkdir /etc/prometheus/ssl

# cp <service-key> /etc/prometheus/ssl/prometheus.key
# cp <service-cert> /etc/prometheus/ssl/prometheus.pem

# chown root:prometheus /etc/prometheus/ssl/prometheus.key
# chown root:prometheus /etc/prometheus/ssl/prometheus.pem

# chmod 640 /etc/prometheus/ssl/prometheus.key
# chmod 644 /etc/prometheus/ssl/prometheus.pem

Then create a web configuration file /etc/prometheus/web-config.yml

tls_server_config:
  # Certificate and key files for server to use to authenticate to client.
  cert_file: /etc/prometheus/ssl/prometheus.pem
  key_file: /etc/prometheus/ssl/prometheus.key

  # CA certificate for client certificate authentication to the server.
  # client_ca_file: /etc/ssl/certs/ca-certificates.crt

  # Minimum TLS version that is acceptable.
  min_version: "TLS12"

  # Server policy for client authentication. Maps to ClientAuth Policies.
  # NOTE: If you want to enable client authentication, you need to use
  # RequireAndVerifyClientCert. Other values are insecure.
  # default = "NoClientCert" or "RequireAndVerifyClientCert"
  client_auth_type: "NoClientCert"

http_server_config:
  # Enable HTTP/2 support. Note that HTTP/2 is only supported with TLS.
  # This can not be changed on the fly.
  http2: true

Update the system default configuration file /etc/default/prometheus and add the web configuration file.

ARGS="--config.file=/etc/prometheus/prometheus.yml --storage.tsdb.path=/var/lib/prometheus/data --web.console.templates=/etc/prometheus/consoles --web.console.libraries=/etc/prometheus/consoles_libraries --storage.tsdb.retention.time=14d --web.config.file=/etc/prometheus/web-config.yml --storage.tsdb.wal-compression"

Restart the Prometheus service

# systemctl restart prometheus

Step 3: Collection

Configure the Prometheus configuration file to gather the Netdata “parent”, edit the configuration file
/etc/prometheus/prometheus.yml

# Sample config for Prometheus.

global:
  scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

  # Attach these labels to any time series or alerts when communicating with
  # external systems (federation, remote storage, Alertmanager).
  external_labels:
      monitor: 'example'

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets: ['localhost:9093']

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.
scrape_configs:
  # The job name is added as a label `job=` to any timeseries scraped from this config.
  - job_name: 'prometheus'
    scrape_interval: 5s
    scrape_timeout: 5s
    scheme: https
    tls_config:
      ca_file: '/etc/ssl/certs/ca-certificates.crt'
    static_configs:
      - targets: ['prometheus.domain.local:9090']

Check the gathering of Prometheus Netdata  by vising the url https://<prometheus server>:9090, check the gathering of data in the menu > Status >Targets