Linux: Prometheus-Node-Exporter

Prometheus is an open-source metrics monitoring and alerting system. It stores numeric time series in its own time-series database (TSDB), evaluates alert rules against them, and hands firing alerts to Alertmanager, which handles routing and notifications. Prometheus uses a pull model: the server scrapes an HTTP endpoint (/metrics) on every target at a fixed interval.

Node Exporter is the official exporter for Linux and Unix host metrics: CPU, memory, disks, filesystems, network, time sync, systemd units and more. (Windows hosts use the separate windows_exporter project. Node Exporter does not run on Windows.)

Out of the box, Node Exporter listens on every interface, port 9100, plain HTTP, with no authentication. Anyone who can reach that port can read your kernel version, mount points, interface names and load profile. This guide turns it into a properly locked-down endpoint:

  • Firewall so that only the Prometheus server can reach port 9100
  • TLS 1.3 with a certificate whose name matches the scrape target
  • HTTP basic auth with a bcrypt hash on the node and the plaintext password in a password_file on the server
  • systemd sandboxing for the exporter process
  • Useful extra collectors (systemd, textfile), with the deprecated ntp collector dropped
  • A scrape job that actually validates, plus a solid set of alert rules
  • Validation with promtool, curl and openssl at every step

Tested on: Debian 13 “trixie” for both the node and the Prometheus server, using the Debian packages prometheus-node-exporter and prometheus. All commands run as root (or with sudo).


Table of contents

  1. Placeholders
  2. Firewall first
  3. Install Node Exporter
  4. TLS certificate and key
  5. Basic-auth credentials
  6. The web configuration file
  7. Exporter flags and collectors
  8. systemd hardening
  9. The textfile collector for custom metrics
  10. The Prometheus scrape job
  11. Alert rules
  12. Testing
  13. Day-2 operations
  14. Hardening checklist
  15. Appendix A: a minimal private CA
  16. Appendix B: mutual TLS (optional)

1. Placeholders

Replace these values everywhere in the guide:

PlaceholderMeaning
host-01.example.netFQDN of the monitored node. It must match the TLS certificate’s SAN and the scrape target.
192.0.2.21IP address on the node that Node Exporter will listen on (management or internal interface)
prom-01.example.netFQDN of the Prometheus server
192.0.2.10IPv4 address of the Prometheus server (add a second server here if you run an HA pair)
2001:db8::10IPv6 address of the Prometheus server (delete the IPv6 lines if you don’t use IPv6)
monitoring-ca.crtThe CA certificate that signed the node certificates (see Appendix A)

Two terms used below: the node is the machine being monitored, and the server is the Prometheus host.


2. Firewall first

On Debian the package starts the service as soon as it is installed, listening on :9100 over plain HTTP. Lock the port down before you install it.

This uses nftables, Debian’s default firewall framework. The rules live in their own table, so they work next to whatever ruleset you already have.

On the node, create /etc/nftables.d/node-exporter.nft:

install -d -m 0755 /etc/nftables.d

cat > /etc/nftables.d/node-exporter.nft <<'EOF'
# Only the Prometheus server(s) may reach Node Exporter on tcp/9100.
table inet node_exporter {
    set prometheus_v4 {
        type ipv4_addr
        elements = { 192.0.2.10 }
    }

    set prometheus_v6 {
        type ipv6_addr
        elements = { 2001:db8::10 }
    }

    chain input {
        type filter hook input priority filter - 10; policy accept;

        tcp dport 9100 iif "lo" accept
        tcp dport 9100 ip  saddr @prometheus_v4 accept
        tcp dport 9100 ip6 saddr @prometheus_v6 accept
        tcp dport 9100 counter drop
    }
}
EOFCode language: PHP (php)

Make sure /etc/nftables.conf includes the directory. Add this line at the end if it isn’t already there:

grep -q 'nftables.d' /etc/nftables.conf || \
  echo 'include "/etc/nftables.d/*.nft"' >> /etc/nftables.confCode language: JavaScript (javascript)

Check the syntax, then load and enable it:

nft -c -f /etc/nftables.conf      # dry run: no output means OK
systemctl enable --now nftables
systemctl reload nftables          # if it was already running
nft list table inet node_exporterCode language: PHP (php)

Why a separate table with a final drop? In nftables, a drop in any base chain is final, and an accept only ends evaluation of that chain. This table therefore drops every unwanted 9100 packet no matter what your other tables do. If your main ruleset uses policy drop, you also need an accept for tcp/9100 from the Prometheus server in that ruleset.

Using ufw instead? ufw allow from 192.0.2.10 to any port 9100 proto tcp followed by ufw deny 9100/tcp gives the same result. Don’t run ufw and a hand-written nftables ruleset on the same host.


3. Install Node Exporter

apt update
apt install prometheus-node-exporter

That’s all. Debian enables and starts the unit prometheus-node-exporter.service automatically, so you don’t need separate systemctl enable or systemctl start commands.

Check what you got:

prometheus-node-exporter --version
systemctl status prometheus-node-exporter --no-pager

The package also recommends prometheus-node-exporter-collectors, a set of systemd timers that write extra metrics (APT updates pending, reboot required, SMART, NVMe, and more) to the textfile collector directory. Keep it, because the alert rules in section 11 use its node_reboot_required metric.

Debian’s packaging uses these paths:

WhatWhere
Binary/usr/bin/prometheus-node-exporter
ArgumentsARGS="..." in /etc/default/prometheus-node-exporter
Service userprometheus
Textfile directory/var/lib/prometheus/node-exporter

4. TLS certificate and key

The exporter needs a certificate whose Subject Alternative Name (SAN) contains the exact name Prometheus uses as the scrape target (here host-01.example.net). If the names don’t match, Prometheus refuses the connection, which is the correct behaviour.

There are two common sources:

  • An internal CA (recommended for monitoring traffic). Appendix A shows a minimal OpenSSL CA. step-ca, Vault PKI or FreeIPA work just as well.
  • A public ACME certificate (Let’s Encrypt) if the node has a public DNS name and you already run certbot. Copy or deploy it into the paths below with a deploy hook.

Put the certificate and key in one place, readable only by root and the prometheus group:

install -d -m 0750 -o root -g prometheus /etc/prometheus/ssl

install -m 0644 -o root -g prometheus host-01.crt \
  /etc/prometheus/ssl/node-exporter.crt
install -m 0640 -o root -g prometheus host-01.key \
  /etc/prometheus/ssl/node-exporter.key

Full chain: if an intermediate CA signed the certificate, node-exporter.crt must contain the leaf certificate followed by the intermediate(s).

Check that the certificate carries the right name and matches the key:

openssl x509 -in /etc/prometheus/ssl/node-exporter.crt -noout -subject -ext subjectAltName -enddate

# Both hashes must be identical:
openssl x509 -in /etc/prometheus/ssl/node-exporter.crt -noout -pubkey | sha256sum
openssl pkey -in /etc/prometheus/ssl/node-exporter.key -pubout | sha256sumCode language: PHP (php)

5. Basic-auth credentials

Node Exporter stores only a bcrypt hash. Prometheus needs the plaintext password, which it reads from a file, so the secret never appears in prometheus.yml.

5.1 On the Prometheus server: generate and store the password

install -d -m 0750 -o root -g prometheus /etc/prometheus/secrets

( umask 027
  openssl rand -base64 32 | tr -d '\n' > /etc/prometheus/secrets/node-exporter.pass )
chgrp prometheus /etc/prometheus/secrets/node-exporter.pass
ls -l /etc/prometheus/secrets/node-exporter.pass   # -rw-r----- root prometheusCode language: PHP (php)

5.2 Hash it

htpasswd from apache2-utils produces a bcrypt hash and is easier than a custom script. -i reads the password from stdin, so it never appears in the process list or your shell history:

apt install apache2-utils

htpasswd -niBC 12 "" < /etc/prometheus/secrets/node-exporter.pass | tr -d ':\n'; echoCode language: JavaScript (javascript)

The output looks like $2y$12$โ€ฆ. Copy it for the next step. The hash isn’t secret the way a password is, but treat it as sensitive anyway: never publish a real one.

Prefer Python? Install python3-bcrypt, then run python3 -c 'import bcrypt,getpass; print(bcrypt.hashpw(getpass.getpass().encode(), bcrypt.gensalt(12)).decode())'. Note the .decode(). Without it Python prints b'$2b$โ€ฆ', and pasting that into the config breaks authentication.

Why cost 12? The cost factor is a trade-off. Higher values resist offline cracking better, but every cache miss costs CPU on the node. 10โ€“12 is the sensible range for a long random machine password. The exporter caches successful verifications, so repeated scrapes stay cheap.


6. The web configuration file

On the node, create /etc/prometheus/node-exporter-web.yml:

# /etc/prometheus/node-exporter-web.yml
# Re-read on every request: changes apply without a restart.

tls_server_config:
  cert_file: /etc/prometheus/ssl/node-exporter.crt
  key_file: /etc/prometheus/ssl/node-exporter.key
  # Prometheus and Node Exporter are both Go and speak TLS 1.3 natively.
  # Use TLS12 only if some legacy scraper must connect.
  min_version: TLS13

http_server_config:
  headers:
    Strict-Transport-Security: max-age=31536000
    X-Content-Type-Options: nosniff
    X-Frame-Options: deny

basic_auth_users:
  # username: bcrypt hash (quote it, because it contains '$')
  prometheus: "$2y$12$REPLACE.WITH.THE.HASH.FROM.SECTION.5.2..................."Code language: PHP (php)

Lock it down, since it holds the hash:

chown root:prometheus /etc/prometheus/node-exporter-web.yml
chmod 0640 /etc/prometheus/node-exporter-web.yml

Validate it. promtool check web-config comes with the prometheus package, which you don’t want on every node. Copy the file to the Prometheus server and check it there:

scp host-01.example.net:/etc/prometheus/node-exporter-web.yml /tmp/
promtool check web-config /tmp/node-exporter-web.yml && rm /tmp/node-exporter-web.ymlCode language: JavaScript (javascript)

The check only validates syntax and the hash format. It doesn’t open the certificate files.


7. Exporter flags and collectors

Edit /etc/default/prometheus-node-exporter on the node. Replace the existing ARGS= line with:

ARGS="--web.config.file=/etc/prometheus/node-exporter-web.yml \
  --web.listen-address=192.0.2.21:9100 \
  --web.max-requests=10 \
  --collector.textfile.directory=/var/lib/prometheus/node-exporter \
  --collector.systemd \
  --collector.systemd.unit-include=(ssh|cron|nftables|chrony|systemd-timesyncd|unattended-upgrades)[.]service"Code language: JavaScript (javascript)

The flags do the following:

FlagWhy
--web.config.fileTurns on TLS and basic auth. This is the current flag name. The old --web.config was renamed and later removed, so current versions refuse to start with it.
--web.listen-addressBinds only to the internal or management IP instead of every interface. This adds defence in depth on top of the firewall.
--web.max-requests=10Caps concurrent scrapes (the default is 40). Leave headroom for an HA pair plus a manual curl. A very low value such as 5 can cause failed scrapes when a scrape is slow.
--collector.textfile.directoryDebian’s default already, but set explicitly so it’s obvious where custom .prom files go.
--collector.systemdExports node_systemd_unit_state, so you can alert on failed units. It’s off by default.
--collector.systemd.unit-includeLimits the systemd collector to units you care about. Exporting every unit inflates cardinality. The regex is anchored automatically. [.] is used instead of \. so the backslash survives systemd’s environment-file parsing.

Optional collectors worth knowing about:

CollectorNotes
--collector.tcpstatTCP connection states. Useful on busy servers, but it costs extra CPU on hosts with very many sockets.
--collector.processesProcess and thread counts by state.
--collector.interruptsPer-CPU interrupt counts. High cardinality, so enable it only when you need it.
--web.disable-exporter-metricsDrops the exporter’s own go_* and promhttp_* metrics if you don’t want them.

Do not enable --collector.ntp. It is deprecated: it queries an NTP server over the network on every scrape and reports that server’s view, not the kernel’s. The timex collector, enabled by default, reads the kernel clock discipline directly (node_timex_sync_status, node_timex_offset_seconds, node_timex_maxerror_seconds), and works with chrony, systemd-timesyncd or ntpd. The time-sync alerts in section 11 use it.

To see every available flag and which collectors are on by default:

prometheus-node-exporter --help 2>&1 | less

8. systemd hardening

Node Exporter only needs to read /proc, /sys and a few files, and to talk to D-Bus for the systemd collector. Add a drop-in that sandboxes it and waits for the network, since the exporter now binds to a specific IP:

systemctl edit prometheus-node-exporter

Paste:

[Unit]
# Bind to a specific IP: make sure it exists first.
Wants=network-online.target
After=network-online.target

[Service]
NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
UMask=0077Code language: PHP (php)

Apply all of it:

systemctl daemon-reload
systemctl restart prometheus-node-exporter
systemctl status prometheus-node-exporter --no-pager
journalctl -u prometheus-node-exporter -b --no-pager | tail -n 30
systemd-analyze security prometheus-node-exporter

In the journal, look for TLS is enabled. and Listening on address=192.0.2.21:9100. Watch for collector errors such as collector failed or permission denied.

Deliberately left out: ProtectHome=yes and ProtectProc=invisible. They hide /home and /proc/1/mounts, which breaks the filesystem collector. PrivateDevices=yes is usually fine too, but test it on your hardware first.


9. The textfile collector for custom metrics

The textfile collector exports any *.prom file in /var/lib/prometheus/node-exporter. Use it for things only a script can know, such as backup age, certificate expiry of local services, or RAID state.

The rule: write to a temporary file in the same directory, then mv it into place. The rename is atomic, so the exporter never reads a half-written file. The temporary name must not end in .prom.

Example: record a successful backup.

cat > /usr/local/sbin/node-metric-backup-ok <<'EOF'
#!/bin/sh
# Call at the end of a successful backup run.
set -eu
dir=/var/lib/prometheus/node-exporter
tmp=$(mktemp "$dir/.backup.XXXXXX")
cat > "$tmp" <<METRICS
# HELP backup_last_success_timestamp_seconds Unix time of the last successful backup.
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
METRICS
chmod 0644 "$tmp"
mv "$tmp" "$dir/backup.prom"
EOF
chmod 0755 /usr/local/sbin/node-metric-backup-okCode language: PHP (php)

If a file has a syntax error, the exporter reports node_textfile_scrape_error 1. Section 11 alerts on that.


10. The Prometheus scrape job

Everything in this section happens on the Prometheus server.

10.1 Trust the CA

install -d -m 0755 /etc/prometheus/ssl
install -m 0644 monitoring-ca.crt /etc/prometheus/ssl/monitoring-ca.crt

If you use publicly trusted certificates, point ca_file at /etc/ssl/certs/ca-certificates.crt instead. For internal monitoring, a dedicated CA is better: Prometheus then trusts only your CA for this job, not every public CA.

10.2 The job

Add this under scrape_configs: in /etc/prometheus/prometheus.yml. YAML is indentation-sensitive. Use spaces, never tabs, and keep every key of the job at the same level:

scrape_configs:
  # ... your existing jobs ...

  - job_name: node
    scrape_interval: 15s
    scrape_timeout: 10s
    scheme: https
    tls_config:
      ca_file: /etc/prometheus/ssl/monitoring-ca.crt
      min_version: TLS13
    basic_auth:
      username: prometheus
      # The PLAINTEXT password, read from a file. Never the bcrypt hash.
      password_file: /etc/prometheus/secrets/node-exporter.pass
    static_configs:
      - targets:
          - host-01.example.net:9100
          - host-02.example.net:9100
    relabel_configs:
      # instance="host-01.example.net" instead of "host-01.example.net:9100"
      - source_labels: [__address__]
        regex: '(.+):\d+'
        target_label: instance
        replacement: '$1'Code language: PHP (php)

Notes on this job:

  • Targets are FQDNs that match each node’s certificate SAN. Short names like host-01 only work if the certificate also lists them.
  • The relabel regex is anchored by Prometheus, and the greedy (.+) keeps everything up to the last colon. That means it also handles [2001:db8::21]:9100. Without it, dashboards show the port in every legend.
  • Many hosts? Replace static_configs with file_sd_configs pointing at /etc/prometheus/targets/node/*.yml. Prometheus picks up changes to those files without a reload.

10.3 Load rules too

Add this at the top level of prometheus.yml if it isn’t there yet:

rule_files:
  - /etc/prometheus/rules/*.ymlCode language: JavaScript (javascript)

10.4 Validate and reload

promtool check config /etc/prometheus/prometheus.yml
systemctl reload prometheus
journalctl -u prometheus -n 20 --no-pager   # look for "Completed loading of configuration file"Code language: PHP (php)

promtool check config also checks that password_file and ca_file exist and loads every rule file. A failed check means don’t reload.


11. Alert rules

Create /etc/prometheus/rules/node.yml on the server. The thresholds are sane defaults. Tune the for: durations and percentages to your environment.

# /etc/prometheus/rules/node.yml
groups:
  - name: node-exporter
    rules:

      # --- Availability -------------------------------------------------------
      - alert: NodeExporterDown
        expr: up{job="node"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Node exporter on {{ $labels.instance }} is unreachable"
          description: "Prometheus has failed to scrape {{ $labels.instance }} for 5 minutes (host down, firewall, TLS or auth problem)."

      # --- Filesystems --------------------------------------------------------
      - alert: NodeFilesystemAlmostFull
        expr: |
          (
            node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|ramfs"}
              / node_filesystem_size_bytes{job="node",fstype!~"tmpfs|ramfs"} < 0.10
          )
          and node_filesystem_readonly{job="node",fstype!~"tmpfs|ramfs"} == 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} has less than 10% free"
          description: "Only {{ $value | humanizePercentage }} space left on {{ $labels.device }} ({{ $labels.mountpoint }})."

      - alert: NodeFilesystemFillingUp
        expr: |
          (
            node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|ramfs"}
              / node_filesystem_size_bytes{job="node",fstype!~"tmpfs|ramfs"} < 0.40
          )
          and predict_linear(node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|ramfs"}[6h], 24 * 3600) < 0
          and node_filesystem_readonly{job="node",fstype!~"tmpfs|ramfs"} == 0
        for: 1h
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 24h"
          description: "At the current rate, {{ $labels.mountpoint }} runs out of space within a day."

      - alert: NodeFilesystemInodesLow
        expr: |
          (
            node_filesystem_files_free{job="node",fstype!~"tmpfs|ramfs"}
              / node_filesystem_files{job="node",fstype!~"tmpfs|ramfs"} < 0.10
          )
          and node_filesystem_readonly{job="node",fstype!~"tmpfs|ramfs"} == 0
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is running out of inodes"
          description: "Only {{ $value | humanizePercentage }} inodes left."

      - alert: NodeFilesystemReadOnly
        expr: node_filesystem_readonly{job="node",fstype=~"ext4|xfs|btrfs"} == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is read-only"
          description: "A normally writable filesystem is mounted read-only. This often means I/O errors or filesystem corruption. Check dmesg."

      # --- Memory & CPU -------------------------------------------------------
      - alert: NodeMemoryLow
        expr: node_memory_MemAvailable_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"} < 0.10
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} is low on memory"
          description: "Only {{ $value | humanizePercentage }} of RAM is available."

      - alert: NodeOOMKill
        expr: increase(node_vmstat_oom_kill{job="node"}[15m]) > 0
        labels:
          severity: warning
        annotations:
          summary: "OOM killer fired on {{ $labels.instance }}"
          description: "The kernel killed at least one process for lack of memory in the last 15 minutes."

      - alert: NodeHighCPU
        expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) > 0.90
        for: 30m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.instance }} CPU above 90% for 30 minutes"
          description: "Average CPU utilisation is {{ $value | humanizePercentage }}."

      # --- Time sync (timex collector) ---------------------------------------
      - alert: NodeClockNotSynchronising
        expr: min_over_time(node_timex_sync_status{job="node"}[5m]) == 0 and node_timex_maxerror_seconds{job="node"} >= 16
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Clock on {{ $labels.instance }} is not synchronising"
          description: "The kernel reports the clock as unsynchronised. Check chrony or systemd-timesyncd."

      - alert: NodeClockSkew
        expr: abs(node_timex_offset_seconds{job="node"}) > 0.05
        for: 10m
        labels:
          severity: warning
        annotations:
          summary: "Clock on {{ $labels.instance }} is off by more than 50 ms"
          description: "Current offset: {{ $value | humanizeDuration }}."

      # --- systemd & textfile -------------------------------------------------
      - alert: NodeSystemdUnitFailed
        expr: node_systemd_unit_state{job="node",state="failed"} == 1
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "{{ $labels.name }} failed on {{ $labels.instance }}"
          description: "systemd reports the unit {{ $labels.name }} in state 'failed'."

      - alert: NodeTextfileScrapeError
        expr: node_textfile_scrape_error{job="node"} == 1
        for: 15m
        labels:
          severity: warning
        annotations:
          summary: "Textfile collector error on {{ $labels.instance }}"
          description: "A .prom file in /var/lib/prometheus/node-exporter is unreadable or malformed."

      - alert: NodeRebootRequired
        # Provided by prometheus-node-exporter-collectors.
        expr: node_reboot_required{job="node"} > 0
        for: 24h
        labels:
          severity: info
        annotations:
          summary: "{{ $labels.instance }} needs a reboot"
          description: "Updated packages (usually the kernel or libc) need a reboot to take effect."

      - alert: BackupTooOld
        # Only fires for hosts that publish the metric from section 9.
        expr: time() - backup_last_success_timestamp_seconds{job="node"} > 26 * 3600
        labels:
          severity: warning
        annotations:
          summary: "No successful backup on {{ $labels.instance }} for over 26 hours"
          description: "Last success was {{ $value | humanizeDuration }} ago."Code language: PHP (php)

Validate the rules, then reload:

promtool check rules /etc/prometheus/rules/node.yml
promtool check config /etc/prometheus/prometheus.yml
systemctl reload prometheus

Alerts only reach a human if Prometheus is connected to Alertmanager (the alerting: block in prometheus.yml). Setting up Alertmanager is outside the scope of this guide. Until you have it, firing alerts show up under Alerts in the Prometheus web UI.


12. Testing

Don’t skip these checks. Each one proves a different layer works.

12.1 On the node: listening on the right address only

ss -tlnp 'sport = :9100'
# Expect: LISTEN ... 192.0.2.21:9100 ... prometheus-nodeCode language: PHP (php)

12.2 From the Prometheus server: TLS and authentication

CA=/etc/prometheus/ssl/monitoring-ca.crt
URL=https://host-01.example.net:9100/metrics

# No credentials -> 401
curl -sS -o /dev/null -w '%{http_code}\n' --cacert "$CA" "$URL"

# Wrong password -> 401
curl -sS -o /dev/null -w '%{http_code}\n' --cacert "$CA" -u prometheus:wrong "$URL"

# Correct credentials -> 200 and metrics (curl prompts for the password)
curl -sS --cacert "$CA" -u prometheus "$URL" | grep -E '^node_(uname_info|timex_sync_status|systemd_unit_state.*failed)' | head

# Plain HTTP must not work (expect HTTP 400: "client sent an HTTP request to an HTTPS server")
curl -sS -o /dev/null -w '%{http_code}\n' "http://host-01.example.net:9100/metrics"Code language: PHP (php)

12.3 TLS version and certificate

# TLS 1.2 must be refused (handshake failure / alert protocol version)
openssl s_client -connect host-01.example.net:9100 -tls1_2 </dev/null

# TLS 1.3 works and the chain verifies (look for "Verify return code: 0 (ok)")
openssl s_client -connect host-01.example.net:9100 -tls1_3 \
  -CAfile /etc/prometheus/ssl/monitoring-ca.crt -verify_hostname host-01.example.net </dev/nullCode language: PHP (php)

12.4 From any other host: the firewall

curl -sS --connect-timeout 5 -k https://host-01.example.net:9100/metrics
# Expect: connection timed outCode language: PHP (php)

On the node, the drop counter should go up:

nft list chain inet node_exporter inputCode language: PHP (php)

12.5 In Prometheus: the target is healthy

curl -s 'http://localhost:9090/api/v1/targets?state=active' \
  | jq '.data.activeTargets[] | select(.labels.job=="node") | {instance: .labels.instance, health, lastError}'Code language: JavaScript (javascript)

Every target should show "health": "up" and an empty lastError. In the web UI, up{job="node"} should return 1 for each host.

Common lastError messages:

ErrorCause
x509: certificate is valid for X, not YTarget name doesn’t match the certificate SAN
x509: certificate signed by unknown authorityWrong ca_file, or missing intermediate in the node’s certificate file
server returned HTTP status 401 UnauthorizedPassword mismatch, a hash in password_file, or a broken hash in the web config
context deadline exceededFirewall, wrong listen address, or scrape_timeout too short
server returned HTTP status 503 / 429--web.max-requests too low

13. Day-2 operations

Certificate renewal. The exporter reads the certificate and the web config file on every new connection, so renewed certificates and changed hashes take effect without a restart. Deploy the new files with the same ownership and modes, and you’re done. Alert on expiry with the blackbox exporter, or with a textfile metric from your renewal job.

Rotating the password without a gap. basic_auth_users accepts several users:

  1. Add a second user (for example prometheus2) with the new hash on every node.
  2. Switch username and password_file in prometheus.yml, run promtool check config, then reload.
  3. Remove the old user from the nodes.

Upgrades. apt upgrade updates the exporter. Your /etc/default file and the systemd drop-in survive upgrades. After a major version bump, run prometheus-node-exporter --help again to check for renamed or removed flags. That is exactly what broke --web.config.

Logs. Use journalctl -u prometheus-node-exporter. For more detail while debugging, add --log.level=debug to ARGS temporarily.

Adding a host. Repeat sections 2โ€“8 on the new node. That means firewall, install, certificate, web config (same hash), ARGS and the drop-in. Then add the target to the scrape job. This is the part to automate with Ansible, Salt or Puppet once you have more than a handful of machines.


14. Hardening checklist

  • โ˜ Port 9100 is reachable only from the Prometheus server(s), enforced by nftables before the package was installed
  • โ˜ Exporter is bound to the internal IP (--web.listen-address), not :9100
  • โ˜ TLS is enabled with min_version: TLS13, and the certificate SAN matches the scrape target
  • โ˜ Private key is 0640 root:prometheus in a 0750 directory
  • โ˜ Web config is passed with --web.config.file, is 0640 root:prometheus, and passes promtool check web-config
  • โ˜ Basic auth uses a long random password: only the bcrypt hash on the node, only the plaintext in a password_file (0640 root:prometheus) on the server
  • โ˜ prometheus.yml contains no secrets and passes promtool check config
  • โ˜ The scrape job trusts a dedicated CA (ca_file), not every public CA
  • โ˜ The deprecated ntp collector is not enabled. Time sync is monitored through timex
  • โ˜ The systemd collector is limited with unit-include
  • โ˜ The systemd sandbox drop-in is in place, systemd-analyze security has been reviewed, and the journal shows no collector errors
  • โ˜ Textfile metrics are written atomically (temp file plus mv)
  • โ˜ The alert rules pass promtool check rules and Prometheus sends alerts to Alertmanager
  • โ˜ All tests in section 12 pass, including negative tests (401, plain HTTP, TLS 1.2, foreign host)
  • โ˜ Optional: mutual TLS (Appendix B)

Appendix A: a minimal private CA

This is a small, offline-friendly CA for monitoring certificates. Run it on the Prometheus server, or better on an admin workstation, and keep ca.key somewhere safe.

mkdir -p ~/monitoring-ca && cd ~/monitoring-ca
umask 077

# 1. CA key and self-signed CA certificate (10 years, ECDSA P-256)
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -days 3650 -subj "/CN=Monitoring CA" \
  -keyout ca.key -out monitoring-ca.crt

# 2. Key and CSR for a node
H=host-01.example.net
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -subj "/CN=$H" -keyout "$H.key" -out "$H.csr"

# 3. Sign it: server certificate with SAN, about 13 months
openssl x509 -req -in "$H.csr" -CA monitoring-ca.crt -CAkey ca.key -CAcreateserial \
  -days 397 -out "$H.crt" \
  -extfile <(printf 'subjectAltName=DNS:%s\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=serverAuth\n' "$H")

openssl verify -CAfile monitoring-ca.crt "$H.crt"Code language: PHP (php)

Copy $H.crt and $H.key to the node (section 4), and monitoring-ca.crt to the Prometheus server (section 10.1). Delete the .csr files afterwards.


Appendix B: mutual TLS (optional)

With mTLS, the node only accepts connections from clients that present a certificate signed by your CA. That’s a strong second factor on top of basic auth, and it doesn’t depend on network position.

Issue a client certificate for Prometheus, using the CA from Appendix A:

C=prom-01.example.net
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
  -subj "/CN=$C" -keyout "$C-client.key" -out "$C-client.csr"
openssl x509 -req -in "$C-client.csr" -CA monitoring-ca.crt -CAkey ca.key -CAcreateserial \
  -days 397 -out "$C-client.crt" \
  -extfile <(printf 'basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=clientAuth\n')Code language: JavaScript (javascript)

On each node, copy monitoring-ca.crt to /etc/prometheus/ssl/monitoring-ca.crt (mode 0644). Then extend tls_server_config in /etc/prometheus/node-exporter-web.yml:

tls_server_config:
  cert_file: /etc/prometheus/ssl/node-exporter.crt
  key_file: /etc/prometheus/ssl/node-exporter.key
  min_version: TLS13
  client_auth_type: RequireAndVerifyClientCert
  client_ca_file: /etc/prometheus/ssl/monitoring-ca.crtCode language: JavaScript (javascript)

On the Prometheus server, install the client pair as /etc/prometheus/ssl/prom-01-client.crt (0644) and /etc/prometheus/ssl/prom-01-client.key (0640 root:prometheus), then add it to the job’s tls_config:

    tls_config:
      ca_file: /etc/prometheus/ssl/monitoring-ca.crt
      cert_file: /etc/prometheus/ssl/prom-01-client.crt
      key_file: /etc/prometheus/ssl/prom-01-client.key
      min_version: TLS13Code language: JavaScript (javascript)

Validate with promtool check config, then reload. Without the client certificate, curl now fails during the TLS handshake. With --cert and --key it works again.