Prometheus is an open-source metrics monitoring and alerting system. It stores numeric time series in its own time-series database (TSDB), evaluates alert rules against them, and hands firing alerts to Alertmanager, which handles routing and notifications. Prometheus uses a pull model: the server scrapes an HTTP endpoint (/metrics) on every target at a fixed interval.
Node Exporter is the official exporter for Linux and Unix host metrics: CPU, memory, disks, filesystems, network, time sync, systemd units and more. (Windows hosts use the separate windows_exporter project. Node Exporter does not run on Windows.)
Out of the box, Node Exporter listens on every interface, port 9100, plain HTTP, with no authentication. Anyone who can reach that port can read your kernel version, mount points, interface names and load profile. This guide turns it into a properly locked-down endpoint:
- Firewall so that only the Prometheus server can reach port 9100
- TLS 1.3 with a certificate whose name matches the scrape target
- HTTP basic auth with a bcrypt hash on the node and the plaintext password in a
password_fileon the server - systemd sandboxing for the exporter process
- Useful extra collectors (systemd, textfile), with the deprecated
ntpcollector dropped - A scrape job that actually validates, plus a solid set of alert rules
- Validation with
promtool,curlandopensslat every step
Tested on: Debian 13 “trixie” for both the node and the Prometheus server, using the Debian packages
prometheus-node-exporterandprometheus. All commands run as root (or withsudo).
Table of contents
- Placeholders
- Firewall first
- Install Node Exporter
- TLS certificate and key
- Basic-auth credentials
- The web configuration file
- Exporter flags and collectors
- systemd hardening
- The textfile collector for custom metrics
- The Prometheus scrape job
- Alert rules
- Testing
- Day-2 operations
- Hardening checklist
- Appendix A: a minimal private CA
- Appendix B: mutual TLS (optional)
1. Placeholders
Replace these values everywhere in the guide:
| Placeholder | Meaning |
|---|---|
host-01.example.net | FQDN of the monitored node. It must match the TLS certificate’s SAN and the scrape target. |
192.0.2.21 | IP address on the node that Node Exporter will listen on (management or internal interface) |
prom-01.example.net | FQDN of the Prometheus server |
192.0.2.10 | IPv4 address of the Prometheus server (add a second server here if you run an HA pair) |
2001:db8::10 | IPv6 address of the Prometheus server (delete the IPv6 lines if you don’t use IPv6) |
monitoring-ca.crt | The CA certificate that signed the node certificates (see Appendix A) |
Two terms used below: the node is the machine being monitored, and the server is the Prometheus host.
2. Firewall first
On Debian the package starts the service as soon as it is installed, listening on :9100 over plain HTTP. Lock the port down before you install it.
This uses nftables, Debian’s default firewall framework. The rules live in their own table, so they work next to whatever ruleset you already have.
On the node, create /etc/nftables.d/node-exporter.nft:
install -d -m 0755 /etc/nftables.d
cat > /etc/nftables.d/node-exporter.nft <<'EOF'
# Only the Prometheus server(s) may reach Node Exporter on tcp/9100.
table inet node_exporter {
set prometheus_v4 {
type ipv4_addr
elements = { 192.0.2.10 }
}
set prometheus_v6 {
type ipv6_addr
elements = { 2001:db8::10 }
}
chain input {
type filter hook input priority filter - 10; policy accept;
tcp dport 9100 iif "lo" accept
tcp dport 9100 ip saddr @prometheus_v4 accept
tcp dport 9100 ip6 saddr @prometheus_v6 accept
tcp dport 9100 counter drop
}
}
EOFCode language: PHP (php)
Make sure /etc/nftables.conf includes the directory. Add this line at the end if it isn’t already there:
grep -q 'nftables.d' /etc/nftables.conf || \
echo 'include "/etc/nftables.d/*.nft"' >> /etc/nftables.confCode language: JavaScript (javascript)
Check the syntax, then load and enable it:
nft -c -f /etc/nftables.conf # dry run: no output means OK
systemctl enable --now nftables
systemctl reload nftables # if it was already running
nft list table inet node_exporterCode language: PHP (php)
Why a separate table with a final
drop? In nftables, adropin any base chain is final, and anacceptonly ends evaluation of that chain. This table therefore drops every unwanted 9100 packet no matter what your other tables do. If your main ruleset usespolicy drop, you also need anacceptfor tcp/9100 from the Prometheus server in that ruleset.
Using ufw instead?
ufw allow from 192.0.2.10 to any port 9100 proto tcpfollowed byufw deny 9100/tcpgives the same result. Don’t run ufw and a hand-written nftables ruleset on the same host.
3. Install Node Exporter
apt update
apt install prometheus-node-exporter
That’s all. Debian enables and starts the unit prometheus-node-exporter.service automatically, so you don’t need separate systemctl enable or systemctl start commands.
Check what you got:
prometheus-node-exporter --version
systemctl status prometheus-node-exporter --no-pager
The package also recommends prometheus-node-exporter-collectors, a set of systemd timers that write extra metrics (APT updates pending, reboot required, SMART, NVMe, and more) to the textfile collector directory. Keep it, because the alert rules in section 11 use its node_reboot_required metric.
Debian’s packaging uses these paths:
| What | Where |
|---|---|
| Binary | /usr/bin/prometheus-node-exporter |
| Arguments | ARGS="..." in /etc/default/prometheus-node-exporter |
| Service user | prometheus |
| Textfile directory | /var/lib/prometheus/node-exporter |
4. TLS certificate and key
The exporter needs a certificate whose Subject Alternative Name (SAN) contains the exact name Prometheus uses as the scrape target (here host-01.example.net). If the names don’t match, Prometheus refuses the connection, which is the correct behaviour.
There are two common sources:
- An internal CA (recommended for monitoring traffic). Appendix A shows a minimal OpenSSL CA. step-ca, Vault PKI or FreeIPA work just as well.
- A public ACME certificate (Let’s Encrypt) if the node has a public DNS name and you already run certbot. Copy or deploy it into the paths below with a deploy hook.
Put the certificate and key in one place, readable only by root and the prometheus group:
install -d -m 0750 -o root -g prometheus /etc/prometheus/ssl
install -m 0644 -o root -g prometheus host-01.crt \
/etc/prometheus/ssl/node-exporter.crt
install -m 0640 -o root -g prometheus host-01.key \
/etc/prometheus/ssl/node-exporter.key
Full chain: if an intermediate CA signed the certificate,
node-exporter.crtmust contain the leaf certificate followed by the intermediate(s).
Check that the certificate carries the right name and matches the key:
openssl x509 -in /etc/prometheus/ssl/node-exporter.crt -noout -subject -ext subjectAltName -enddate
# Both hashes must be identical:
openssl x509 -in /etc/prometheus/ssl/node-exporter.crt -noout -pubkey | sha256sum
openssl pkey -in /etc/prometheus/ssl/node-exporter.key -pubout | sha256sumCode language: PHP (php)
5. Basic-auth credentials
Node Exporter stores only a bcrypt hash. Prometheus needs the plaintext password, which it reads from a file, so the secret never appears in prometheus.yml.
5.1 On the Prometheus server: generate and store the password
install -d -m 0750 -o root -g prometheus /etc/prometheus/secrets
( umask 027
openssl rand -base64 32 | tr -d '\n' > /etc/prometheus/secrets/node-exporter.pass )
chgrp prometheus /etc/prometheus/secrets/node-exporter.pass
ls -l /etc/prometheus/secrets/node-exporter.pass # -rw-r----- root prometheusCode language: PHP (php)
5.2 Hash it
htpasswd from apache2-utils produces a bcrypt hash and is easier than a custom script. -i reads the password from stdin, so it never appears in the process list or your shell history:
apt install apache2-utils
htpasswd -niBC 12 "" < /etc/prometheus/secrets/node-exporter.pass | tr -d ':\n'; echoCode language: JavaScript (javascript)
The output looks like $2y$12$โฆ. Copy it for the next step. The hash isn’t secret the way a password is, but treat it as sensitive anyway: never publish a real one.
Prefer Python? Install
python3-bcrypt, then runpython3 -c 'import bcrypt,getpass; print(bcrypt.hashpw(getpass.getpass().encode(), bcrypt.gensalt(12)).decode())'. Note the.decode(). Without it Python printsb'$2b$โฆ', and pasting that into the config breaks authentication.
Why cost 12? The cost factor is a trade-off. Higher values resist offline cracking better, but every cache miss costs CPU on the node. 10โ12 is the sensible range for a long random machine password. The exporter caches successful verifications, so repeated scrapes stay cheap.
6. The web configuration file
On the node, create /etc/prometheus/node-exporter-web.yml:
# /etc/prometheus/node-exporter-web.yml
# Re-read on every request: changes apply without a restart.
tls_server_config:
cert_file: /etc/prometheus/ssl/node-exporter.crt
key_file: /etc/prometheus/ssl/node-exporter.key
# Prometheus and Node Exporter are both Go and speak TLS 1.3 natively.
# Use TLS12 only if some legacy scraper must connect.
min_version: TLS13
http_server_config:
headers:
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
X-Frame-Options: deny
basic_auth_users:
# username: bcrypt hash (quote it, because it contains '$')
prometheus: "$2y$12$REPLACE.WITH.THE.HASH.FROM.SECTION.5.2..................."Code language: PHP (php)
Lock it down, since it holds the hash:
chown root:prometheus /etc/prometheus/node-exporter-web.yml
chmod 0640 /etc/prometheus/node-exporter-web.yml
Validate it.
promtool check web-configcomes with theprometheuspackage, which you don’t want on every node. Copy the file to the Prometheus server and check it there:scp host-01.example.net:/etc/prometheus/node-exporter-web.yml /tmp/ promtool check web-config /tmp/node-exporter-web.yml && rm /tmp/node-exporter-web.ymlCode language: JavaScript (javascript)The check only validates syntax and the hash format. It doesn’t open the certificate files.
7. Exporter flags and collectors
Edit /etc/default/prometheus-node-exporter on the node. Replace the existing ARGS= line with:
ARGS="--web.config.file=/etc/prometheus/node-exporter-web.yml \
--web.listen-address=192.0.2.21:9100 \
--web.max-requests=10 \
--collector.textfile.directory=/var/lib/prometheus/node-exporter \
--collector.systemd \
--collector.systemd.unit-include=(ssh|cron|nftables|chrony|systemd-timesyncd|unattended-upgrades)[.]service"Code language: JavaScript (javascript)
The flags do the following:
| Flag | Why |
|---|---|
--web.config.file | Turns on TLS and basic auth. This is the current flag name. The old --web.config was renamed and later removed, so current versions refuse to start with it. |
--web.listen-address | Binds only to the internal or management IP instead of every interface. This adds defence in depth on top of the firewall. |
--web.max-requests=10 | Caps concurrent scrapes (the default is 40). Leave headroom for an HA pair plus a manual curl. A very low value such as 5 can cause failed scrapes when a scrape is slow. |
--collector.textfile.directory | Debian’s default already, but set explicitly so it’s obvious where custom .prom files go. |
--collector.systemd | Exports node_systemd_unit_state, so you can alert on failed units. It’s off by default. |
--collector.systemd.unit-include | Limits the systemd collector to units you care about. Exporting every unit inflates cardinality. The regex is anchored automatically. [.] is used instead of \. so the backslash survives systemd’s environment-file parsing. |
Optional collectors worth knowing about:
| Collector | Notes |
|---|---|
--collector.tcpstat | TCP connection states. Useful on busy servers, but it costs extra CPU on hosts with very many sockets. |
--collector.processes | Process and thread counts by state. |
--collector.interrupts | Per-CPU interrupt counts. High cardinality, so enable it only when you need it. |
--web.disable-exporter-metrics | Drops the exporter’s own go_* and promhttp_* metrics if you don’t want them. |
Do not enable
--collector.ntp. It is deprecated: it queries an NTP server over the network on every scrape and reports that server’s view, not the kernel’s. Thetimexcollector, enabled by default, reads the kernel clock discipline directly (node_timex_sync_status,node_timex_offset_seconds,node_timex_maxerror_seconds), and works with chrony, systemd-timesyncd or ntpd. The time-sync alerts in section 11 use it.
To see every available flag and which collectors are on by default:
prometheus-node-exporter --help 2>&1 | less
8. systemd hardening
Node Exporter only needs to read /proc, /sys and a few files, and to talk to D-Bus for the systemd collector. Add a drop-in that sandboxes it and waits for the network, since the exporter now binds to a specific IP:
systemctl edit prometheus-node-exporter
Paste:
[Unit]
# Bind to a specific IP: make sure it exists first.
Wants=network-online.target
After=network-online.target
[Service]
NoNewPrivileges=yes
CapabilityBoundingSet=
AmbientCapabilities=
ProtectSystem=strict
ProtectHome=read-only
PrivateTmp=yes
ProtectKernelTunables=yes
ProtectKernelModules=yes
ProtectKernelLogs=yes
ProtectControlGroups=yes
ProtectClock=yes
ProtectHostname=yes
RestrictAddressFamilies=AF_UNIX AF_INET AF_INET6 AF_NETLINK
RestrictNamespaces=yes
RestrictRealtime=yes
RestrictSUIDSGID=yes
LockPersonality=yes
MemoryDenyWriteExecute=yes
SystemCallArchitectures=native
SystemCallFilter=@system-service
SystemCallErrorNumber=EPERM
UMask=0077Code language: PHP (php)
Apply all of it:
systemctl daemon-reload
systemctl restart prometheus-node-exporter
systemctl status prometheus-node-exporter --no-pager
journalctl -u prometheus-node-exporter -b --no-pager | tail -n 30
systemd-analyze security prometheus-node-exporter
In the journal, look for TLS is enabled. and Listening on address=192.0.2.21:9100. Watch for collector errors such as collector failed or permission denied.
Deliberately left out:
ProtectHome=yesandProtectProc=invisible. They hide/homeand/proc/1/mounts, which breaks the filesystem collector.PrivateDevices=yesis usually fine too, but test it on your hardware first.
9. The textfile collector for custom metrics
The textfile collector exports any *.prom file in /var/lib/prometheus/node-exporter. Use it for things only a script can know, such as backup age, certificate expiry of local services, or RAID state.
The rule: write to a temporary file in the same directory, then mv it into place. The rename is atomic, so the exporter never reads a half-written file. The temporary name must not end in .prom.
Example: record a successful backup.
cat > /usr/local/sbin/node-metric-backup-ok <<'EOF'
#!/bin/sh
# Call at the end of a successful backup run.
set -eu
dir=/var/lib/prometheus/node-exporter
tmp=$(mktemp "$dir/.backup.XXXXXX")
cat > "$tmp" <<METRICS
# HELP backup_last_success_timestamp_seconds Unix time of the last successful backup.
# TYPE backup_last_success_timestamp_seconds gauge
backup_last_success_timestamp_seconds $(date +%s)
METRICS
chmod 0644 "$tmp"
mv "$tmp" "$dir/backup.prom"
EOF
chmod 0755 /usr/local/sbin/node-metric-backup-okCode language: PHP (php)
If a file has a syntax error, the exporter reports node_textfile_scrape_error 1. Section 11 alerts on that.
10. The Prometheus scrape job
Everything in this section happens on the Prometheus server.
10.1 Trust the CA
install -d -m 0755 /etc/prometheus/ssl
install -m 0644 monitoring-ca.crt /etc/prometheus/ssl/monitoring-ca.crt
If you use publicly trusted certificates, point
ca_fileat/etc/ssl/certs/ca-certificates.crtinstead. For internal monitoring, a dedicated CA is better: Prometheus then trusts only your CA for this job, not every public CA.
10.2 The job
Add this under scrape_configs: in /etc/prometheus/prometheus.yml. YAML is indentation-sensitive. Use spaces, never tabs, and keep every key of the job at the same level:
scrape_configs:
# ... your existing jobs ...
- job_name: node
scrape_interval: 15s
scrape_timeout: 10s
scheme: https
tls_config:
ca_file: /etc/prometheus/ssl/monitoring-ca.crt
min_version: TLS13
basic_auth:
username: prometheus
# The PLAINTEXT password, read from a file. Never the bcrypt hash.
password_file: /etc/prometheus/secrets/node-exporter.pass
static_configs:
- targets:
- host-01.example.net:9100
- host-02.example.net:9100
relabel_configs:
# instance="host-01.example.net" instead of "host-01.example.net:9100"
- source_labels: [__address__]
regex: '(.+):\d+'
target_label: instance
replacement: '$1'Code language: PHP (php)
Notes on this job:
- Targets are FQDNs that match each node’s certificate SAN. Short names like
host-01only work if the certificate also lists them. - The relabel regex is anchored by Prometheus, and the greedy
(.+)keeps everything up to the last colon. That means it also handles[2001:db8::21]:9100. Without it, dashboards show the port in every legend. - Many hosts? Replace
static_configswithfile_sd_configspointing at/etc/prometheus/targets/node/*.yml. Prometheus picks up changes to those files without a reload.
10.3 Load rules too
Add this at the top level of prometheus.yml if it isn’t there yet:
rule_files:
- /etc/prometheus/rules/*.ymlCode language: JavaScript (javascript)
10.4 Validate and reload
promtool check config /etc/prometheus/prometheus.yml
systemctl reload prometheus
journalctl -u prometheus -n 20 --no-pager # look for "Completed loading of configuration file"Code language: PHP (php)
promtool check config also checks that password_file and ca_file exist and loads every rule file. A failed check means don’t reload.
11. Alert rules
Create /etc/prometheus/rules/node.yml on the server. The thresholds are sane defaults. Tune the for: durations and percentages to your environment.
# /etc/prometheus/rules/node.yml
groups:
- name: node-exporter
rules:
# --- Availability -------------------------------------------------------
- alert: NodeExporterDown
expr: up{job="node"} == 0
for: 5m
labels:
severity: critical
annotations:
summary: "Node exporter on {{ $labels.instance }} is unreachable"
description: "Prometheus has failed to scrape {{ $labels.instance }} for 5 minutes (host down, firewall, TLS or auth problem)."
# --- Filesystems --------------------------------------------------------
- alert: NodeFilesystemAlmostFull
expr: |
(
node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|ramfs"}
/ node_filesystem_size_bytes{job="node",fstype!~"tmpfs|ramfs"} < 0.10
)
and node_filesystem_readonly{job="node",fstype!~"tmpfs|ramfs"} == 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} has less than 10% free"
description: "Only {{ $value | humanizePercentage }} space left on {{ $labels.device }} ({{ $labels.mountpoint }})."
- alert: NodeFilesystemFillingUp
expr: |
(
node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|ramfs"}
/ node_filesystem_size_bytes{job="node",fstype!~"tmpfs|ramfs"} < 0.40
)
and predict_linear(node_filesystem_avail_bytes{job="node",fstype!~"tmpfs|ramfs"}[6h], 24 * 3600) < 0
and node_filesystem_readonly{job="node",fstype!~"tmpfs|ramfs"} == 0
for: 1h
labels:
severity: warning
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} will be full within 24h"
description: "At the current rate, {{ $labels.mountpoint }} runs out of space within a day."
- alert: NodeFilesystemInodesLow
expr: |
(
node_filesystem_files_free{job="node",fstype!~"tmpfs|ramfs"}
/ node_filesystem_files{job="node",fstype!~"tmpfs|ramfs"} < 0.10
)
and node_filesystem_readonly{job="node",fstype!~"tmpfs|ramfs"} == 0
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is running out of inodes"
description: "Only {{ $value | humanizePercentage }} inodes left."
- alert: NodeFilesystemReadOnly
expr: node_filesystem_readonly{job="node",fstype=~"ext4|xfs|btrfs"} == 1
for: 5m
labels:
severity: critical
annotations:
summary: "{{ $labels.mountpoint }} on {{ $labels.instance }} is read-only"
description: "A normally writable filesystem is mounted read-only. This often means I/O errors or filesystem corruption. Check dmesg."
# --- Memory & CPU -------------------------------------------------------
- alert: NodeMemoryLow
expr: node_memory_MemAvailable_bytes{job="node"} / node_memory_MemTotal_bytes{job="node"} < 0.10
for: 15m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} is low on memory"
description: "Only {{ $value | humanizePercentage }} of RAM is available."
- alert: NodeOOMKill
expr: increase(node_vmstat_oom_kill{job="node"}[15m]) > 0
labels:
severity: warning
annotations:
summary: "OOM killer fired on {{ $labels.instance }}"
description: "The kernel killed at least one process for lack of memory in the last 15 minutes."
- alert: NodeHighCPU
expr: 1 - avg by (instance) (rate(node_cpu_seconds_total{job="node",mode="idle"}[5m])) > 0.90
for: 30m
labels:
severity: warning
annotations:
summary: "{{ $labels.instance }} CPU above 90% for 30 minutes"
description: "Average CPU utilisation is {{ $value | humanizePercentage }}."
# --- Time sync (timex collector) ---------------------------------------
- alert: NodeClockNotSynchronising
expr: min_over_time(node_timex_sync_status{job="node"}[5m]) == 0 and node_timex_maxerror_seconds{job="node"} >= 16
for: 10m
labels:
severity: warning
annotations:
summary: "Clock on {{ $labels.instance }} is not synchronising"
description: "The kernel reports the clock as unsynchronised. Check chrony or systemd-timesyncd."
- alert: NodeClockSkew
expr: abs(node_timex_offset_seconds{job="node"}) > 0.05
for: 10m
labels:
severity: warning
annotations:
summary: "Clock on {{ $labels.instance }} is off by more than 50 ms"
description: "Current offset: {{ $value | humanizeDuration }}."
# --- systemd & textfile -------------------------------------------------
- alert: NodeSystemdUnitFailed
expr: node_systemd_unit_state{job="node",state="failed"} == 1
for: 5m
labels:
severity: warning
annotations:
summary: "{{ $labels.name }} failed on {{ $labels.instance }}"
description: "systemd reports the unit {{ $labels.name }} in state 'failed'."
- alert: NodeTextfileScrapeError
expr: node_textfile_scrape_error{job="node"} == 1
for: 15m
labels:
severity: warning
annotations:
summary: "Textfile collector error on {{ $labels.instance }}"
description: "A .prom file in /var/lib/prometheus/node-exporter is unreadable or malformed."
- alert: NodeRebootRequired
# Provided by prometheus-node-exporter-collectors.
expr: node_reboot_required{job="node"} > 0
for: 24h
labels:
severity: info
annotations:
summary: "{{ $labels.instance }} needs a reboot"
description: "Updated packages (usually the kernel or libc) need a reboot to take effect."
- alert: BackupTooOld
# Only fires for hosts that publish the metric from section 9.
expr: time() - backup_last_success_timestamp_seconds{job="node"} > 26 * 3600
labels:
severity: warning
annotations:
summary: "No successful backup on {{ $labels.instance }} for over 26 hours"
description: "Last success was {{ $value | humanizeDuration }} ago."Code language: PHP (php)
Validate the rules, then reload:
promtool check rules /etc/prometheus/rules/node.yml
promtool check config /etc/prometheus/prometheus.yml
systemctl reload prometheus
Alerts only reach a human if Prometheus is connected to Alertmanager (the
alerting:block inprometheus.yml). Setting up Alertmanager is outside the scope of this guide. Until you have it, firing alerts show up under Alerts in the Prometheus web UI.
12. Testing
Don’t skip these checks. Each one proves a different layer works.
12.1 On the node: listening on the right address only
ss -tlnp 'sport = :9100'
# Expect: LISTEN ... 192.0.2.21:9100 ... prometheus-nodeCode language: PHP (php)
12.2 From the Prometheus server: TLS and authentication
CA=/etc/prometheus/ssl/monitoring-ca.crt
URL=https://host-01.example.net:9100/metrics
# No credentials -> 401
curl -sS -o /dev/null -w '%{http_code}\n' --cacert "$CA" "$URL"
# Wrong password -> 401
curl -sS -o /dev/null -w '%{http_code}\n' --cacert "$CA" -u prometheus:wrong "$URL"
# Correct credentials -> 200 and metrics (curl prompts for the password)
curl -sS --cacert "$CA" -u prometheus "$URL" | grep -E '^node_(uname_info|timex_sync_status|systemd_unit_state.*failed)' | head
# Plain HTTP must not work (expect HTTP 400: "client sent an HTTP request to an HTTPS server")
curl -sS -o /dev/null -w '%{http_code}\n' "http://host-01.example.net:9100/metrics"Code language: PHP (php)
12.3 TLS version and certificate
# TLS 1.2 must be refused (handshake failure / alert protocol version)
openssl s_client -connect host-01.example.net:9100 -tls1_2 </dev/null
# TLS 1.3 works and the chain verifies (look for "Verify return code: 0 (ok)")
openssl s_client -connect host-01.example.net:9100 -tls1_3 \
-CAfile /etc/prometheus/ssl/monitoring-ca.crt -verify_hostname host-01.example.net </dev/nullCode language: PHP (php)
12.4 From any other host: the firewall
curl -sS --connect-timeout 5 -k https://host-01.example.net:9100/metrics
# Expect: connection timed outCode language: PHP (php)
On the node, the drop counter should go up:
nft list chain inet node_exporter inputCode language: PHP (php)
12.5 In Prometheus: the target is healthy
curl -s 'http://localhost:9090/api/v1/targets?state=active' \
| jq '.data.activeTargets[] | select(.labels.job=="node") | {instance: .labels.instance, health, lastError}'Code language: JavaScript (javascript)
Every target should show "health": "up" and an empty lastError. In the web UI, up{job="node"} should return 1 for each host.
Common lastError messages:
| Error | Cause |
|---|---|
x509: certificate is valid for X, not Y | Target name doesn’t match the certificate SAN |
x509: certificate signed by unknown authority | Wrong ca_file, or missing intermediate in the node’s certificate file |
server returned HTTP status 401 Unauthorized | Password mismatch, a hash in password_file, or a broken hash in the web config |
context deadline exceeded | Firewall, wrong listen address, or scrape_timeout too short |
server returned HTTP status 503 / 429 | --web.max-requests too low |
13. Day-2 operations
Certificate renewal. The exporter reads the certificate and the web config file on every new connection, so renewed certificates and changed hashes take effect without a restart. Deploy the new files with the same ownership and modes, and you’re done. Alert on expiry with the blackbox exporter, or with a textfile metric from your renewal job.
Rotating the password without a gap. basic_auth_users accepts several users:
- Add a second user (for example
prometheus2) with the new hash on every node. - Switch
usernameandpassword_fileinprometheus.yml, runpromtool check config, then reload. - Remove the old user from the nodes.
Upgrades. apt upgrade updates the exporter. Your /etc/default file and the systemd drop-in survive upgrades. After a major version bump, run prometheus-node-exporter --help again to check for renamed or removed flags. That is exactly what broke --web.config.
Logs. Use journalctl -u prometheus-node-exporter. For more detail while debugging, add --log.level=debug to ARGS temporarily.
Adding a host. Repeat sections 2โ8 on the new node. That means firewall, install, certificate, web config (same hash), ARGS and the drop-in. Then add the target to the scrape job. This is the part to automate with Ansible, Salt or Puppet once you have more than a handful of machines.
14. Hardening checklist
- โ Port 9100 is reachable only from the Prometheus server(s), enforced by nftables before the package was installed
- โ Exporter is bound to the internal IP (
--web.listen-address), not:9100 - โ TLS is enabled with
min_version: TLS13, and the certificate SAN matches the scrape target - โ Private key is
0640 root:prometheusin a0750directory - โ Web config is passed with
--web.config.file, is0640 root:prometheus, and passespromtool check web-config - โ Basic auth uses a long random password: only the bcrypt hash on the node, only the plaintext in a
password_file(0640 root:prometheus) on the server - โ
prometheus.ymlcontains no secrets and passespromtool check config - โ The scrape job trusts a dedicated CA (
ca_file), not every public CA - โ The deprecated
ntpcollector is not enabled. Time sync is monitored throughtimex - โ The systemd collector is limited with
unit-include - โ The systemd sandbox drop-in is in place,
systemd-analyze securityhas been reviewed, and the journal shows no collector errors - โ Textfile metrics are written atomically (temp file plus
mv) - โ The alert rules pass
promtool check rulesand Prometheus sends alerts to Alertmanager - โ All tests in section 12 pass, including negative tests (401, plain HTTP, TLS 1.2, foreign host)
- โ Optional: mutual TLS (Appendix B)
Appendix A: a minimal private CA
This is a small, offline-friendly CA for monitoring certificates. Run it on the Prometheus server, or better on an admin workstation, and keep ca.key somewhere safe.
mkdir -p ~/monitoring-ca && cd ~/monitoring-ca
umask 077
# 1. CA key and self-signed CA certificate (10 years, ECDSA P-256)
openssl req -x509 -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-days 3650 -subj "/CN=Monitoring CA" \
-keyout ca.key -out monitoring-ca.crt
# 2. Key and CSR for a node
H=host-01.example.net
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-subj "/CN=$H" -keyout "$H.key" -out "$H.csr"
# 3. Sign it: server certificate with SAN, about 13 months
openssl x509 -req -in "$H.csr" -CA monitoring-ca.crt -CAkey ca.key -CAcreateserial \
-days 397 -out "$H.crt" \
-extfile <(printf 'subjectAltName=DNS:%s\nbasicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=serverAuth\n' "$H")
openssl verify -CAfile monitoring-ca.crt "$H.crt"Code language: PHP (php)
Copy $H.crt and $H.key to the node (section 4), and monitoring-ca.crt to the Prometheus server (section 10.1). Delete the .csr files afterwards.
Appendix B: mutual TLS (optional)
With mTLS, the node only accepts connections from clients that present a certificate signed by your CA. That’s a strong second factor on top of basic auth, and it doesn’t depend on network position.
Issue a client certificate for Prometheus, using the CA from Appendix A:
C=prom-01.example.net
openssl req -newkey ec -pkeyopt ec_paramgen_curve:P-256 -nodes \
-subj "/CN=$C" -keyout "$C-client.key" -out "$C-client.csr"
openssl x509 -req -in "$C-client.csr" -CA monitoring-ca.crt -CAkey ca.key -CAcreateserial \
-days 397 -out "$C-client.crt" \
-extfile <(printf 'basicConstraints=critical,CA:FALSE\nkeyUsage=critical,digitalSignature\nextendedKeyUsage=clientAuth\n')Code language: JavaScript (javascript)
On each node, copy monitoring-ca.crt to /etc/prometheus/ssl/monitoring-ca.crt (mode 0644). Then extend tls_server_config in /etc/prometheus/node-exporter-web.yml:
tls_server_config:
cert_file: /etc/prometheus/ssl/node-exporter.crt
key_file: /etc/prometheus/ssl/node-exporter.key
min_version: TLS13
client_auth_type: RequireAndVerifyClientCert
client_ca_file: /etc/prometheus/ssl/monitoring-ca.crtCode language: JavaScript (javascript)
On the Prometheus server, install the client pair as /etc/prometheus/ssl/prom-01-client.crt (0644) and /etc/prometheus/ssl/prom-01-client.key (0640 root:prometheus), then add it to the job’s tls_config:
tls_config:
ca_file: /etc/prometheus/ssl/monitoring-ca.crt
cert_file: /etc/prometheus/ssl/prom-01-client.crt
key_file: /etc/prometheus/ssl/prom-01-client.key
min_version: TLS13Code language: JavaScript (javascript)
Validate with promtool check config, then reload. Without the client certificate, curl now fails during the TLS handshake. With --cert and --key it works again.