Operations

This page covers day-to-day operations: exemptions, state management, recovery procedures, systemd setup, graph maintenance, and troubleshooting.

Exemptions

Whitelist (permanent)

Sources in the whitelist are never scored, never rate-limited, never blocked — regardless of their traffic pattern. Use for known monitoring systems, office NAT gateways, and trusted partners.

Default path: /opt/kernloom/etc/whitelist.txt

Format:

# Office NAT / trusted gateway
203.0.113.7
203.0.113.0/24
2001:db8::1
2001:db8::/64

IQ reloads the file automatically every --whitelist-reload (default 10s) when it changes.

Feedback (temporary exemptions)

Use feedback when you need a time-bound exemption without permanently whitelisting. This is the right tool for incident response.

Default path: /opt/kernloom/var/feedback.json

[
  {"target":"203.0.113.7","action":"forgive","ttl":"24h","notes":"partner NAT during migration"},
  {"target":"198.51.100.0/24","action":"whitelist","until":"2026-06-01T00:00:00Z"}
]

Actions:

• forgive — move source back to OBSERVE for the TTL, do not whitelist
• whitelist — treat as whitelisted for the TTL

Use until (RFC3339 timestamp) instead of ttl for stable expiry across process restarts.

IQ reloads feedback automatically every --feedback-reload (default 10s).

CIDR de-enforcement

When a CIDR feedback entry is loaded, IQ will scan Shield’s enforcement maps and remove any existing rate-limit or deny entries for IPs in that range. Controlled by:

--feedback-deenforce-cidr=true    # enabled by default
--feedback-cidr-every=30s         # scan interval
--feedback-cidr-max=5000          # max deletions per scan

Recovery: quick actions

Immediate source release — SIGUSR1

Send SIGUSR1 to a running IQ process to trigger an immediate re-evaluation of all active exemptions and release any sources covered by the current whitelist or feedback:

sudo kill -SIGUSR1 $(pidof kliq)

Use this after editing the whitelist or feedback file when you need the release to take effect faster than the next reload cycle.

Immediate source release — feedback file

Edit /opt/kernloom/var/feedback.json and add a forgive entry. IQ picks it up within --feedback-reload (default 10s):

[{"target":"203.0.113.7","action":"forgive","ttl":"2h","notes":"incident response"}]

Manual Shield reset after IQ restart

If IQ is restarted and its in-memory state is lost, Shield’s enforcement maps may still contain stale rate-limit or deny entries from the previous run. Clear them:

sudo klshield reset

This removes all per-IP rate-limit overrides and deny entries from the kernel maps. IQ will repopulate them as needed on the next ticks.

State persistence

state.json

IQ saves its tuned thresholds and bootstrap metadata to state.json (default /opt/kernloom/var/state.json).

State contains:

• tuned trigger values (trig_pps, trig_syn, trig_scan)
• autotune metadata and history
• bootstrap phase and timing
• integrity hash (SHA256)

Startup: IQ loads state if the integrity hash matches and the file age is within --max-state-age (default 14 days).

Writes are atomic: IQ writes to a .tmp file, then renames it, with a .bak kept as fallback.

graph.db

The Graph Learner stores its edge database in a local SQLite file (default /opt/kernloom/var/graph.db).

This file is independent of state.json — it persists across restarts and across profile changes. It contains the full edge history: every candidate, learned, frozen, approved, and denied edge.

Backup before freezing:

cp /opt/kernloom/var/graph.db /opt/kernloom/var/graph.db.bak

View graph status:

kliq graph export                          # print all edges
kliq graph export --state=frozen           # print only frozen edges
kliq graph export --format=json            # machine-readable output

Manual edge management:

kliq graph approve-ip <ip>    # approve an edge (add weight, never auto-removed)
kliq graph deny-ip <ip>       # deny an edge (blocked, never overwritten)
kliq graph freeze              # lock all learned edges to frozen

Observability

Shield counters

sudo klshield stats                         # global totals
sudo klshield top-src -n 20 -by pkts        # top sources by packet count
sudo klshield top-src -n 20 -by droprl      # top sources by rate-limit drops
sudo klshield top-src -n 20 -by drops       # top sources by all drops

Sampled events (drop reasons, scan hints)

sudo klshield set-sampling 1023    # ~1/1024 of events (default, low overhead)
sudo klshield set-sampling 3       # ~1/4 of events (more detail)
sudo klshield set-sampling 0       # disable events entirely
sudo klshield events               # stream events (Ctrl+C to stop)

systemd

Shield (oneshot attach)

/etc/systemd/system/kernloom-shield.service:

[Unit]
Description=Kernloom Shield (XDP attach)
After=network-online.target
Wants=network-online.target

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/opt/kernloom/bin/klshield attach-xdp \
  -iface eth0 \
  -obj /opt/kernloom/bpf/klshield.bpf.o
ExecStop=/opt/kernloom/bin/klshield detach-xdp

[Install]
WantedBy=multi-user.target

IQ controller

/etc/systemd/system/kernloom-iq.service:

[Unit]
Description=Kernloom IQ (controller)
After=kernloom-shield.service
Requires=kernloom-shield.service

[Service]
Type=simple
Restart=always
RestartSec=2
ExecStart=/opt/kernloom/bin/kliq \
  --profile public-web \
  --interval 1s \
  --top 100 \
  --bootstrap=true \
  --state-file /opt/kernloom/var/state.json \
  --whitelist /opt/kernloom/etc/whitelist.txt \
  --feedback-file /opt/kernloom/var/feedback.json \
  --dry-run=true
# Change --dry-run=true to --dry-run=false when ready to enforce.

[Install]
WantedBy=multi-user.target

Enable and start:

sudo systemctl daemon-reload
sudo systemctl enable --now kernloom-shield kernloom-iq
sudo systemctl status kernloom-shield kernloom-iq

Troubleshooting

Shield attach fails

ip link                          # verify interface name
mount | grep bpf                 # verify bpffs is mounted
sudo klshield attach-xdp -iface eth0 -obj ... -force   # -force replaces existing XDP

IQ cannot read pinned maps

ls -la /sys/fs/bpf/kernloom_src4_stats \
       /sys/fs/bpf/kernloom_rl_policy4 \
       /sys/fs/bpf/kernloom_deny4_hash \
       /sys/fs/bpf/kernloom_src6_stats \
       /sys/fs/bpf/kernloom_rl_policy6 \
       /sys/fs/bpf/kernloom_deny6_hash

If any are missing, Shield is not attached. Re-run klshield attach-xdp.

Too many false positives

1. Add a feedback entry for the affected source (immediate, time-bound)
2. Send SIGUSR1 to release it immediately
3. Increase --block-min-sev and --block-min-dur to raise the bar for blocking
4. Increase --up-need, --min-hold-soft, --min-hold-hard to slow escalation
5. Consider switching to a *-bootstrap profile temporarily

Graph Learner: unexpected blocks after freeze

Check which edge triggered the block:

kliq graph export --state=frozen    # review baseline
kliq graph export --violations      # show recent violations

If the blocked source is legitimate:

kliq graph approve-ip <ip>          # add to baseline

Then send SIGUSR1 or wait for the block TTL to expire.

Cheat sheet

# Attach and validate
sudo klshield attach-xdp -iface eth0 -obj /opt/kernloom/bpf/klshield.bpf.o
sudo klshield stats
sudo klshield top-src -n 20 -by pkts

# Dry-run → enforce
sudo kliq --profile public-web --interval 1s --top 50 --dry-run=true
sudo kliq --profile public-web --interval 1s --top 100 --dry-run=false

# Forgive a source immediately
# 1. Edit /opt/kernloom/var/feedback.json
# 2. sudo kill -SIGUSR1 $(pidof kliq)

# Reset Shield maps after IQ restart
sudo klshield reset

# Graph: export and freeze
kliq graph export
kliq graph freeze

# Check pins
ls -la /sys/fs/bpf/kernloom_*

See also