SIP Watchdog

SIP Watchdog is an external, end-to-end canary for inbound SIP paths.

It exists because a PBX may report a provider as registered and reachable while inbound calls are no longer delivered. A useful health check therefore has to originate outside the PBX and traverse a real telephone provider before reaching the monitored DID.

The reference deployment runs on relay.lodlp.net, calls monitored DIDs through an external SIP trunk and validates observable SIP call stages.

What it checks

For one target at a time, the watchdog launches baresip and can require any combination of these stages:

• connected: the upstream SIP provider accepted the request / began call setup;
• ringing: a 180 or 183 progress response was observed;
• answered: the call reached 200 OK / Call established;
• audio: the decoded RTP stream contains the ordered spectral canary fingerprint.

The required stages are configured per target. A target may also define success_any_of: all required_stages must be observed, plus at least one stage from success_any_of. The production policy uses connected, answered and audio: OVH does not reliably expose the PBX's 180/183, and a bare 200 OK is not accepted as proof because a provider may answer locally to play an announcement.

In normal operation, one target is checked every hour in round-robin order. If a probe fails, the same target is retried every 15 minutes until recovery.

Notifications

The private NTFY topic sip-watchdog carries the four machine messages FAIL, PROBE, OK, and KO. The operator topic network receives only the final verdict after an automatic recovery attempt: repaired (OK) or intervention required (KO). Intermediate recovery steps are intentionally silent to humans.

Architecture

systemd timer (15 min scheduler)
        |
        v
sip_watchdog.py
        |
        +-- state says "not due" ----------> exit 0
        |
        +-- select target (round-robin)
        |
        v
     baresip
        |
        v
external SIP provider -> PSTN/SIP interconnect -> monitored provider -> MikoPBX
        |
        +-- success: next target due in 60 min
        |
        +-- confirmed failure: FAIL -> NTFY machine bus

The systemd timer intentionally wakes every 15 minutes. The Python state machine decides whether a normal hourly probe is due or an incident retry is due.

Immediate probe after MikoPBX startup or PJSIP reload

Terry also runs scripts/mikopbx-event-trigger.py once per minute from the unprivileged asami crontab. It reads the host-mounted Asterisk verbose log and recognizes two events:

• Asterisk Ready. — Asterisk startup/restart;
• reload of res_pjsip_outbound_registration.so — the global PJSIP/provider reload observed during the October 2026 incident recovery.

A new event is debounced for 120 seconds, allowed 30 seconds for provider registrations to settle, then Terry publishes PROBE on the private NTFY control topic. Relay's listener performs the external canary. No SSH is used for this trigger path.

run-probe.sh serializes timer and event-driven probes with flock on /var/lib/sip-watchdog/probe.lock. A successful event-driven probe counts as a normal health check: it advances the round-robin target and schedules the next nominal probe one hour later.

The Terry watcher keeps its event fingerprint under ~/.local/state/sip-watchdog-mikopbx-trigger.json. Installation seeds the current event first, preventing an old log entry from causing a spurious initial call.

Home Assistant / MQTT metrics

The LAN MQTT broker is intentionally not exposed to Relay. Terry therefore pulls a sanitized metrics snapshot from Relay once per minute through the same restricted SSH key, using the allowed metrics command, then publishes it locally to Mosquitto. The forced-command dispatcher on Relay accepts only probe and metrics; arbitrary SSH commands remain impossible.

Runtime state is retained on sip-watchdog/state. Home Assistant MQTT Discovery configurations are retained under homeassistant/.../sip-watchdog.../config. The publisher refreshes the state once per minute and discovery entities use expire_after: 180, so loss of Terry→Relay collection makes the entities unavailable instead of leaving a stale green state indefinitely.

The global HA device exposes inbound-path health, incident state, current audio score, probe duration, last/next probe, last target and consecutive failures. Each monitored DID also gets diagnostic entities for path health, last spectral score, last probe and consecutive failures. Phone numbers and credentials are never present in MQTT payloads.

Relay records per-target operational history (last_success_at, last_failure_at, stage, SIP code, duration, spectral score and consecutive failures) in /var/lib/sip-watchdog/state.json. --metrics-json exports only the sanitized projection consumed by Terry.

Requirements

Reference platform: Debian 13.

apt install baresip-core python3-yaml python3-numpy

The watchdog supports both fixed-IP SIP authentication and digest authentication. For digest authentication, the SIP password is stored outside Git in a dedicated mode-0600 file and referenced by sip.password_file; it is copied only into Baresip's ephemeral mode-0600 account file for the duration of a probe.

NTFY uses a dedicated Bearer token stored outside Git in /etc/asami/sip-watchdog.ntfy-token. The token file must be mode 0600.

Configuration

Production configuration lives outside Git at /etc/asami/sip-watchdog.yaml. Start from config/sip-watchdog.example.yaml.

sip:
  domain: sip.example.net
  username: "example-account"
  auth_username: "example-account"
  password_file: /etc/asami/sip-watchdog.sip-password
  register: true
  transport: udp
  audio_codecs: [G722/16000/1, PCMA/8000/1, PCMU/8000/1]

probe:
  baresip: /usr/bin/baresip
  timeout_seconds: 35
  hold_after_answer_seconds: 1
  audio_capture_seconds: 5
  audio_signature: /home/asami/sip-watchdog/assets/sip-watchdog-canary-signature.json
  audio_match_threshold: 0.78

schedule:
  normal_minutes: 60
  retry_minutes: 15

ntfy:
  server: https://ntfy.example.net
  topic: network
  token_file: /etc/asami/sip-watchdog.ntfy-token
  timeout_seconds: 8

targets:
  - name: line-a
    number: "0033123456789"
    required_stages: [connected, answered, audio]

For audio validation, Baresip writes the decoded speaker stream to an ephemeral WAV. The matcher does not compare PCM samples: it recognizes the five ordered spectral sections declared in assets/sip-watchdog-canary-signature.json, with tolerance for G.711 quantization, gain changes, leading silence and RTP jitter. The PBX source WAV and its deterministic generator live under assets/.

State

Runtime state is stored in /var/lib/sip-watchdog/state.json. It contains the next due time, round-robin index, incident state, and the last result. It contains no credentials.

On failure, the target index is held in place so the same DID is retried 15 minutes later. On recovery, round-robin scheduling resumes with the next DID.

Installation

sudo ./scripts/install.sh
python3 ./src/sip_watchdog.py --config /etc/asami/sip-watchdog.yaml --check-config
sudo systemctl enable --now sip-watchdog.timer

Useful commands:

systemctl list-timers sip-watchdog.timer
journalctl -u sip-watchdog.service
sudo systemctl start sip-watchdog.service

Manual probe

python3 ./src/sip_watchdog.py \
  --config /etc/asami/sip-watchdog.yaml \
  --state /var/lib/sip-watchdog/state.json \
  --force

A failed probe exits with status 2. Configuration/internal execution failures exit with status 3.

Security

• No credentials are committed.
• SIP and NTFY secrets are read from external mode-0600 files and are never stored in YAML or Git.
• Fixed-IP SIP authentication remains supported without a SIP password file.
• Production phone numbers remain in /etc/asami/sip-watchdog.yaml, not Git.
• Temporary Baresip account files are mode 0600 and deleted after each probe.
• NTFY messages never contain SIP authorization material.
• The systemd service runs as the unprivileged asami user with filesystem and kernel hardening.
• Recovery and post-reload validation use the private NTFY control topic; the old Terry→Relay SSH trigger is not part of the functional watchdog path.

Tests

python3 -m unittest discover -s tests -v

GitLab CI runs the unit tests and GitLab Secret Detection.

License

GPL-3.0-or-later.

Confirmed failure and bounded recovery

A first canary failure does not open an incident immediately. src/watchdog_runner.py repeats the same target after a 10 s confirmation delay. A transient first failure followed by a successful confirmation is discarded without NTFY or incident state. Two consecutive failures open the incident with failures=2; later incident retries remain single probes on the normal retry cadence.

The recovery helper is scripts/mikopbx-self-heal.py, invoked only when Terry's NTFY listener receives FAIL. Its dedicated localhost-only AMI account requires only read=call,reporting and write=reporting. It refuses recovery while any Asterisk channel is active, verifies the three expected OVH outbound registration objects, runs PJSIPRegister on each, waits for registrations to settle, then publishes PROBE. Relay answers OK or KO after the external canary. It records the attempt before mutation and permits at most one re-registration attempt for one uninterrupted incident.

The AMI password is never versioned. Terry reads it from ~/.config/sip-watchdog/ami-password with 0600-style permissions. There is no recovery polling cron.

AMI qualification on MikoPBX 2026.3.40 confirmed the actual outbound-registration object names end in -REG. A live PJSIPRegister against SIP-TRUNK-D21CC3B7-REG returned Success / Reregistration sent, briefly moved the object through Unregistered and an OVH 403 Forbidden, then returned to Registered; the external spectral canary immediately passed again at score ~0.894. The self-heal therefore waits for settlement and is additionally gated to the original green-but-dead failure pattern: all three registrations must already report Registered before any re-registration is attempted.

NTFY control bus (current recovery architecture)

The private NTFY topic sip-watchdog is the machine control bus. The complete application protocol is four plain messages: FAIL, PROBE, OK, KO.

Relay publishes FAIL only after two consecutive canary failures. Terry performs the bounded AMI recovery and publishes PROBE. Relay executes the external spectral canary and answers OK or KO.

There is no recovery polling cron and no functional Terry-to-Relay SSH in this workflow. Both peers keep one long-lived NTFY subscription. NTFY delivery IDs are used only internally to resume the HTTP stream after reconnects; they are not part of the watchdog protocol.

The operator topic network receives only the final verdict. OK produces one normal-priority recovered notification; KO produces one high-priority intervention-required notification. Intermediate recovery plumbing is intentionally silent to humans.

The Terry helper remains guarded by: no active Asterisk channels, all three OVH registration objects already Registered (the green-but-dead pattern), and one recovery attempt per uninterrupted incident.