Four failure modes observed in-house since the polling-first refactor
(709fdf4):
1. Half-open DTLS sessions where the socket stays writable but the peer
has gone silent. Ping sends succeed against a wedged peer because
RT-OCF doesn't reliably emit a RST; only successful polls prove the
session is live.
- PollScheduler exposes last_success_ts (bumped on every 2.05).
- KeepaliveTask takes liveness_fn(); ticks fail if no 2.05 in the
last 60s, even when the ping send succeeded.
- Bridge force-closes the session after 120s unreachable so
run_forever() breaks out of sess.join() and reconnects.
2. RT-OCF cascade under load. One wedged path can eat 8s of timeout,
the next tier tick fires immediately and stacks another attempt,
and the device wedges harder.
- PollTier.timeout_s per-tier override (hot=2s, warm=4s, sweep=15s).
- On TimeoutError, the href goes into a 5-60s cooldown via the
existing _defer_until mechanism.
- take_window_stats() now reports successful-poll RTT separately
from a timeout count, exposed as the "Poll Timeouts (window)"
diagnostic entity in HA.
- Active-window throttle: if the previous health window saw >=3
timeouts and is_active=True, drop back to idle cadence -- stops
stacking polls on a stalled responder.
3. OBSERVE table aging across cloud-auth blips. The device stays
DTLS-reachable but the on-device stack clears its observer table
during the blip, so push delivery stays dead even after upstream
recovers.
- New ObserveRefreshTask per bridge; every 6h derregs all current
observer tokens and re-subscribes on the existing session.
4. Oven lamp/door coupling. Oven hardware auto-drives the lamp from
door state but /mode/vs/0 is warm-tier (30s) so HA showed stale lamp
during a cook.
- Track door + lamp value-change timestamps in descriptor_state.
When the door transition is newer, derive lamp from door_open.
When an HA optimistic write is newer, the cache value wins.
Also: ANSI-coloured WARNING/ERROR lines (NO_COLOR=1 opt-out), jittered
reconnect backoff so dryer + oven don't sync up after a router blip.
In-house verification: running on dryer + oven since 2026-06-03.
State freshness now comes from a tiered PollScheduler over the persistent
DTLS session; OBSERVE registrations are kept as an opportunistic
acceleration layer. Behaviour is identical online vs air-gapped except
for worst-case freshness latency.
Adds three modules:
- StateCache: single source of truth, source-tagged change events
- PollScheduler: hot/warm/cold + sweep tiers, write-defer past the
fetchback-revert window, per-window RTT/slow-poll tracking
- KeepaliveTask: CoAP empty-CON ping with consecutive-fail detection
driving MQTT availability
Bridge publishes per-appliance diagnostic entities (Push Active, Last
Update Source, Poll Max RTT, Slow Polls, Poll Errors, Stalest Resource
Age, Last OBSERVE Age) under HA's Diagnostic section. Tier cadences
are descriptor-declared, calibrated against measured per-firmware
ceilings (dryer ~14 req/s, oven ~8 req/s via probe_poll_rate_combined.py).
Drops HEARTBEAT_INTERVAL_S in favour of the descriptor-declared sweep
tier; PING_INTERVAL_S now consumed by KeepaliveTask inside the bridge
rather than driven from main.py.
README explains the push/poll split and what happens when the appliance
is blocked from internet.
Jack Nagy