Production-hardening pass: half-open detection, cascade throttle, OBSERVE refresh, lamp/door coupling
Four failure modes observed in-house since the polling-first refactor
(709fdf4):
1. Half-open DTLS sessions where the socket stays writable but the peer
has gone silent. Ping sends succeed against a wedged peer because
RT-OCF doesn't reliably emit a RST; only successful polls prove the
session is live.
- PollScheduler exposes last_success_ts (bumped on every 2.05).
- KeepaliveTask takes liveness_fn(); ticks fail if no 2.05 in the
last 60s, even when the ping send succeeded.
- Bridge force-closes the session after 120s unreachable so
run_forever() breaks out of sess.join() and reconnects.
2. RT-OCF cascade under load. One wedged path can eat 8s of timeout,
the next tier tick fires immediately and stacks another attempt,
and the device wedges harder.
- PollTier.timeout_s per-tier override (hot=2s, warm=4s, sweep=15s).
- On TimeoutError, the href goes into a 5-60s cooldown via the
existing _defer_until mechanism.
- take_window_stats() now reports successful-poll RTT separately
from a timeout count, exposed as the "Poll Timeouts (window)"
diagnostic entity in HA.
- Active-window throttle: if the previous health window saw >=3
timeouts and is_active=True, drop back to idle cadence -- stops
stacking polls on a stalled responder.
3. OBSERVE table aging across cloud-auth blips. The device stays
DTLS-reachable but the on-device stack clears its observer table
during the blip, so push delivery stays dead even after upstream
recovers.
- New ObserveRefreshTask per bridge; every 6h derregs all current
observer tokens and re-subscribes on the existing session.
4. Oven lamp/door coupling. Oven hardware auto-drives the lamp from
door state but /mode/vs/0 is warm-tier (30s) so HA showed stale lamp
during a cook.
- Track door + lamp value-change timestamps in descriptor_state.
When the door transition is newer, derive lamp from door_open.
When an HA optimistic write is newer, the cache value wins.
Also: ANSI-coloured WARNING/ERROR lines (NO_COLOR=1 opt-out), jittered
reconnect backoff so dryer + oven don't sync up after a router blip.
In-house verification: running on dryer + oven since 2026-06-03.
This commit is contained in:
1 parent
0374f8cf68
commit
36acf3a209
9 files changed
+438
-62
No files matched your search
@@ -1,8 +1,19 @@
|
||||
"""DTLS-layer liveness via CoAP empty-CON ping.
|
||||
"""DTLS-layer liveness via CoAP empty-CON ping + poll-success watchdog.
|
||||
|
||||
Pings the appliance every interval_s. After fail_threshold consecutive
|
||||
failures, fires on_unreachable. First success after a fail streak fires
|
||||
on_reachable. Bridge wires these to MQTT availability.
|
||||
Each interval_s the task:
|
||||
1. Sends a CoAP ping. This is fire-and-forget — Samsung's RT-OCF
|
||||
doesn't reliably reply with an RST, so the send itself is the
|
||||
keepalive (it tickles Samsung's observer state). The send only
|
||||
fails if the underlying socket is gone, in which case the failure
|
||||
counts toward fail_threshold.
|
||||
2. Calls liveness_fn() if provided. This is the real half-open
|
||||
detection: PollScheduler exposes last_success_ts, and the bridge
|
||||
wraps it as "did we get a 2.05 in the last 60s". If not, count
|
||||
the tick as a failure even though the ping send succeeded.
|
||||
|
||||
After fail_threshold consecutive failures, fires on_unreachable. First
|
||||
success after a fail streak fires on_reachable. Bridge wires these to
|
||||
MQTT availability.
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
@@ -20,13 +31,15 @@ class KeepaliveTask:
|
||||
fail_threshold: int = 3,
|
||||
on_reachable: Optional[Callable[[], None]] = None,
|
||||
on_unreachable: Optional[Callable[[], None]] = None,
|
||||
logger=None):
|
||||
logger=None,
|
||||
liveness_fn: Optional[Callable[[], bool]] = None):
|
||||
self.session = session
|
||||
self.interval_s = interval_s
|
||||
self.fail_threshold = fail_threshold
|
||||
self.on_reachable = on_reachable
|
||||
self.on_unreachable = on_unreachable
|
||||
self.log = logger
|
||||
self.liveness_fn = liveness_fn
|
||||
|
||||
self._fail_streak = 0
|
||||
self._reachable = True
|
||||
@@ -56,6 +69,21 @@ class KeepaliveTask:
|
||||
ok = True
|
||||
except Exception as e:
|
||||
if self.log: self.log.warning("ping: %s", e)
|
||||
# Real half-open detection: ping sends can succeed against a
|
||||
# silently-wedged peer, but polls won't. If the scheduler
|
||||
# hasn't recorded a 2.05 inside the liveness window, treat
|
||||
# this tick as a failure even though the ping itself went out.
|
||||
if ok and self.liveness_fn is not None:
|
||||
try:
|
||||
alive = bool(self.liveness_fn())
|
||||
except Exception as e:
|
||||
if self.log: self.log.warning("liveness_fn: %s", e)
|
||||
alive = True
|
||||
if not alive:
|
||||
if self.log:
|
||||
self.log.warning("liveness: no successful poll "
|
||||
"in the liveness window")
|
||||
ok = False
|
||||
self._ping_count += 1
|
||||
if ok:
|
||||
if not self._reachable:
|
||||
|
||||
Reference in new issue
Block a user