A second review of the observe-mode phase split (previous commit) found
two real regressions it introduced, both in the same failure family it
was built to close:
- async_send_command's failed-retry branch closed the session, then
raised without downgrading observe mode -- the downgrade only ran on
the retry's success path. A retry that also fails still leaves the
session dead, so mode was left claiming "Push" on a session that no
longer exists, same as the bug this whole fix targets. Moved the
downgrade to run right after the close, unconditionally on how the
retry goes.
- _attempt_observe_mode's stale-session abandon (the identity-check
branch added in the previous commit) didn't flag a resubscribe. A
session swap discovered there means a fresh, never-tried session now
exists, but _last_observe_attempt_ts was already stamped for the
now-abandoned attempt -- so that new session sat unsubscribed for up
to _RECOVERY_RETRY_S (600s) instead of being retried on the next
cycle. Now sets _resubscribe_due, same as the two reconnect paths do.
Also closes two test-coverage gaps the same review surfaced by mutation
testing: no test asserted the lock actually holds during the subscribe
burst (only that it's released for the wait), and no test distinguished
the max() in _maybe_retry_observe_mode's throttle from using
_last_observe_attempt_ts alone -- both mutations left the full suite
green. Added one test for each, plus extended two existing tests for the
bug fixes above; all four confirmed via mutation testing (revert the
fix, watch the new/extended test fail; restore it, watch it pass).
One finding from the same review is intentionally left open: async_close
is the one self._session writer that doesn't take _session_lock, so a
close racing _attempt_observe_mode isn't covered by today's identity
check. This is pre-existing (the lock didn't cover any of
_attempt_observe_mode before this branch's earlier commits either), not
a regression from this branch's work, and is a shutdown/unload-path
question rather than the write-vs-observe-mode race this branch set out
to fix.
Marc Billow
a3cc918343