12 lines
1.0 KiB
HTML
12 lines
1.0 KiB
HTML
<p>False alerts are not a minor annoyance. They are how on-call culture dies. A live monitor that cries wolf will be muted, and then a real disconnect goes unnoticed.</p>
|
|
<h2>Common sources</h2>
|
|
<p>Reconnect flaps, duplicated gift combos, timezone-shifted “new day” counters, and keyword lists that match ordinary slang. Log why an alert fired, not only that it fired.</p>
|
|
<h2>Hysteresis</h2>
|
|
<p>Require a condition to persist for N seconds before paging. Viewer drops of one second are noise. Disconnects that last 15 seconds are incidents.</p>
|
|
<h2>Dedup windows</h2>
|
|
<p>The same gifter sending the same gift three times in two seconds is usually a combo, not three incidents. Collapse it.</p>
|
|
<h2>Operator feedback</h2>
|
|
<p>Add a “not useful” control on toasts. Review the pile weekly. Settings that cannot be tuned will be abandoned.</p>
|
|
<h2>Never hide failures</h2>
|
|
<p>Suppressing flaps is good. Suppressing “no events for 90 seconds while the room should be live” is not. Distinguish noisy metrics from dead pipelines.</p>
|