Watchdog and Tiered Self-Recovery Design for Embedded Devices (2026)

Key takeaway: a watchdog is only worth anything if the device comes back to a usable state after the reset. Many products have a watchdog and still end up in reboot loops or lose their configuration on every reset. What needs designing is three things: who decides the system is still alive, in what order recovery proceeds, and how to fall back to a safe state when recovery fails.

1. Classify the hang first

Type	Symptom	Who detects it
Application hang	Process exists but does no work; UI frozen, no reporting	App heartbeat / systemd watchdog
Process exit	Service crashes or is OOM-killed	Supervisor / systemd Restart
Kernel fault	Oops, panic, driver deadlock; whole system unresponsive	Hardware watchdog
Peripheral hang	Camera, cellular module or NIC stops responding; rest of system fine	Business-layer timeouts
Network loss	Device running but invisible to the platform	Platform offline alarm + local self-check
Power / hardware	Repeated resets, brown-outs	Reset-cause register + supply monitoring

Each type needs a different tool. Using a hardware watchdog to rescue an application hang is overkill and causes unnecessary full-device reboots; application supervision alone cannot rescue a kernel deadlock.

2. A tiered watchdog architecture

• Tier 1 — SoC internal WDT. Simplest: timeout resets the SoC. Its weakness is that it only knows whether it was fed, not whether the system is healthy. Wrong feeding logic is equivalent to no watchdog;
• Tier 2 — external watchdog IC. Independent of the SoC, fed over GPIO or I²C. It still pulls reset when the SoC is fully hung, when the internal WDT has failed, or when the supply misbehaves. Recommended for unattended equipment;
• Tier 3 — systemd watchdog. The service sends sd_notify(WATCHDOG=1); on timeout systemd restarts that service instead of the whole device. Finer granularity, no full reboot for one stuck app;
• Tier 4 — business heartbeat. Not "the process is alive" but "the business is running": samples acquired, data pushed, queue drained. This is closest to what the user perceives;
• How the tiers interact: business heartbeat fails → restart the service; repeated service failures → reset the device; still unable to boot → roll back to the backup partition or enter recovery mode.

3. Feeding the watchdog correctly

• Feed only when every health check passes: in the main loop check critical thread heartbeats, queue backlog, peripheral accessibility and that no network task has blocked too long. Feed once if all pass; stop feeding on any failure and let the device reset by design;
• Never let threads feed independently: if every thread feeds, any surviving thread keeps the watchdog happy while another is dead. Each thread sets a health bit and one supervisor thread aggregates and feeds;
• Never feed from an interrupt or timer callback: interrupts running does not mean the application is running. This is the classic way to make a watchdog decorative;
• Allow for legitimate blocking: set the timeout at 2–3× the longest legitimate operation. Long tasks such as large writes, upgrades or log rotation must be chunked or explicitly extend the deadline;
• Treat boot separately: kernel boot, filesystem checks and app initialisation can exceed the timeout. Disable or extend the watchdog in the bootloader or init script, then hand over once the app is up;
• Log the feeding: record the last successful feed time and the health bits so that after a reset you can tell which condition failed first.

4. Reset is not recovery

This is the step most often skipped: the device does reboot, comes up in the same broken state, and enters a reset–hang–reset loop.

• Record the reset cause: most SoCs expose a reset-cause register distinguishing power-on, watchdog, software and external reset. Read and persist it at the earliest opportunity;
• Count failed boots: accumulate "boots that did not complete" in a writable partition or RTC-backed register. Clear it once the application is fully up and has reported success;
• Degrade after repeated failure: after several failed boots, switch to the backup rootfs, disable non-essential services and enter a minimal mode that at least stays remotely reachable;
• A/B rollback: a failed boot after an upgrade falls back to the previous version — the counterpart of the A/B mechanism in Firmware OTA and A/B Upgrades;
• Recovery mode: keep a minimal system with network and a maintenance interface for on-site or remote repair. Without it, a damaged boot chain means a factory return;
• Avoid reboot storms: back off the interval instead of retrying every second. Rapid cycling floods platform alarms and wears flash.

5. Application supervision

• systemd is the cheapest option: Restart=always, a few seconds of RestartSec, and StartLimitBurst to bound restarts within a window — exceeding it leaves the unit failed instead of retrying forever;
• Separate recoverable from fatal exit codes: configuration errors and authorisation failures will not fix themselves on restart. Exit and alarm instead of hiding the problem in a restart loop;
• Ordering: start the business only after network, storage and time sync are ready, so it does not come up, find dependencies missing and exit;
• Resource limits: cap memory with cgroups so the OOM killer does not take down a critical process — and treat OOM as a fault to be alarmed, not "it restarts anyway";
• Graceful exit: on SIGTERM stop streaming, commit data and release devices before exiting. SIGKILL leaves hardware resources held and files half-written.

6. Preserving the crash scene

• pstore / ramoops: write kernel oops logs into reserved memory so they survive a reboot. The most effective way to answer "why did it die", at the cost of one reserved-memory node in the device tree;
• Core dumps: store application cores on the writable partition for symbolised analysis, with count and size caps so they cannot fill it;
• Logs must survive: logs on tmpfs vanish on reset. Critical logs belong on persistent storage with size and count limits;
• Black box: a small ring buffer of key events — boot, reset cause, config change, upgrade, peripheral error — that can be exported as a whole when something goes wrong;
• Trustworthy time: without RTC or NTP, timestamps jump. At minimum record uptime seconds and boot count so events can still be ordered.

7. Remote observability and recovery

• Heartbeat reporting: periodically report online state, health, reset count, version and key metrics. Online status alone cannot reveal "running but business broken";
• Offline alarms: the platform declaring a device offline after a timeout is more reliable than the device reporting its own death;
• Remote reboot and diagnostics: controlled reboot, log retrieval and configuration read-back, with permission control and an audit trail — not a button anyone can press;
• Out-of-band path: when the primary network is down, can you get in over cellular, serial or a local maintenance port? Worth reserving for unattended sites;
• On-site fallback: remote access fails when the network does. A physical "long-press to recover" gesture beats asking a customer to open the enclosure and short a jumper.

8. Diagnosing spurious resets

• Supply droop: load steps (cellular transmit bursts, relays, disk spin-up) pull the rail down and trigger reset, which looks like a watchdog misfire. Scope the supply at the reset instant;
• Timeout too short: normal peak load exceeds the watchdog window, so the device reboots under heavy use;
• Boot too slow: filesystem checks or OTA verification push boot past the timeout, producing a device that never comes up;
• Flash stalls: eMMC garbage collection or bad-block retries block writes for seconds and starve the feeding thread;
• Scheduling starvation: a low-priority feeding thread is pre-empted indefinitely by heavy work. Raise priority or isolate the thread;
• EMC: a long, unfiltered reset trace picks up relay switching and resets the board. Add RC filtering and a TVS — see Embedded EMC Design & Certification.

9. Proving self-recovery works

• Fault injection: kill -STOP the main process, create infinite loops, unplug the camera, pull the network cable, fill the partition, simulate OOM — verify each recovery path;
• Measure recovery time: how long from fault to working business? Unattended deployments usually need minutes, not merely "it recovers eventually";
• Run cycles: loop between fault and recovery dozens of times to confirm no leaks accumulate and no reboot storm develops;
• Combine reset with power loss: cut power during a reset to exercise the failed-boot counter and rollback logic;
• Watch the reset-cause distribution: over long runs, a high watchdog share means a systemic problem is still unresolved.

10. Common pitfalls

• Feeding from a timer interrupt. The app is dead but the dog is fed — the watchdog does nothing;
• Only checking that the process exists. The platform sees "online", the user sees "no data";
• No failed-boot counter. A bad upgrade becomes an infinite reboot loop and a factory return;
• One-second restart intervals. Logs flood, platform alarms flood, flash wears faster;
• No crash scene. You know it rebooted, never why, and the same fault recurs.

11. FAQ

• Q1: Is the internal watchdog enough? For consumer devices with a human nearby, yes. For unattended, outdoor or vehicle installations where a site visit is expensive, add an external watchdog IC;
• Q2: What timeout should I use? Two to three times the longest legitimate block, with boot handled separately. Do not copy someone else's 10 or 30 seconds;
• Q3: Does a reboot fix everything? No. It clears transient state only. Configuration errors, hardware damage and network policy problems survive any number of reboots and must be distinguished and alarmed;
• Q4: Is a dual-system backup worth it? For frequently upgraded, hard-to-reach devices, yes. A/B partitions with failed-boot rollback cost flash space and upgrade complexity;
• Q5: Can you build this in? Yes. Hongyin Technology designs tiered watchdogs, failed-boot rollback, crash-log retention and remote diagnostics as part of firmware and unit delivery, with fault-injection verification. Send your deployment scenario and maintenance model for a proposal within 48 hours.

12. Related solutions & contact

Related solutions: IoT Edge Gateway、Security NVR Recording、Unattended Pump Station Terminal. Further reading: 7×24 Long-Run Stability Design、Firmware OTA and A/B Upgrades、Boot Time Optimisation.

Shenzhen Hongyin Technology Co., Ltd. (HONGYIN TECH) provides embedded project customization on RK3588/RK3576/RK3568 platforms — free requirement review, engineer-direct communication and low-volume ordering. Send your requirement list and receive selection advice and quotes within 48 hours.
phone +86 13728726926 | email sales@hoyin-tech.com | website www.hoyin-tech.com | address: Room 301, Building A, Fanrong Science Park, Sanwei Community, Hangcheng Street, Bao'an District, Shenzhen, China.