Hi all, Had a Pi 3B feeder (running the ADSB image) lock up overnight. Feeder has been running fine for months. Symptoms: 1. Old timestamp reported on the dashboard https://www.adsbexchange.com/api/feeders/ 2. No aircraft on local dump1090 feed (map loads but error saying it can't contact instance) 3. Local Grafana wouldn't load 4. No SSH (but saw that TCP 3-way handshake completed in wireshark OK) 5. Ping worked 6. Disk space was fine (checked SD in a desktop) The system was in this state for 12 hours, didn't self-recover so manual intervention is needed. Saw some errors in /var/log/syslog (when mounted on desktop): Sep 2 06:31:19 HOSTNAME rc.local[516]: Error fetching dump1090 aircraft data: Request timed out to http://localhost/dump1090//data/aircraft.json Sep 2 06:31:42 HOSTNAME kernel: [21546018.241206] INFO: task kworker/2:2:30773 blocked for more than 120 seconds. Sep 2 06:31:43 HOSTNAME kernel: [21546018.241213] Tainted: G C 4.19.66-v7+ #1253 Sep 2 06:31:43 HOSTNAME kernel: [21546018.241216] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message. Sep 2 06:31:43 HOSTNAME kernel: [21546018.241221] kworker/2:2 D 0 30773 2 0x00000000 Sep 2 06:31:43 HOSTNAME kernel: [21546018.241241] Workqueue: events_freezable mmc_rescan Sep 2 06:31:43 HOSTNAME kernel: [21546018.241262] [<80830934>] (__schedule) from [<80830fa4>] (schedule+0x50/0xa8) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241272] [<80830fa4>] (schedule) from [<80686520>] (__mmc_claim_host+0x120/0x228) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241281] [<80686520>] (__mmc_claim_host) from [<80686660>] (mmc_get_card+0x38/0x3c) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241289] [<80686660>] (mmc_get_card) from [<8068f6cc>] (mmc_sd_detect+0x24/0x7c) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241297] [<8068f6cc>] (mmc_sd_detect) from [<80688c88>] (mmc_rescan+0x1cc/0x39c) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241308] [<80688c88>] (mmc_rescan) from [<8013bf0c>] (process_one_work+0x170/0x458) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241318] [<8013bf0c>] (process_one_work) from [<8013c250>] (worker_thread+0x5c/0x5a4) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241327] [<8013c250>] (worker_thread) from [<8014253c>] (kthread+0x138/0x168) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241337] [<8014253c>] (kthread) from [<801010ac>] (ret_from_fork+0x14/0x28) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241342] Exception stack(0xb0fb3fb0 to 0xb0fb3ff8) Sep 2 06:31:43 HOSTNAME kernel: [21546018.241347] 3fa0: 00000000 00000000 00000000 00000000 Sep 2 06:31:43 HOSTNAME kernel: [21546018.241353] 3fc0: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 Sep 2 06:31:43 HOSTNAME kernel: [21546018.241359] 3fe0: 00000000 00000000 00000000 00000000 00000013 00000000 Sep 2 06:31:56 HOSTNAME rc.local[516]: Error fetching dump1090 aircraft data: Request timed out to http://localhost/dump1090//data/aircraft.json Found https://github.com/raspberrypi/linux/issues/2810 which explains the problem and provides the fix. Fix: Add these 2 lines to your /etc/sysctl.conf (you can do this from a desktop machine and edit the file on the SD - just remember to change the path to point to the Pi SD and not your desktop machine!) vm.dirty_background_ratio=5 vm.dirty_ratio=10 The Pi booted fine (that's likely enough to get things working again).
