**** BEGIN LOGGING AT Mon Jul 29 03:00:13 2019 Jul 29 05:30:53 ksft: ehhhhhhhhhh Jul 29 05:32:03 ksft: sounds like filesystem corruption? maybe sudo apt-get --reinstall install make Jul 29 06:08:45 ah, that did it! Jul 29 06:17:41 doesn't sound good that it's happening in the first place though, who knows what else might be corrupted Jul 29 06:24:33 nothing I've noticed so far... Jul 29 06:25:53 sure, but there's probably still plenty of files you've never (directly or indirectly) accessed Jul 29 06:26:54 and there's still the question of how this happened in the first place. you flashed eMMC from sd card right? did you verify the sd card contents after writing it? Jul 29 06:29:19 did you make sure that the OS was done writing to the SD card before ejecting it or cutting power? (I normally use the "sync" command myself in such cases, though doing a clean poweroff *should* also ensure it) Jul 29 06:35:39 I did not verify the SD card's contents Jul 29 06:36:34 it shut off after it had flashed the eMMC Jul 29 06:36:38 a common mode of failure for sd cards (e.g. if they're worn out) is to exhibit a handful of random bit flips in one or more pages (which are typ 1KB) Jul 29 06:36:58 the flash itself just copies files, it is unlikely to detect corruption Jul 29 06:37:36 I had never used the SD card before Jul 29 06:37:41 hmm Jul 29 06:38:18 then I'm back to: did you make sure that the OS was done writing to the SD card before ejecting it or cutting power? Jul 29 06:38:42 (note I'm here talking about the process of writing the sd card, not the process of flashing to eMMC) Jul 29 06:39:55 after "cat" or "dd" has finished, the OS is typically not yet done writing data to SD card Jul 29 06:40:26 I believe I ran sync afterwards, but I can't be sure Jul 29 06:40:30 ok Jul 29 06:41:33 after "sync" I generally also wait a second or so before ejecting an sd card, to make sure internal processes in the card have settled down. maybe it's just superstition to do so, but I'm pretty sure I've had problems with cards I ejected immediately after "sync" completed Jul 29 06:46:10 I do the same. Never trust "managed" nand to do what it's told. Jul 29 06:46:56 I trust eMMC a bit more than SD card, since eMMC is at least designed to be used for a root filesystem Jul 29 06:49:14 eMMC also has commands for the OS to inform it that loss of power is imminent. dunno if SD has those Jul 29 06:49:25 (nor do I know when linux uses those commands) Jul 29 07:27:41 what can be done at the hardware level about that? Jul 29 07:39:34 wait synchronously for all writes to complete, i.e. murder performance to death Jul 29 07:40:16 (which probably also greatly increases write-amplification, hence will dimish the card lifetime) Jul 29 07:40:29 basically it's all optimized for performance and storage capacity, not reliability Jul 29 07:42:06 for our production beaglebones (i.e. the ones we embed into products) we reconfigure the eMMC into SLC mode with reliable writes enabled. SLC mode means you sacrifice half the storage capacity in exchange for greatly increased reliability and performance, and about a tenfold increase in lifetime Jul 29 07:42:31 reliable writes increases reliability, especially in the face of unexpected power failures Jul 29 07:42:55 presumably at the expense of performance and probably lifetime Jul 29 07:43:38 (enabling both overall still seems to result in a substantial performance increase. the other parameters are harder to estimate. basically it's just our best guess that this is the best configuration for us) **** BEGIN LOGGING AT Mon Jul 29 21:59:51 2019 **** ENDING LOGGING AT Tue Jul 30 03:01:12 2019