**** BEGIN LOGGING AT Wed Jun 25 02:59:58 2014 Jun 25 05:45:16 build #100 of pxa is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/pxa/builds/100 Jun 25 05:47:07 build #2 of brcm47xx.mips74k is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/brcm47xx.mips74k/builds/2 Jun 25 05:48:52 build #2 of brcm47xx.legacy is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/brcm47xx.legacy/builds/2 Jun 25 08:06:26 build #194 of mvebu is complete: Failure [failed compile_8] Build details are at http://buildbot.openwrt.org:8010/builders/mvebu/builds/194 Jun 25 08:07:40 build #650 of atheros is complete: Failure [failed shell_14] Build details are at http://buildbot.openwrt.org:8010/builders/atheros/builds/650 Jun 25 08:11:26 build #557 of kirkwood is complete: Failure [failed shell_14] Build details are at http://buildbot.openwrt.org:8010/builders/kirkwood/builds/557 Jun 25 08:12:43 build #101 of x86_64 is complete: Failure [failed compile_1] Build details are at http://buildbot.openwrt.org:8010/builders/x86_64/builds/101 Jun 25 08:34:09 nbd r41337 trunk/package/system/ubus/Makefile Jun 25 08:34:09 ubus: update to the latest version, fixes netifd related crashes on ubus calls during reload Jun 25 08:49:25 build #100 of adm8668 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/adm8668/builds/100 Jun 25 10:08:13 KanjiMonster: can you clarify this for me please?: "< KanjiMonster> karlp: quite likely not. As long as jffs2 is not ready a tmpfs overlay is Jun 25 10:08:17 used, and only if jffs2 is ready the changes are copied over to jffs2." Jun 25 10:08:48 so things written to tmpfs overlay before jffs is mounted are gone? or written, but later? Jun 25 10:55:48 karlp: no, they are copied to the jffs2 overlay when the jffs2 overlay is ready. but before that it temporarily uses the tmpfs overlay to allow scripts to run and modify files without having to wait for jffs2 first Jun 25 11:11:59 ok, good, thanks. Jun 25 12:03:49 build #504 of adm5120 is complete: Success [build successful] Build details are at http://buildbot.openwrt.org:8010/builders/adm5120/builds/504 Jun 25 13:13:35 packages that are built into an image, and provide init scripts, are automatically enabled by ..."something"..., is this something opkg --force-postinstall in the build process where it says "Installing xxxx to root...." ? Jun 25 13:14:47 no, it happens in package/Makefile Jun 25 13:17:11 I should probably just leave it alone, even if monit is going to be watching things, it can just ignore things that got started automatically by enabled init scripts causing the rc.d links Jun 25 13:18:26 hrmm, or just take out START variables... Jun 25 13:50:32 When I enabled perl package, it doesn't seem to build libperl.so file. So, is it possible to build the shared perl library? Jun 25 15:08:24 build #101 of mpc83xx is complete: Success [build successful] Build details are at http://buildbot.openwrt.org:8010/builders/mpc83xx/builds/101 Jun 25 17:40:41 build #652 of orion is complete: Failure [failed compile_1] Build details are at http://buildbot.openwrt.org:8010/builders/orion/builds/652 Jun 25 17:41:38 build #606 of rb532 is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/rb532/builds/606 Jun 25 17:41:50 build #575 of sibyte is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/sibyte/builds/575 Jun 25 17:51:30 build #606 of ppc44x is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/ppc44x/builds/606 Jun 25 17:57:57 ping nbd Jun 25 18:26:21 jow_laptop: submodules work cool! :) Jun 25 18:26:48 excuse me whilst I open a can of worms Jun 25 18:27:11 I've just realized about the NAND vs squashfs vs badblocks problems Jun 25 18:27:25 what about f2fs Jun 25 18:27:32 what about it? Jun 25 18:27:57 is there a credible patch to squashfs to do the NAND bad block handling, or do I need to move to ubi? Jun 25 18:28:23 there isn't Jun 25 18:28:29 unfortunetely :( Jun 25 18:28:34 i miss that a lot Jun 25 18:28:48 so given that openwrt uses squash nominally, what is recommended? Jun 25 18:29:11 Chocks: there are some UBIFS users Jun 25 18:29:17 and some initramfs solutions Jun 25 18:29:23 nothing is really perfect for me Jun 25 18:29:39 initramfs is a waste of space and RAM Jun 25 18:29:49 I'm using squashfs elsewere. I think in one case the flash driver has patches for handling bad blocks. In the other case, well, I'm not sure Jun 25 18:29:50 UBIFS means a lot of layers Jun 25 18:29:59 elsewhere = other openwrt platforms Jun 25 18:30:11 squashfs doesn't even use MTD directly Jun 25 18:30:13 it uses mtdblock Jun 25 18:30:18 yeah Jun 25 18:30:20 so you first have to rewrite it to mtd Jun 25 18:30:25 then add badblocks handling Jun 25 18:31:14 I have systems in the field I might need to convert to ubi. ugh Jun 25 18:32:02 this is an at91 system Jun 25 18:34:45 this is not win Jun 25 18:34:50 * Chocks sobs for a bit Jun 25 18:35:34 Chocks: some business? Jun 25 18:35:45 maybe a chance to find funds for NAND support in squashfs? ;) Jun 25 18:35:47 what do you mean? Jun 25 18:35:52 oh Jun 25 18:35:58 not much chance. Jun 25 18:36:03 too bad Jun 25 18:36:22 I read a lengthy discussion about the matter and some bits about how it might be done. but I can't do it Jun 25 18:36:35 I need a solution this week Jun 25 18:36:36 me too ;/ Jun 25 18:36:39 oh Jun 25 18:36:44 that's a problem Jun 25 18:36:57 and I understand that support in squashfs is nominally the "wrong" place Jun 25 18:36:58 but still Jun 25 18:38:16 build #586 of avr32 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/avr32/builds/586 Jun 25 18:38:17 yeah, so I'm using squashfs on ramips with NAND too. from memory, there was some patch done there to the driver Jun 25 18:38:33 hell Jun 25 18:39:54 will think. back in a bit Jun 25 18:40:56 build #646 of cobalt is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/cobalt/builds/646 Jun 25 18:43:04 build #513 of iop32x is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/iop32x/builds/513 Jun 25 18:53:03 Why there isn't any /usr/lib/libperl.so.* ? Jun 25 19:10:35 build #617 of uml is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/uml/builds/617 Jun 25 20:01:02 build #541 of au1000 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/au1000/builds/541 Jun 25 20:15:54 build #99 of omap is complete: Failure [failed shell_15] Build details are at http://buildbot.openwrt.org:8010/builders/omap/builds/99 Jun 25 20:29:47 Hauke: hello :) thx for the review Jun 25 20:32:42 so it seems that running squashfs on top of ubi is going to be the quickest solution Jun 25 20:37:30 the only one ready i guess Jun 25 20:54:46 what's the problem exactly? Jun 25 20:55:24 (I got a couple of devices recently that are failing to flash, and they're some of our oldest field devices) Jun 25 20:56:54 the exact problem is that squashfs doesn't handle NAND errors Jun 25 20:57:07 whether it should or not is another debate Jun 25 21:01:48 <__trine> Chocky you're back :0) have you been making another film ? Jun 25 21:07:50 what nw Jun 25 21:07:51 now Jun 25 21:07:54 what film? Jun 25 21:14:02 <__trine> https://www.youtube.com/watch?v=Y16TEl6Ek1M have you been making a sequel Jun 25 21:26:57 build #501 of mcs814x is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/mcs814x/builds/501 Jun 25 21:28:24 karlp: the main issue is that nand flash produces bitflips even when only reading from it. These bitflips accumulate over time, and sometimes they get stuck at 1 or 0 (causing the sector to become "bad"). ECC can correct only so many bits, so everntually the fs must relocate the data to a different sector before it becomes uncorrectable. Jun 25 21:29:48 Also having bad sectors even fresh from factory is normal, and the fs must be able to cope with it (skip bad ones). squash fs does none of the mentioned things. Jun 25 21:30:30 right, I get the general idea of nand sucking, just hadn't really thought of it with squashfs much, I guess I'd just blithely assumed that read only, mean, "no real problems" Jun 25 21:30:42 does the jffs2 overlay deal with it at all? Jun 25 21:31:03 more, Chocks, why is it so pressing to find a soluton to this _NOW_? Jun 25 21:33:40 karlp: jffs2 has nand support, so it can do all these. so theoretically you could do a jffs2 + jffs2 overlay. but squashfs + jffs2 overlay won't work, because jffs2 can't magically handle bad blocks in the squashfs partition Jun 25 21:34:47 if it reboots/ power cycles, does it stay corrupted? Jun 25 21:44:58 karlp: the flash? usually yes, only erasing the sector "resets" it. Jun 25 21:46:27 so, why don't more people see this? or do people tend to just update often enough, therefore doing an erase? Jun 25 21:47:26 * Chocks catches up Jun 25 21:48:15 it's pressing now, because as usual, it has to be done yesterday Jun 25 21:48:34 I think the ubi overlay is the way to go, still. I'll try that soon Jun 25 21:49:41 karlp: I guess most people use a proper filesystem for nand (jffs2, yaffs, ubi(fs)) Jun 25 21:50:01 but the default in openwrt is squashfs Jun 25 21:50:32 it never was for e.g. kirkwood, one of the biggest nand users Jun 25 21:50:45 or ar71xx/mikrotik Jun 25 21:50:53 so anyway, it's not clear to me why this isn't seen more often either Jun 25 21:51:24 also most nand devices weren't supported by openwrt until recently Jun 25 21:51:33 hm Jun 25 22:00:07 Chocks: karlp: also most routers with nand use SLC nand, which deteriorates rather slowly. If you are lucky and the factory bad blocks are only in the higher addresses, that it will likely flash and boot fine for a while, but eventually (a few weeks or months later) you will notice that suddenly squashfs produces file system errors Jun 25 22:00:25 right Jun 25 22:00:25 KanjiMonster: hrm, so my old atheros device is likely to have just 8meg of nor flash, not any nand at all? Jun 25 22:00:32 yes Jun 25 22:00:33 this all figures Jun 25 22:00:57 how do you find out what sort of memory you have on an unknown platform? Jun 25 22:01:14 look up the part number, or look at what the driver reports Jun 25 22:01:28 usually anything up to 32 MiB is quite likely NOR flash, while anything above is quite likely NAND (but there are exceptions) Jun 25 22:01:56 yeah, this is 256MB Jun 25 22:02:15 the original manu was using ubifs. Jun 25 22:02:24 but I wanted to copy what I had always done. Jun 25 22:02:47 the biggest NOR I have seen is 128 MiB, while the smallest NAND is 32 MiB Jun 25 22:02:52 on routers Jun 25 22:03:13 this is a router, but it's not one you'd find in a store Jun 25 22:03:19 high end stuff Jun 25 22:03:28 128 MiB NAND is standard for routers with NAND flash Jun 25 22:06:30 well, I think that clarifies things, which is half way there Jun 25 22:06:30 on NAND you also need to (well, you should) reserve some sectors for storing the bad block table, as well as keep a certain amount of sectors reserved for replacing bad ones/wear leveling. So using very small NAND flashes isn't really feasable. Jun 25 22:06:45 so I appreciate that Jun 25 22:08:24 AFAIK SSDs actually have 40% of their capacity reserved for exactly that, replacing sectors gone bad; so a 128 GiB SSD has actually ~200 GiB (raw) flash **** ENDING LOGGING AT Thu Jun 26 02:59:58 2014