**** BEGIN LOGGING AT Sat Jan 30 02:59:58 2016 Jan 30 05:59:52 build #144 of gemini is complete: Failure [failed shell_14] Build details are at http://buildbot.openwrt.org:8010/builders/gemini/builds/144 Jan 30 06:07:38 build #193 of ppc40x is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/ppc40x/builds/193 Jan 30 07:20:38 build #200 of x86.64 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/x86.64/builds/200 Jan 30 07:43:11 build #160 of lantiq.xrx200 is complete: Success [build successful] Build details are at http://buildbot.openwrt.org:8010/builders/lantiq.xrx200/builds/160 Jan 30 09:26:48 build #183 of brcm2708 is complete: Failure [failed compile_4] Build details are at http://buildbot.openwrt.org:8010/builders/brcm2708/builds/183 Jan 30 09:55:29 build #170 of x86.kvm_guest is complete: Failure [failed compile_4] Build details are at http://buildbot.openwrt.org:8010/builders/x86.kvm_guest/builds/170 Jan 30 09:55:41 build #171 of arm64 is complete: Failure [failed compile_4] Build details are at http://buildbot.openwrt.org:8010/builders/arm64/builds/171 Jan 30 10:10:29 build #185 of x86.xen_domu is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/x86.xen_domu/builds/185 Jan 30 10:13:55 build #164 of avr32 is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/avr32/builds/164 Jan 30 10:13:58 build #168 of xburst is complete: Failure [failed svn shell_15 compile_9] Build details are at http://buildbot.openwrt.org:8010/builders/xburst/builds/168 Jan 30 11:58:51 build #171 of x86 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/x86/builds/171 Jan 30 12:06:33 wigyori r48559 trunk/target/linux/mxs/patches-4.4/ (6 files) * mxs: forward-port regulator patches into 4.4 Jan 30 12:13:00 wigyori r48560 trunk/target/linux/mxs/config-4.4 * mxs: enable regulator in 4.4 config Jan 30 12:13:23 wigyori r48561 trunk/target/linux/mxs/Makefile * mxs: bump to 4.4 Jan 30 13:19:39 nbd r48562 trunk/target/linux/ar71xx/ files/drivers/leds/leds-nu801.c files/arch/mips/ath79/mach-arduino-yun.c * ar71xx: fix a few include errors Jan 30 13:19:55 nbd r48563 trunk/target/linux/ar71xx/ (95 files in 3 dirs) * ar71xx: add support for linux 4.4 Jan 30 13:35:33 rmilecki r48564 trunk/target/linux/generic/ patches-4.4/773-bgmac-add-srab-switch.patch patches-4.4/771-bgmac-support-Ethernet-device-on-BCM47094-SoC.patch patches-4.4/770-bgmac-add-helper-checking-for-BCM4707-BCM53018-chip-.patch * kernel: support bgmac on BCM47094 Jan 30 14:27:34 hauke r48565 packages/devel/make * make: moved to github Jan 30 14:28:02 hauke r48566 packages/mail/mutt * mutt: moved to github Jan 30 14:28:27 hauke r48567 packages/net/cifs-utils * cifs-utils: moved to github Jan 30 14:28:52 hauke r48568 packages/utils/bonnie++ * bonnie++: moved to github Jan 30 14:29:18 hauke r48569 packages/utils/stress * stress: moved to github Jan 30 14:31:40 hauke r48570 packages/ (11 files in 4 dirs) * remove empty directories Jan 30 16:56:53 rmilecki r48571 trunk/target/linux/bcm53xx/patches-4.4/710-b53-add-hacky-CPU-port-fixes-for-devices-not-using-p.patch * bcm53xx: use proper CPU port in b53 for Netgear R8500 Jan 30 16:57:37 rmilecki r48572 trunk/target/linux/bcm53xx/base-files/lib/upgrade/platform.sh * bcm53xx: support Netgear R8500 in sysupgrade Jan 30 18:00:37 Morning, all. Commit d5c17a17 (update musl to 1.1.12 + git) causes userland programs to immediately crash and burn on kirkwood hardware. I tested with init as init, init=/bin/cat, and init=/bin/sh, and all of them kernel panic'd with "killing init" exit code 0x0b and no stack trace or anything of the sort. What do I do to produce more useful debugging information? Jan 30 18:01:04 nwf: did you try a fresh build? Jan 30 18:02:15 nbd: Yeah, I rm -rf bin/ staging_dir/ build_dir/ and the ccache just in case and it's still gone poorly. Jan 30 18:03:27 That commit works fine in my build that I'm running with "qemu-system-arm -M realview-eb-mpcore" to test some userland programs; whatever's wrong appears to be architecture-specific. Jan 30 18:05:42 can you execute binaries from the kirkwood build in the realview one via chroot? Jan 30 18:07:15 Oof; I haven't tried, but I can. Lemme build a not-ubifs for the kirkwood build and get it landed on the realview's SD image. Jan 30 18:07:37 you don't need to build the whole image Jan 30 18:07:43 just copy the files from the staged root dir Jan 30 18:09:00 Oh, duh, good call. Jan 30 18:12:10 This is still going to take a bit as I have to build now with d5c17a17 not reverted. Jan 30 18:34:17 nbd: Yes, running "chroot /root-kirkwood", which is from the kirkwood build's staging_dir/target-arm_xscale_musl-1.1.12_eabi/root-kirkwood, on my realview image works fine. Jan 30 19:10:20 build #152 of mpc83xx is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/mpc83xx/builds/152 Jan 30 19:24:38 build #151 of ep93xx is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/ep93xx/builds/151 Jan 30 19:40:17 build #208 of lantiq is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/lantiq/builds/208 Jan 30 19:41:05 build #205 of ramips.mt7628 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/ramips.mt7628/builds/205 Jan 30 19:45:19 build #180 of netlogic is complete: Failure [failed compile_8] Build details are at http://buildbot.openwrt.org:8010/builders/netlogic/builds/180 Jan 30 19:56:29 nbd: Incidentally, I have a slew of Pogoplug V4 that I use as embedded servers for various things; I would gladly donate one to the OpenWrt project if such a thing would be useful for increasing testing coverage or something. Jan 30 19:57:37 btw. i'm going to commit another musl update very soon Jan 30 19:57:41 maybe it'll fix things Jan 30 19:58:25 Oh, OK. I am at present building with toolchain/gcc/patches/5.2.0/230-musl_libssp.patch, toolchain/musl/patches/000-update-to-git-2016-01-22.patch, and toolchain/musl/patches/200-add_libssp_nonshared.patch deleted. Jan 30 19:59:10 Just to see if musl-as-close-to-1.1.12-as-I-could-get worked. Jan 30 19:59:58 (200-add_libssp... seems to not apply if I remove 000-update-to... which is why it and 230-musl... had to go.) Jan 30 20:00:15 If it works out, I'll let you know. Jan 30 20:00:56 000-update-to... is a pretty big and exciting patch; lots of changes to very early and primitive code. Jan 30 20:02:28 it's simply the result of applying a git diff from v1.1.2 to master in the musl repo Jan 30 20:02:56 Oh, no, I understand; I am more commenting about the changes made in musl upstream. Jan 30 20:03:05 ;) Jan 30 20:15:06 nbd r48573 trunk/package/devel/perf/musl-compat.h Jan 30 20:15:07 perf: include asm/unistd.h instead of syscall.h to fix conflict with kernel headers Jan 30 20:15:12 nbd r48574 trunk/toolchain/musl/ (7 files) * musl: update to the latest git version, fixes TLS issues Jan 30 20:15:18 nbd r48575 trunk/toolchain/musl/patches/020-ldso-fix-GDB-dynamic-linker-info-on-MIPS.patch * musl: fix GDB debugging on mips (local, remote) Jan 30 21:02:04 build #154 of at91 is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/at91/builds/154 Jan 30 21:07:38 nbd: FYI, removing those patches (and commenting out the -lssp_noshared in package/libs/uclibc++/Makefile) gets me a system that boots. I will test now with your recent push, but expect it to be a while to rebuild and all that. Jan 30 21:08:12 you can just rm -rf build_dir/toolchain-*/musl* bin/* and run make again Jan 30 21:08:20 that way it won't rebuild the full toolchain Jan 30 21:08:43 Thanks Jan 30 21:24:11 That was much faster, but doesn't boot, either. Jan 30 21:27:30 ok Jan 30 21:27:58 then let's debug this Jan 30 21:28:02 first put an image on there that boots Jan 30 21:28:11 e.g. the one you built previously Jan 30 21:28:22 make sure you keep the build tree with the current (broken one) Jan 30 21:28:26 I am going to try with 000-update_musl and not the ssp patches, just to try to narrow it down, first. Jan 30 21:28:35 no need Jan 30 21:28:37 OK Jan 30 21:28:38 i don't think it's the ssp stuff Jan 30 21:28:43 it's most likely the update that broke it Jan 30 21:28:50 and testing individual changes is way too slow Jan 30 21:28:56 we should see where exactly it crashes Jan 30 21:29:09 OK, so you want me to grab the root-kirkwood/ of a bad build and land it on a good build? Jan 30 21:29:14 yes Jan 30 21:29:20 Can do Jan 30 21:29:39 you can do: ulimit -c unlimited; ./root-kirkwood/lib/libc.so /bin/busybox Jan 30 21:29:48 it should produce a core file Jan 30 21:29:53 you can copy that to your host Jan 30 21:30:01 and feed it to ./scripts/remote-gdb Jan 30 21:30:30 libc is built with debug info by default Jan 30 21:30:43 so gdb should be able to extract meaningful information out of the coredump Jan 30 21:31:52 Sounds like a plan Jan 30 21:52:30 For reasons unclear, it looks like I need to do a more clean build. That's going to take a while, so I am going to dash off. I will be back later this evening. Jan 30 21:53:02 i'll probably be asleep then Jan 30 21:53:14 so maybe tomorrow Jan 30 23:28:48 build #150 of mpc85xx is complete: Failure [failed compile_5] Build details are at http://buildbot.openwrt.org:8010/builders/mpc85xx/builds/150 Jan 31 00:02:19 nbd: I finally got a backtrace for you: http://pastebin.com/FKrcSLV9 Jan 31 00:03:17 nwf: yeah, i've reproduced it on an xscale device now Jan 31 00:03:24 Whoo. :) Jan 31 00:03:27 talking to musl devs to figure out exactly what's going on Jan 31 00:03:37 <3 Jan 31 00:03:56 If anything I can provide would be useful, please don't hesitate to ask. Jan 31 00:41:56 nwf: with the help of the musl guys we figured out the bug Jan 31 00:42:01 So I see! Jan 31 00:42:06 really nasty alignment stuff Jan 31 00:42:12 which gets broken by processor quirks Jan 31 00:42:24 ah, you're in that channel too Jan 31 00:42:30 Just joined. :) Jan 31 00:42:57 Still, thank you for the heads up. Jan 31 00:43:20 this is the third weird musl bug i ran into today Jan 31 00:43:46 It looks like post-1.1.12 is traumatic to the codebase and shaking out a lot of accidental success. ;) Jan 31 00:43:47 first one was one that broke GDB on mips Jan 31 00:43:56 which was in the way of debugging a second one Jan 31 00:44:17 which was a thread local storage init bug leading to crashes in elfutils (used by perf) Jan 31 00:44:22 and now this Jan 31 00:45:00 I have a storied history of encountering TLS bugs. Jan 31 00:46:02 ( https://sourceware.org/bugzilla/show_bug.cgi?id=6577 perhaps the strangest to debug, at least of the ones I remember ) Jan 31 00:46:33 ;) Jan 31 00:47:28 good thing i still had that xscale board lying around Jan 31 00:47:39 without looking into this directly, this would have taken ages to figure out Jan 31 00:47:47 I see that https://dev.openwrt.org/wiki/WantedHardware enumerates pogoplugs as desired; is that still the case? Jan 31 00:47:52 no idea Jan 31 00:48:24 btw. handling of alignment in ARMv5 is truly idiotic Jan 31 00:48:45 a few years ago i debugged something there that was even nastier Jan 31 00:49:04 ARM seems like it's as freaky as x86 in some ways. Jan 31 00:49:10 ARM has multiple different instructions for 64-bit load/store Jan 31 00:49:14 Oh boy Jan 31 00:49:18 one is faster, but requires 64 bit alignment Jan 31 00:49:25 Ah ha. Jan 31 00:49:36 the other one is a tiny bit slower, but works with unaligned addresses as well Jan 31 00:49:47 GCC was feeling lucky and used the faster one Jan 31 00:49:52 without knowing anything about the real alignment Jan 31 00:50:02 i actually patched GCC to prevent the use of this stupid instruction ;) Jan 31 00:50:09 it was causing random kernel crashes Jan 31 00:52:37 Codegen bugs are the worst. Jan 31 00:53:00 How did Mickens put it? "I have no tools because I have used my tools to break the tools that I use to fix my tools" or something like that? Jan 31 00:53:14 ;) Jan 31 00:53:17 "GCC was feeling lucky" hahahah Jan 31 00:55:01 nbd r48576 trunk/target/linux/ixp4xx/image/Makefile * ixp4xx: fix initramfs builds Jan 31 00:55:07 nbd r48577 trunk/toolchain/musl/patches/000-update-to-git-2016-01-30.patch * musl: fix an alignment issue that was breaking ldso on ARMv5 Jan 31 00:55:12 nwf: there. fixed ;) Jan 31 00:55:20 :D <3 Jan 31 01:36:58 build #181 of netlogic is complete: Success [build successful] Build details are at http://buildbot.openwrt.org:8010/builders/netlogic/builds/181 Jan 31 01:40:44 build #205 of cobalt is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/cobalt/builds/205 Jan 31 01:44:06 build #205 of cns21xx is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/cns21xx/builds/205 Jan 31 01:46:20 build #208 of bcm53xx is complete: Failure [failed compile_4] Build details are at http://buildbot.openwrt.org:8010/builders/bcm53xx/builds/208 Jan 31 01:47:51 build #205 of orion is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/orion/builds/205 Jan 31 02:36:53 build #202 of pxa is complete: Failure [failed shell_10] Build details are at http://buildbot.openwrt.org:8010/builders/pxa/builds/202 **** ENDING LOGGING AT Sun Jan 31 02:59:58 2016