**** BEGIN LOGGING AT Wed Mar 27 02:59:57 2019 Mar 27 07:27:17 Morning! Mar 27 07:27:28 JaMa: let's remove ld ! oh, wait... Mar 27 08:12:26 Morning! Mar 27 08:17:39 Tofe: So if I understand correctly and things work OK on Thud, we'll switch to Thud? Mar 27 08:18:02 And then maybe push out a release and switch to Warrior when issues have been ironed out? Mar 27 08:56:12 Herrie|Laptop: that's the idea Mar 27 08:56:45 I'll have to double check that the 2min delay is indeed gone, as maybe it's now present when systemd-udev starts during init Mar 27 09:18:13 Tofe: we can switch to musl :) Mar 27 09:18:42 http://jenkins.nas-admin.org/job/LuneOS/job/luneos-unstable_tissot/45/console Mar 27 09:18:49 the overnight build didn't get very far Mar 27 09:19:45 ah, well, let's wait a bit then Mar 27 09:22:51 Tofe: there is another issue in thud, the nyx-modules are missing because of openssl issue Mar 27 09:23:24 in thud already (not only in warrior), I guess we should try to fix that one before release as well, so intermediate release from sumo might be in order Mar 27 09:25:55 I should try to disable just the security module without removing whole nyx-modules first Mar 27 09:26:12 that might fix nyx-utils systemd service which might fix storaged/sleepd Mar 27 09:33:25 One solution could be to release the current sumo state: iirc, we're good to go on that level Mar 27 09:33:42 but there's no hurry Mar 27 09:36:30 if we release sumo and warrior works at least a bit, could we switch testing to warrior? to resolve remaning issues there? Mar 27 09:38:01 that would allow me to check out the yocto upgrade on my sabbatical wishlist :) starting next week I'll have less time again :/ Mar 27 09:43:19 JaMa: yes, that could be done Mar 27 09:45:43 ok, TODO for me: 1) busybox for sumo 2) gobject for warrior 3) openssl in nyx-modules (or at least better work around) Mar 27 09:47:04 for 2) I've switched qemuarm in oe-core to use the same cortexa8thf-neon as hammerhead and it didn't fail, now trying to build this qemuarm inside luneos, to compare what goes wrong in glibc Mar 27 09:47:50 it might be the old gcc building newer glibc (as nobody else was using this combination), but still I doubt that a bit Mar 27 10:04:13 JaMa: You became a MUSL fan all of the sudden? Khem infected you LOL ? Mar 27 10:05:10 hehe, not really, but as we cannot remove it, we can replace it with something else Mar 27 10:05:27 which would surely bring more new issues than solve the qemu-arm issue Mar 27 10:05:48 musl for us isn't as useful as for commercial products Mar 27 12:36:24 Tofe: Was working on some side project just to refresh the memory with a OnePlus device. Seems that our init is not being started even. I guess it's SELinux related? https://bpaste.net/show/3db717f79ee0 Mar 27 12:37:57 Ah seems that's actually a recovery log :S Mar 27 12:38:08 Not even a LunOS kernel since it's GCC .49 Mar 27 12:45:20 right Mar 27 12:45:39 OK will let him try with just the boot.img from the image Mar 27 12:45:43 that's the pstore ? Mar 27 12:45:53 To boot that via fastboot Mar 27 12:46:07 He didn't have pstore which is also weird since it's 7.1 device Mar 27 12:46:17 it's a 3.4 kernel Mar 27 12:46:22 Just OPO changed a lot with their OS switch etc Mar 27 12:46:30 ==> patched fstat ?... Mar 27 12:46:50 This was still a pyro image I put together like a month ago ;) Mar 27 12:46:59 though we should get some kmsg... where is the defconfig ? is RAMCONSOLE enabled ? Mar 27 12:48:00 Tofe: https://github.com/Herrie82/thunder_kernel_oneplus_bacon/blob/oreo-oss/arch/arm/configs/thunder_bacon_defconfig Mar 27 12:48:48 https://github.com/Herrie82/thunder_kernel_oneplus_bacon/blob/oreo-oss/arch/arm/configs/thunder_bacon_defconfig#L3138 seems ok Mar 27 13:21:24 interesting, qemu-arm crashes on armv7a ld whenever ld is built with gold instead of bfd Mar 27 13:38:14 JaMa: I'm really at a loss with this crash Mar 27 13:38:42 which crash? Mar 27 13:38:48 qemu-arm? Mar 27 13:39:44 yes Mar 27 13:40:04 but at the same time, I didn't really experience it myself :) Mar 27 13:41:40 now with the reproducer in oe-core without any luneos changes I can easily compare qemuarm builds with bfd and gold and possibly ask for help upstream Mar 27 13:42:46 ok good Mar 27 13:43:12 it will be also interesting installing some small image on hammerhead to see if it crashes on the device as well or causes issues only inside qemu-arm Mar 27 14:05:46 nyx-modules work around updated Mar 27 15:06:55 hmm sumo image just finished building, but adb on my tissot recovery refuses to talk with me :/ Mar 27 15:49:38 either I've missed something in my jansa/sumo branches or the mdev doesn't work well in sumo Mar 27 15:49:41 [ 5.653419] ======= LuneOS/Halium =========== Mar 27 15:49:44 [ 5.655486] Starting mdev Mar 27 15:49:46 [ 6.316998] Begin: Running /scripts/local-top ... done. Mar 27 15:49:49 [ 6.317353] Begin: Running /scripts/local-premount ... done. Mar 27 15:49:51 [ 6.319476] /halium-boot.sh: line 395: udevadm: command not found Mar 27 15:49:54 [ 6.331594] initrd: checking filesystem integrity for the userdata partition Mar 27 15:49:57 let me recheck in thud Mar 27 15:56:52 JaMa: I think udevadm gets installed by udev which we dropped with Tofe's PR in meta-smartphone Mar 27 15:57:39 JaMa: this looks actually good Mar 27 15:57:44 Added comments on the PR on the lines Mar 27 15:58:17 Tofe: OK, then I didn't say anything Mar 27 15:58:20 Tofe: do you have some pending chagnes for initramfs-tools-halium/tofe/ab-sheme to stop calling udevadm? Mar 27 15:59:27 https://github.com/Tofee/initramfs-tools-halium/blob/tofe/ab-scheme/scripts/halium#L395 Mar 27 15:59:47 ok, if this looks good, than I need to find other reason for my tissot not booting Mar 27 16:00:41 Tofe/JaMa: I guess it would be good to add a check for existence of udevadm binary in the script and only execute udevadm in case it exists. This way we keep compatibility with upstream Halium as well for now Mar 27 16:00:57 Until we figure out best course of action here going forward with udev/mdev etc Mar 27 16:01:03 not booting to UI.. whole dmesg: http://dpaste.com/286YT8F Mar 27 16:02:49 Don't we need ueventd? Also storaged failing like this is suspicious Mar 27 16:02:58 Though could be nyx related Mar 27 16:04:25 luna-next is starting as well, just showing: Mar 27 16:04:26 Mar 08 04:50:55 tissot luna-next[4087]: library "libgui.so" wasn't loaded and RTLD_NOLOAD prevented it Mar 27 16:04:46 which I don't remember seeing before Mar 27 16:05:21 JaMa, Herrie|Laptop: yes I would need to PR something to drop that little line, or at least to call it only when udevadm is present Mar 27 16:05:36 but I didn't actually do anything yet Mar 27 16:06:01 I'll try on hammerhead once it charges a bit Mar 27 16:06:01 JaMa: libgui.so is a usual warning Mar 27 16:06:07 ok, ty Mar 27 16:06:15 wayland-egl ?... Mar 27 16:06:31 let me look at that dmesg Mar 27 16:07:08 exe="/usr/sbin/storaged" sig=11 <-- not sure if that's fatal ? Mar 27 16:07:54 storaged doesn't run with thud and newer at all, because of missing nyx-modules and it's not fatal there Mar 27 16:08:22 ok Mar 27 16:08:31 it's also possible that I've broke it when doing my silly cosmetics changes on top of your PR Mar 27 16:08:44 we'll see with thud image Mar 27 16:09:14 then "init: Timed out waiting for /dev/.coldboot_done" is probably the issue, or its visible side Mar 27 16:09:37 ** be back in a big hour or so Mar 27 16:19:24 heh the qemu-arm/glibc/gold issue was introduced during christmas, world builds detect the same issue Mar 27 16:21:44 JaMa: Oh well good it's not isolated to us ;) Mar 27 16:21:56 That means it's likely there's a fix or will be upstream ;) Mar 27 17:59:36 re Mar 27 18:00:22 JaMa: do you have the image somewhere, if I can be of some help? Mar 27 18:01:02 only locally, I can upload it somewhere Mar 27 18:01:56 as you wish Mar 27 18:03:14 ~5 mins uploading to milla Mar 27 18:03:35 heh and then it went to 500kB/s Mar 27 18:05:24 might be faster if you do local build with the few changes from jansa/sumo for meta-webos-ports and meta-smartphone Mar 27 18:07:10 unfortunately no: I've dropped tmp-glibc and sstate when I saw that the changes to libhybris/libwayland triggered a qtbase rebuild... Mar 27 18:14:19 I see 82% 508MB 915.2KB/s 01:58 ETA Mar 27 18:14:48 definitely faster than a 5h rebuild :p Mar 27 18:15:53 the sstate on jenkins should be up2date now, rsync finished few minutes ago Mar 27 18:17:13 Tofe: http://build.webos-ports.org/jama/luneos-dev-package-tissot-20190327153004-jama.zip Mar 27 18:23:10 ~12min left Mar 27 18:38:19 I've created another pipeline jobs like http://jenkins.nas-admin.org/job/LuneOS/view/testing/job/luneos-testing_build/ (without -all) suffix which build just qemux86, tissot, hammerhead Mar 27 18:38:28 ok, thanks Mar 27 18:38:35 there is another issue with shared TUNE_PKGARCH which is making our builds slow I think Mar 27 18:39:11 raspberrypi3-64 is using the same aarch64 TUNE_PKGARCH like e.g. tissot, but even openssl is a bit different for both Mar 27 18:39:54 once luneos-testing_workspace-sstate-cleanup is executed one of the versions is removed (and whatever was built last - usually raspberrypi3-64 is kept) Mar 27 18:40:07 and in next build the 1st version needs to be rebuilt again, then it gets removed etc Mar 27 18:40:44 nobody is fetching those rpi images, so we can stop building them until they are fixed Mar 27 18:43:12 yes Mar 27 18:43:38 ok rebooting to newly flashed image Mar 27 18:44:01 grep "images/raspberrypi.*rpi-sdimg" build.webos-ports.org.access.log | wc -l Mar 27 18:44:04 270 Mar 27 18:45:03 ok android is dead Mar 27 18:47:22 from 20 "/2019.*images/raspberrypi.*rpi-sdimg" cases we have 4 Googlebot 1 applebot 4 www.opensiteexplorer.org 1 YandexBot 2 www.semrush.com Mar 27 18:49:05 JaMa: I think a lot of android partitions are not mounted here Mar 27 18:49:24 but why did it work, in my case ?.... Mar 27 18:50:17 I might have broken it :/ Mar 27 18:50:34 maybe, but I don't see how ! Mar 27 18:50:34 either by cosmetics or the backport from thud to sumo Mar 27 18:50:52 or just the busybox version in sumo works differently Mar 27 18:51:01 or it's one of my other local changes causing this Mar 27 18:51:44 let me check if it was included in latest tissot/warrior build or not Mar 27 18:52:40 meta-smartphone: HEAD is now at 8f06eb35 hammerhead: temporarily disable gobject introspection Mar 27 18:52:49 meta-webos-ports: HEAD is now at be1a2df26 initramfs-boot-android: cosmetics Mar 27 18:52:52 so should be in Mar 27 18:53:11 http://build.webos-ports.org/luneos-unstable-warrior/images/tissot/luneos-dev-package-tissot--unstable-0-45.zip Mar 27 18:55:09 https://github.com/Halium/initramfs-tools-halium/blob/halium/scripts/halium#L155 doesn't look in by-partname, so it can't work Mar 27 18:55:26 I really don't see why it worked in my case... Mar 27 18:55:54 only the nyx-modules seems to be missing Mar 27 18:56:54 interesting Mar 27 18:58:30 I'll create the same symlinks in by-name, it should fix this Mar 27 18:59:00 how difficult is it to get networking or adb running inside the initramfs before mounting the rest? or how did you debug this in first place? Mar 27 18:59:58 in theory I can use qemu to debug busybox-mdev as it's easy to get serial out Mar 27 19:02:40 https://github.com/shr-distribution/meta-smartphone/pull/96 <-- can you try this ? Mar 27 19:03:08 sure, sec Mar 27 19:04:46 might take a while even with such small change.. load average: 30.72, 27.30, 25.24 Mar 27 19:06:21 let me modify it inside already built image :) Mar 27 19:16:26 with warrior it probably worked even without this PR Mar 27 19:16:58 my guess is that with newer busybox in thud and warriror we already had by-name from somewhere else Mar 27 19:17:47 warrior seems to work quite well on tissot, got the firstuse, ui is working, let me check if there was boot delay Mar 27 19:19:01 no delay in dmesg Mar 27 19:20:18 dmesg on tissot/warrior: http://dpaste.com/2VA12KQ Mar 27 19:29:43 and the diff between older and newer warrior also looks good http://dpaste.com/2AR5PFF Mar 27 19:31:14 I haven't found anything about by-name in busybox mdev between 1.27.2 and 1.29.3 nor in oe-core diff between sumo and thud, strange Mar 27 19:33:26 looks like someone had similar issue with yocto: https://stackoverflow.com/questions/49822694/how-is-dev-block-bootdevic-by-name-created Mar 27 19:44:46 killed ext4 IMAGE_FSTYPE before next unstable rebuild: https://github.com/shr-distribution/meta-smartphone/commit/c24e3bcb8bdf4737b66311e2ee92539e1def60c2 Mar 27 19:51:53 damn, now tar gets killed by OOMK even before unpacking our tar or ld :/ cannot flash new sumo build again Mar 27 20:00:03 JaMa: I'm lost, did my little patch work for your image ? Mar 27 20:01:43 Tofe: I haven't tried on sumo yet Mar 27 20:01:57 Tofe: but your PR doesn't seem to be needed with warrior (last build from jenkins) Mar 27 20:02:01 http://build.webos-ports.org/luneos-unstable-warrior/images/tissot/luneos-dev-package-tissot--unstable-0-45.zip Mar 27 20:02:27 Tofe: now my local sumo build was already finished, but cannot flash it on my device normally again Mar 27 20:02:34 so I'm doing it manually Mar 27 20:03:00 well flashing just boot.img should be enough, right? but I already had the warrior image installed between Mar 27 20:07:00 doesn't seem to be working any better with sumo: http://dpaste.com/3R8YTQM Mar 27 20:09:41 can you check what mounts do you have ? Mar 27 20:11:16 during init or from running system? Mar 27 20:11:39 https://paste.ubuntu.com/p/ZK7VhGdkMR/ Mar 27 20:15:57 http://build.webos-ports.org/jama/boot.img has the modified initramfs Mar 27 20:22:19 mmh I still don't see the necessary mounts, so I must have made a mistake here... Mar 27 20:24:03 For instance, just after "initrd: checking mount label modem", we should see "initrd: mounting /dev/disk/by-name/modem_a as /rfs/firmware" Mar 27 20:24:15 n -d /dev/disk/by-partname ] || mkdir -p /dev/disk/by-partname Mar 27 20:24:22 this is missing for by-name Mar 27 20:25:35 good catch ! Mar 27 20:26:19 added to https://github.com/shr-distribution/meta-smartphone/commits/jansa/sumo Mar 27 20:26:21 re-pushed. fwiw... Mar 27 20:26:47 Bisecting: 5 revisions left to test after this (roughly 3 steps) Mar 27 20:27:20 JaMa: shouldn't we flatten a bit all this busybox rework ? Mar 27 20:29:29 Tofe: to squash it a bit? it's already applied except the last 2 fixes Mar 27 20:29:41 oh, ok Mar 27 20:39:48 current state of qemu-arm issue: http://lists.openembedded.org/pipermail/openembedded-devel/2019-March/198937.html Mar 27 20:43:48 I would have the same a-priori guess as yours Mar 27 20:50:24 from objdump -x diff it looks like ld-*.so wasn't linked with gold before even when gold was enabled, so first work around might be to disable gold even when ld-is-gold is being used Mar 27 20:52:24 yes, all this seems to be tied to the gold support Mar 27 20:52:45 (which is supposed to be faster, I guess) Mar 27 21:01:00 yes, using less memory while linking and a bit stricter when linking so supposed to drop few more links with as-needed Mar 27 21:06:05 my by-name directory fix doesn't seem to work as well Mar 27 21:08:38 but we're getting closer Mar 27 21:08:39 -initrd: mounting /dev/disk/by-name/misc as /rfs/android//misc Mar 27 21:08:39 -mount: mounting /dev/disk/by-name/misc on /rfs/android//misc failed: No such device Mar 27 21:10:10 some of the partitions were mounted correctly http://dpaste.com/2DMQCCT modem_a dsp persist Mar 27 21:10:58 updated mounts: https://paste.ubuntu.com/p/GQyZhyZCCm/ Mar 27 21:12:34 should I push it to sumo and trigger another build on jenkins so that we have clean build without any of my other changes? Mar 27 21:12:49 or there is still an option to keep udevd in sumo branch Mar 27 21:30:13 let's rebuild yes, during the night Mar 27 21:48:03 I've added a work around for glibc as well, not tested though Mar 27 22:07:26 Tofe: https://github.com/Freescale/meta-freescale/commit/7dc760ad8b3b8d0f8562d60b9a60114f332abbea https://github.com/Freescale/meta-freescale/commit/34002dbb315467e8eb022c61e4cdfd9c7f7819b4 this is how freescale is solving use-mainline-bsp override Mar 27 22:08:44 Tofe: it's a bit ugly and I'm not sure if it solves all issues with shared TUNE_PKGARCH, so take it only as an example Mar 28 02:24:52 another small fix was needed in meta-smartphone to fix initramfs parsing for hammerhead (or !tissot android MACHINEs) Mar 28 02:25:21 lets see if it builds over night **** ENDING LOGGING AT Thu Mar 28 02:59:57 2019