**** BEGIN LOGGING AT Thu Mar 28 02:59:57 2019 Mar 28 07:11:45 Morning! Mar 28 07:12:15 Great progress on the various issues :) Mar 28 07:12:45 Lots of it is low level Linux so a bit over my head still :'( Mar 28 07:18:02 Morning! Mar 28 10:14:51 morning Mar 28 10:23:59 latest warrior and sumo images are being rsynced now Mar 28 11:35:36 warrior works on hammerhead, it just rebooted once and now I'm in firstuse :) Mar 28 12:03:45 JaMa: ah, that's good ! hammerhead, like, non-mainline? Mar 28 12:06:43 yes non-mainline Mar 28 12:07:08 but it seems to randomly reboot from time to time, not sure if the same is/was happening with thud or sumo Mar 28 12:07:33 will trigger new thud builds in half hour or so when current warrior builds are cleaned Mar 28 12:08:29 it might be that something is still missing (like some android part poking the watchdog?) because of missing by-name (the latest mdev changes are only in sumo, because they didn't appear needed in warrior) Mar 28 12:08:34 JaMa: if I had to build (from scratch) a new build, what would you suggest ? tissot on thud? Mar 28 12:09:01 here is the dmesg if you see anything strange in there http://dpaste.com/2S1GB00 Mar 28 12:09:16 I would go with tissot on warrior Mar 28 12:09:45 it seems to work reasonably well and most of the remaining issues (like openssl) exist in both thud and warrior Mar 28 12:11:00 btw: do you have serial cable for hammerhead (through audio jack)? Mar 28 12:11:07 JaMa: lines 682 to 695 are a bit surprising, nothing seems to be mounted; but at least a firmware should be there, I guess Mar 28 12:11:23 JaMa: no, I don't have such wonderful cable :p Mar 28 12:11:43 http://www.abclinuxu.cz/blog/Lorris/2013/12/serial-console-on-google-nexus-5 Mar 28 12:11:58 today I realized that I might have one :) Mar 28 12:12:22 JaMa: though it might be mounted with line 767 to 769 Mar 28 12:13:16 and we have uart_console=enable in kernel cmdline, but we might need console=ttyHSL0,115200,n8 as well to get some output (I didn't get any with current image) Mar 28 12:14:12 Tofe: I can quickly rebuild with added by-name to compare with that Mar 28 12:14:21 btw, for fastboot-based boot images, we don't need to pass on the "CMDLINE" value; that means the kernel doesn't need to be rebuilt when we change the cmdline Mar 28 12:14:28 but first I want to finish this serial cable to test the qemu-arm/glibc fix Mar 28 12:15:05 Tofe: you mean that I can modify just the command line when creating .fastboot, right? Mar 28 12:15:15 not the one built-in kernel itself Mar 28 12:17:08 the cmd line set in the machine conf, but modifying it should only impact the bootimg itself, not the kernel, so we should remove the line CMDLINE="blabla" from the kernel recipes Mar 28 12:17:43 hmm rebooting tissot caused firstuse to be shown again Mar 28 12:17:49 I just remarked that yesterday when toying with it Mar 28 12:19:34 interesting, that can mainly mean two things: 1) the rootfs is mounted read-only, or 2) the ls2 hub was broken when the signal was sent by firstuse Mar 28 12:20:46 http://dpaste.com/2S1GB00#line-669 this indicates rw, iirc Mar 28 12:25:31 this is from hammerhead, but the firstuse was shown again there as well Mar 28 12:26:09 mounts on hammerhead http://dpaste.com/36MNR2S Mar 28 12:28:57 hey, what's this: "/dev/mmcblk0p16 on /run/media/mmcblk0p16" Mar 28 12:29:16 can you check mdev.conf in your boot image initrd ? Mar 28 12:29:38 it looks like it still calls mdev-mount.sh somehow Mar 28 12:30:58 https://github.com/openembedded/openembedded-core/blob/thud/meta/recipes-core/busybox/files/mdev.conf#L44 this should _not_ be here Mar 28 12:36:03 I saw that your modified the FILESEXTRAPATHS_prepend in https://github.com/Tofee/meta-smartphone-1/commit/5343bed92aab868239668a6fcf9a47312e6c2383#diff-cee02d7f89cd24d70dd2277b4a6f06dd , could it be it doesn't override mdev.conf anymore? Mar 28 12:36:57 (though your version seems fine too...) Mar 28 12:46:37 Tofe: sorry got distracted for a bit, I'll check the initrd soon Mar 28 12:53:53 Tofe: mmcblk[0-9].* 0:0 660 */etc/mdev/mdev-partname.sh Mar 28 12:54:37 is in the /etc/mdev.conf included in the initrd of the boot.img I've flashed with the http://build.webos-ports.org/luneos-testing-sumo/images/hammerhead/luneos-dev-package-hammerhead-20190328033342-testing-0-148.zip Mar 28 12:57:40 that's weird... who mounted the /run/media stuff ? udev ? Mar 28 12:58:59 note that it's probably not a problem in itself. But I thought we were calling the wrong script for mdev. Mar 28 13:01:44 many more /run/media mounts on tissot http://dpaste.com/3F5T7BT Mar 28 13:02:58 JaMa: while you're on the device: what do you have in /android folder ? any "firmware" or "modem" subfolder ? Mar 28 13:03:19 we still don't have those by-name links in thud and warrior, maybe halium-boot.sh is using those /dev/disk/by-partlabel instead? Mar 28 13:03:43 sh-4.4# ls /android/ Mar 28 13:03:44 data system vendor Mar 28 13:03:47 ^ on tissot Mar 28 13:04:19 the by-name links are probably still needed then; on our side we reuse exactly the same halium-boot.sh script Mar 28 13:04:46 I should probably add find /dev/disk in the halium-boot.sh to see what is really available during the init Mar 28 13:05:10 or in init.sh, that'll be easier, as it isn't fetched Mar 28 13:05:30 (in meta-webos-ports's override) Mar 28 13:05:30 what about even calling it with -x? Mar 28 13:06:01 what does -x do ? Mar 28 13:06:32 it will be annoyingly long, but probably better than debugging with echo statements (reminds me very early days of javascript debug with alert()) :) Mar 28 13:06:38 traces all shell code Mar 28 13:08:05 mmmh but where would you put that "-x" ? halium-boot.sh is sourced by init.sh, and the latter is called by the kernel Mar 28 13:08:45 init.sh shebang Mar 28 13:08:51 oh. Mar 28 13:08:58 well, go ahead, and we'll see Mar 28 13:09:02 will do both find /dev/disk and -x Mar 28 13:09:40 need to finish few qtwebengine builds first :/ Mar 28 13:09:46 I just hope it'll fit in the kernel's message ring before journald kicks off Mar 28 13:10:21 hmm maybe I should try on hammerhead together with serial console change Mar 28 13:10:42 it will be much easier with getty started inside initramfs Mar 28 13:11:40 on the + side, the thud build on jenkins is relatively quick, should finish in an hour or 2 Mar 28 13:11:50 NOTE: do_populate_sysroot: 98.9% sstate reuse(548 setscene, 6 scratch) Mar 28 13:13:21 systemd mounts: https://paste.ubuntu.com/p/6j278k9rRn/ Mar 28 13:16:20 where is the source of the initramfs init? it doesn't look like ~/meta-smartphone/meta-android/recipes-core/initrdscripts/initramfs-boot-android/init.sh Mar 28 13:16:29 ok, so I guess it's systemd who takes the initiative for these mounts; it's not really a problem. But if they are not mounted also during halium's boot, then it's a real problem Mar 28 13:16:51 metta-webos-ports overrides it Mar 28 13:17:57 (when the mainline work about this is merged, this will be simplified) Mar 28 13:18:06 ok, I see it now Mar 28 13:18:21 I was looking in jansa/warrior branch with your mainline changes Mar 28 13:18:45 I have few changes for your mainline changes, but we can go through them once the yocto upgrade is resolved Mar 28 13:19:10 yes, one thing at a time. Or, in this case, about 3 or 4 :) Mar 28 13:20:21 e.g. https://github.com/shr-distribution/meta-smartphone/commit/c3d2c9aa42e97b6d001c5de89839f3b975e4e98c broke the build yesterday Mar 28 13:20:36 and similar issue exists with the new VIRTUAL-RUNTIME variables you've added Mar 28 13:20:54 it's always safer to set some sane default with ?= before using this variable Mar 28 13:21:05 and then just override the value e.g. in MACHINE.conf where needed Mar 28 13:21:22 oh, my bad, this kind of thing gets quite over my head Mar 28 13:21:26 and you tend to add some unnecessary spaces around values :) Mar 28 13:21:45 yes, I never know if it manages the spacing automagically... Mar 28 13:21:59 += always adds leading space Mar 28 13:22:24 you need to provide one only when using _append for space separate list (like DEPENDS etc) Mar 28 13:22:48 and you almost never need trailing space (only exception is _prepend to space separated list) Mar 28 13:23:45 trailing spaces on the line are just annoying, because git highlights them (so I tend to clean them when applying such diff) Mar 28 13:24:19 i'll do my best in the future :) Mar 28 13:24:47 and inconsistent indentations (especially tabs and spaces) annoy me when git diff shows the line incorrectly indented, because the +- characters on first character in diff mess with tab size Mar 28 13:26:02 do_compile () { Mar 28 13:26:02 # -Wl,-rpath-link /lib in LDFLAGS can cause breakage if another glibc is in staging Mar 28 13:26:05 - unset LDFLAGS Mar 28 13:26:07 + LDFLAGS="-fuse-ld=bfd" Mar 28 13:26:10 base_do_compile Mar 28 13:26:12 echo "Adjust ldd script" Mar 28 13:26:23 this looks just wrong because of tabs, but it's actually all lines using tabs consistently :/ Mar 28 13:27:09 not sure why tab after - is longer than the same tab after + Mar 28 13:29:17 Tofe: also did you notice: Mar 28 13:29:17 [ 8.172518] initrd: Stopping mdev Mar 28 13:29:18 [ 8.197018] killall: mdev: no process killed Mar 28 13:31:13 I don't see where it would be killed before stop_mdev Mar 28 13:33:02 JaMa: I copied this from postmarketos's init scripts, but I think they have the same message: mdev -s will just not stay alive one it has processed all the /sys nodes; it's called afterwards by kernel's uevent via the hotplug capability Mar 28 13:33:15 I could just remove stop_mdev Mar 28 13:34:39 aha, ok, then it's fine Mar 28 13:35:24 as long as it didn't die processing the uevents, I didn't know what -s does Mar 28 13:35:39 almost ready to test [17068/17842] CXX obj/third_party/WebKit/Source/core/css/css/css_jumbo_49.o Mar 28 13:38:07 anything else to add to debug? https://github.com/shr-project/meta-webos-ports/commit/18782b8b39bd22f7b1ffad7d3b65eac701eb5f82 Mar 28 13:39:21 looks good; the "mount" debug is a good idea Mar 28 13:39:51 note that "/dev/disk" shouldn't exist before mdev is started Mar 28 13:50:17 yes, that is expected result of the first one :) Mar 28 13:51:02 I can also sed the halium-boot.sh to add some debug for the mounts there Mar 28 13:51:10 but if -x works it shouldn't be needed Mar 28 13:53:02 well there are already quite some useful echos Mar 28 13:53:15 btw: qt upgrade to 5.12 is on hold until 5.12.2 issues are resolved https://github.com/meta-qt5/meta-qt5/pull/181 Mar 28 13:53:20 for the mount, if there is no message, there is no mount Mar 28 13:57:04 maybe ls ${mount_root}/$2 after the mount to show that something was indeed mounted there and looks sane? Mar 28 13:57:43 or some warning message instead of "[ ! -e "$path" ] && continue" Mar 28 13:58:45 as you wish, but it's usually pretty painful to modify things in there, because bitbake refetches it the modifs are lost Mar 28 13:58:56 +if Mar 28 13:59:29 I guess I've missed that there are more fstab files like /var/lib/lxc/android/rootfs/fstab Mar 28 13:59:53 I can modify it from init.sh with sed before it's sourced :) Mar 28 14:00:58 why would you want to modify these? Mar 28 14:01:34 I don't want to modify fstab, just halium-boot.sh for debug Mar 28 14:02:58 what I was missing was that mount_android_partitions is called for 2 different fstab files, I've completely ignored the one in lxc, so I was wondering where it finds the partitions to mount (as there aren't any android specific ones in /etc/fstab in initramfs) Mar 28 14:03:44 ah, yes, right Mar 28 14:03:52 that was the main thing I wanted to find out with -x but while waiting for the build to finish I guess I can continue to read how init/halium-boot.sh actually work Mar 28 14:04:32 [17722/17842] Mar 28 14:06:59 https://github.com/Halium/projectmanagement/issues/55 <-- "What does it do" Mar 28 14:10:16 also: https://github.com/Halium/docs/blob/2706dcca32aee476bf4e6e4e617dacd1cf32e773/standards/pre-rootfs-environment.rst Mar 28 14:11:19 (but the latter is more about how to standardize it a bit more than to describe what's currently available) Mar 28 14:14:14 thanks Mar 28 14:14:36 yes maybe I should read that instead because after loop mounting system.img I don't see the /var/lib/lxc/android/rootfs/fstab :) Mar 28 14:15:36 ah I was missing step 3. :) Mar 28 14:19:12 finally found ./fstab.hammerhead Mar 28 14:21:02 good! Mar 28 14:22:10 it comes from https://github.com/Halium/android_device_lge_hammerhead/blob/ubp-5.1/fstab.hammerhead Mar 28 14:23:25 shows me that I was completely ignoring android side of things for way too long :) Mar 28 14:23:56 with your mainline work I might ignore it again soon :) Mar 28 14:25:25 hehe, for me it's quite the contrary: working on mainline was a refreshing update on how things *should* work Mar 28 14:31:40 it's surprising that some things which sound that should be relatively simple (like just booting the device with few mounted partitions) get quite complicated quickly, not that booting TV or its partition scheme would be any simpler even without all android stuff.. :) Mar 28 14:32:53 thud build finished on jenkins, rsyncing now Mar 28 14:53:52 btw: there is small issue with jenkins-job.sh which can cause repeated builds to fail, we call delete_unnecessary_images function at the end of each build, so that rsync job is started it can rsync everything left in deploy/images, but if you start e.g. 2 qemux86 builds without cleanup job between them, then the 2nd will fail when trying to find the kernel image (unless something invalidated Mar 28 14:53:58 kernel.do_deploy task between the runs) Mar 28 14:54:52 I probably won't fix it now (the delete_unnecessary_images functions uses MACHINE to decide which images to remove), so this is just FYI that cleanup job is needed to resolve build errors like: Mar 28 14:57:23 http://jenkins.nas-admin.org/job/LuneOS/view/testing/job/luneos-testing_qemux86-64/99/console Mar 28 15:00:07 "install: cannot stat '/home/jenkins/workspace/luneos-testing/webos-ports/tmp-glibc/deploy/images/qemux86-64/bzImage': No such file or directory" ok Mar 28 15:14:56 I might just fix it in jenkins-job.sh :) Mar 28 15:15:21 but still if you call rsync (with delete_unnecessary_images) then the build before cleanup it would be broken still Mar 28 15:31:28 https://github.com/webOS-ports/jenkins-jobs/commit/14731a20a082588e6b0fb4e8271bd68b280111d5 should help a bit Mar 28 16:11:17 Tofe: finally finished the build.. here is debug output on tissot: http://dpaste.com/1S16QW6 Mar 28 16:11:29 Tofe: building hammerhead kernel now Mar 28 16:12:20 first part is there: "find: /dev/disk: No such file or directory" Mar 28 16:13:23 and then it looks like whole /dev/disk is empty, because there is nothing here: Mar 28 16:13:26 [ 6.325143] Before mountroot Mar 28 16:13:29 [ 6.328088] Begin: Running /scripts/local-top ... done. Mar 28 16:14:02 should we add some wait for mdev -s to finish its job? Mar 28 16:14:58 like we had "udevadm settle" for udev Mar 28 16:15:44 mdev man says: If /dev/mdev.log file exists, debug log will be appended to it. Mar 28 16:15:51 that might be useful here as well now Mar 28 16:19:02 retrying with https://github.com/shr-project/meta-webos-ports/commit/8b8864971e71e718a62092c1ec28463d0bffcd28 Mar 28 17:17:08 Tofe: updated dmesg with more debug: http://dpaste.com/06X43NR but something is still strange, there is no output from starting mdev (other than the kmsg echo) and the 2nd find still doesn't show anything Mar 28 17:17:18 goint go try on hammerhead now Mar 28 17:23:16 interesting discussion about udevadm settle https://github.com/rear/rear/issues/791 Mar 28 17:30:33 and locally built hammerhead kernel doesn't seem to boot at all Mar 28 17:33:21 interesting bit from halium-boot.sh: Mar 28 17:33:22 # Put all of this script's output into /dev/kmsg Mar 28 17:33:22 exec &>/dev/kmsg Mar 28 17:33:31 will do that in init.sh already Mar 28 17:45:33 and now I broke init completely :) Mar 28 17:55:49 http://dpaste.com/1ZVJSX9 Mar 28 17:56:16 strange that /dev/mdev.log is still so short, but clearly something doesn't work correctly here Mar 28 17:56:40 might try with this debug on thud or sumo Mar 28 17:59:23 last try with extra 30s sleep Mar 28 18:10:48 from http://dpaste.com/3M8T6CD it looks like all /dev/ nodes disappear as soon as mdev is started, are you sure that "echo /sbin/mdev > /sys/kernel/uevent_helper" is the right file? Mar 28 18:11:02 https://git.busybox.net/busybox/plain/docs/mdev.txt says: "echo /sbin/mdev > /proc/sys/kernel/hotplug" Mar 28 18:11:25 will try both Mar 28 18:30:52 I'm giving up on warrior, /proc/sys/kernel/hotplug didn't help at all: http://dpaste.com/3ARDNKZ will test with thud and sumo tomorrow Mar 28 18:45:43 testing it from running system creates just a bit longer mdev.log: 81 /dev/mdev.log Mar 28 18:46:12 but the partitions in by-partname are created: Mar 28 18:46:12 sh-4.4# find /dev/disk/by-partname/ | wc -l Mar 28 18:46:13 50 Mar 28 18:47:29 mdev.log: http://dpaste.com/1FK5CH5 Mar 28 19:00:42 http://lists.busybox.net/pipermail/buildroot/2014-February/088766.html has some hints as well, will try to enable sequence numbers in mdev Mar 28 19:12:56 * JaMa has 3 uart jack cables, they don't work yet with nexus5 but if I get it working I will ship one to you Mar 28 19:16:56 Tofe: have you tried to enable adb early in initrd? there is enable_adb flag in kernel cmdline to enable it (implemented in init.sh), but doesn't seem to work for me: Mar 28 19:16:59 # adb shell Mar 28 19:17:02 error: closed Mar 28 19:19:45 ah adb works (e.g. push) just shell doesn't Mar 28 19:30:44 JaMa: Will be back in a couple of hours, dinning outside ;) Mar 28 19:31:29 but yes, I've tried having adb from initrd... but no shell so far Mar 28 19:32:55 might be missing something in initrd as well https://paste.ubuntu.com/p/QPgWXCb5Vz/ Mar 28 19:33:28 Tofe: np, wife wants to watch GoT before new series starts, I'll start new build and call it a day soon Mar 28 19:40:50 maybe useful for adb: https://gitlab.com/nizovn/meta-smartphone/commit/f358cf0c8107f5c95131bc450482fd02118a726c Mar 28 19:43:42 nizovn: always hiding some useful knowledge on gitlab :0 Mar 28 19:44:13 :) Mar 28 19:47:03 the functionfs support is already in init.sh, but /dev/pts and the patch 0001-Allow-adbd-to-be-ran-as-root.patch might be missing Mar 28 19:47:33 0001-Allow-adbd-to-be-ran-as-root.patch should be there, i imported it from LuneOS Mar 28 19:48:41 took much stracing to find it... Mar 28 19:48:48 file://core/0008-adb-Allow-adbd-to-be-ran-as-root.patch;patchdir=system/core \ Mar 28 19:48:58 I've added it to meta-oe already it seems Mar 28 19:49:59 http://git.openembedded.org/meta-openembedded/commit/?id=934a034e0ece3527652b6fa84c1590e6f5847073 Mar 28 19:50:38 still leaves /dev/pts, because I'm sure that it doesn't exist at this point Mar 28 19:52:29 btw it's good to fix this too: https://gitlab.com/nizovn/meta-openembedded/commit/a01e638177ed0e40d80a340e09d0e8a4ba034a88 Mar 28 19:56:22 are you going to submit these upstream? Mar 28 19:59:20 well two- and one- liners i can cleanup and submit at weekend Mar 28 20:02:51 that would be great, hiding them in gitlab doesn't help the comunity much :) Mar 28 20:09:26 cool now it works, thanks to /dev/pts mount Mar 28 20:09:48 ok Mar 28 20:10:51 even /dev/ seems surprisingly well populated (no /dev/disk nor /dev/mdev.log yet) Mar 28 20:13:01 mdev works fine when executed manually Mar 28 20:13:23 /dev/disk/by-partname/ are created Mar 28 20:19:53 [ 37.861081] + for dir in by-partlabel by-name by-label by-path by-uuid by-partuuid by-id Mar 28 20:20:23 why does halium-build.sh try 7 different dirs, but not by-partname which seems to be quite common? Mar 28 20:23:49 or if we don't agree on by-partname, why not use by-partlabel like ./meta-luneos/recipes-core/systemd/systemd/0001-rules-consider-MMC-device-partitions-with-partition-.patch does for udev Mar 28 20:24:36 that's already used by halium-boot.sh, so whatever is by-partname, we can try to change it to by-partlabel as well Mar 28 20:25:06 git grep haven't found by-partname elsewhere than meta-android/recipes-core/busybox/busybox/mdev-partname.sh Mar 28 20:25:28 GoT now, bbl Mar 28 20:44:45 looks related: https://github.com/slashbeast/mdev-like-a-boss/blob/f91ef3636cd6ae15f373d4aca34c262658bb3630/helpers/storage-device#L67-L74 Mar 28 22:21:39 Tofe: with by-partlabel it mounts few more expected partitions: Mar 28 22:21:40 -/dev/mmcblk0p1 on /android/firmware type vfat (ro,noatime,uid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro) Mar 28 22:21:43 -/dev/mmcblk0p13 on /android/dsp type ext4 (ro,nosuid,nodev,relatime,data=ordered) Mar 28 22:21:46 -/dev/mmcblk0p27 on /android/persist type ext4 (rw,nosuid,nodev,relatime,data=ordered) Mar 28 22:21:55 -/dev/mmcblk0p13 on /var/lib/lxc/android/rootfs/dsp type ext4 (ro,nosuid,nodev,relatime,data=ordered) Mar 28 22:21:58 -/dev/mmcblk0p1 on /var/lib/lxc/android/rootfs/firmware type vfat (ro,noatime,uid=1000,fmask=0337,dmask=0227,codepage=437,iocharset=iso8859-1,shortname=lower,errors=remount-ro) Mar 28 22:22:02 -none on /var/lib/lxc/android/rootfs/frp type tmpfs (rw,relatime,size=4k,nr_inodes=455841) Mar 28 22:22:05 -none on /var/lib/lxc/android/rootfs/misc type tmpfs (rw,relatime,size=4k,nr_inodes=455841) Mar 28 22:22:08 -/dev/mmcblk0p27 on /var/lib/lxc/android/rootfs/persist type ext4 (rw,nosuid,nodev,relatime,data=ordered) Mar 28 22:22:43 Tofe: so unless you confirm that by-partname is mandatory somewhere else (where we cannot change it to by-partlabel) I'll push those changes to meta-smartphone tomorrow Mar 28 22:22:54 and revert by-name changes from sumo Mar 28 22:33:46 still don't understand why the find /dev/ doesn't work in init.sh, but at least adb works so we can debug more from there if needed Mar 28 22:34:02 I'll apply the same debug and changes to all my branches for next build over night **** ENDING LOGGING AT Fri Mar 29 02:59:57 2019