**** BEGIN LOGGING AT Sat Apr 10 02:59:58 2021 Apr 10 06:42:05 JaMa, yes, let me fix this. Apr 10 07:03:52 moving unstable-sstate to /media/ra_build_share/luneos-shared/unstable-sstate Apr 10 07:03:57 then I'll symlink that... Apr 10 07:04:04 then I'll look at how you have testing setup. Apr 10 07:04:34 from Bonaire you have permission to setup additional spaces you need. Apr 10 07:04:40 (jenkins user) Apr 10 07:18:13 ka6sox: yes it can wait for noday Apr 10 07:18:18 monday Apr 10 07:18:29 I'm actaully working it now Apr 10 07:18:46 is there also a luneos-testing-sstate? Apr 10 07:20:51 no testing and stable both use the same luneos-shared/sstate-cache Apr 10 07:21:00 unstable is intentionally separate Apr 10 07:21:10 downloads are shared between all 3 Apr 10 07:21:38 okay I have created a ra_build_shared/luneos-shared/unstable-sstate-cache Apr 10 07:21:58 and once its rsync'd I'll symlink it Apr 10 07:23:15 its about 2/3 done now Apr 10 07:24:30 thanks, I can create the symlink when it's done if you need to be somewhere else Apr 10 07:25:19 only heading for bed... Apr 10 07:25:26 but I will fiinsh this first. Apr 10 07:27:17 you shouldn't run out of space with ra_build_share as there is 7TB lett out of 10 and I'm cleaning up oe_sstated periodically. Apr 10 07:30:46 thanks tom Apr 10 07:31:07 khem doesn't use the jenkins job to clean it? Apr 10 07:33:00 looks like he does, but not with separate jenkins, maybe it doesn't work well in http://jenkins.nas-admin.org/job/ftbuild1_oe_world_workspace-cleanup/6/console Apr 10 07:33:39 it hasn't so far. Apr 10 07:33:59 I need to tweak it..normally I just rm -rf the branch. Apr 10 07:34:07 from the filer (less overhead) Apr 10 07:34:25 ah it was triggered last year for last time, nevm, so he doesn't use it Apr 10 07:35:10 eventually I intend to move wop-build to nao_feed_share so that all feeds are using that 4TB space...I have other feeds thawt will bve going there as well. Apr 10 07:35:20 yeah it takes hour+ to cleanup our sstate as well Apr 10 07:36:30 ra_build_share is internal stuff (sstate/downloads/etc) and nao_feed_share is for public consumption. Apr 10 07:36:42 are you generating tarballs for releases? Apr 10 07:49:49 JaMa, I am done with the rsync. Apr 10 07:49:58 files are owned by jenkins:jenkins Apr 10 07:50:19 can you delete the local unstable-sstate-cache on Bonaire and creatte the symlink? Apr 10 07:50:27 I'm past my expiration time. Apr 10 07:51:41 the sstate for unstable is at ra_build_share/luneos-shared/unstable-sstate-cache Apr 10 07:53:09 let me know if I need to do it later..G'nite Apr 10 07:54:14 Alive ! It's aliiive ! Apr 10 07:54:23 My Touchpad just got better :) Apr 10 07:54:35 Morning Tofe....Nite! Apr 10 07:54:45 ka6sox: gn8 ! Apr 10 07:59:19 ka6sox: we're releasing whatever IMAGE_FSTYPES are specified for each MACHINE with jenkins-job.sh removing some intermediate formats and other files we don't want to share, but generally it's vmdk tarball for qemu* and package.zip for android devices + e.g. pinpehone has .bmap as well Apr 10 07:59:41 ka6sox: gnight Apr 10 08:02:30 symlink done, builds re-triggered Apr 10 10:24:35 Herrie: "mount: mounting /android-rootfs/system on /android-system failed: No such file or directory" is it really a rootfs image ?... Apr 10 10:25:26 nope, it's a "system.img" Apr 10 10:25:46 Missing some flags in BoardConfig.mk, I guess Apr 10 10:32:30 Tofe: I think I tried system-as-root but didn't build Apr 10 10:32:36 Or memory is faulty Apr 10 10:50:19 Herrie: well, not sure we'll have a choice: the system image here doesn't have any ramdisk, so it won't boot Apr 10 10:51:11 I really don't recall to be honest Apr 10 10:51:18 Tree is on my github Apr 10 10:51:36 yes, I'll have a look, I'll probably build it here too Apr 10 10:52:16 can you give me the manifest ? I'll put it on my tofee fork Apr 10 10:58:16 Tofe: Well seems I tried somehow: https://github.com/Herrie82/android_device_hp_tenderloin-common/commit/e15eff69f18958adf9ad0c93eff15d03d6a3c479 Apr 10 10:58:20 Let me find the manifest Apr 10 11:01:07 Tofe: https://bpa.st/447Q Apr 10 11:01:14 Just don't remember which one I built :( Apr 10 11:01:22 I tried Evervolv and DirtyUnicorn I think Apr 10 11:01:45 Thanks, I'll try on my side and see what it gives me Apr 10 12:04:43 JaMa/Tofe: https://github.com/webOS-ports/jenkins-jobs/pull/15 Apr 10 12:04:56 Moved Hammerhead build to Halium 9.0 and disabled Mido for now Apr 10 12:05:01 Need to rebuild it locally and check status Apr 10 12:05:13 I think I had it working with 3.18 kernel, but not with 4.9 but don't really recall Apr 10 13:59:52 Herrie: ok, you had an issue with selinux Apr 10 14:00:49 I'll try to fix that, I had a similar issue with hammerhead https://github.com/Tofee/android_device_lge_hammerhead/commit/5a7086365fcc14722938ff69ea1b803fd4efa7bf Apr 10 14:06:50 Tofe: I think my read only was due to issue on file system Apr 10 14:06:59 I fixed it in Tailor on webOS side at some point Apr 10 14:07:13 My /media/internal got read-only at some point during my trials Apr 10 14:07:24 But if you say it's SELinux, I trust you of course! Apr 10 14:14:04 mmh there absolutely no sepolicy defined in this repo Apr 10 14:14:19 maybe I should retry with DirtyUnicorn Apr 10 14:31:00 Tofe: Well I had that my partition somehow got damaged Apr 10 14:31:10 And got to read-only Apr 10 14:31:19 I used Tailor to fix the partition Apr 10 14:31:23 Not sure you have the same Apr 10 14:31:35 Just I couldn't write anything to /media/internal in terms of logs Apr 10 14:33:09 Herrie: no it's really an initramfs issue Apr 10 14:33:18 Ah OK Apr 10 14:40:21 I now have a .img which look like a root image Apr 10 14:41:16 probably still wildly incorrect, but let's give it a try Apr 10 14:42:05 Tofe: Don't we need the skip initramfs patch? Apr 10 14:42:11 Like we have for other targets? Apr 10 14:42:15 I might have forgotten that one Apr 10 14:42:28 no, should be fine with good ol' uboot Apr 10 14:43:10 I.e.: https://github.com/Herrie82/android_kernel_xiaomi_msm8998/commit/917d5d898766594acef1bffc2f661db06032863b Apr 10 14:43:14 Tofe: OK Apr 10 14:43:20 Just thinking out loud here Apr 10 14:43:41 at least, it did work this mornng Apr 10 14:45:54 fixing sepolicy is a nightmare, I just don't know how it works :p Apr 10 14:48:18 ok, initrd looks a bit better Apr 10 14:49:35 looks like some "storaged" process is making the kernel crash Apr 10 14:49:54 FAT-fs (dm-6): error, fat_free_clusters: deleting FAT entry beyond EOF Apr 10 14:50:15 probably related to your /media/internal going read-only Apr 10 14:50:21 FAT-fs (dm-6): Filesystem has been set read-only Apr 10 14:51:34 storaged being an android service, interesting Apr 10 14:52:09 my guess is that it doesn't like the container, and tries to access some lvm devices a bit brutally Apr 10 14:52:28 or some conflict with our LVM, possible too Apr 10 14:52:47 JaMa, all good now? Apr 10 15:02:35 Herrie: in a couple of minutes, you'll find halium-luneos-9.0-20210410-1-tenderloin.tar.bz2 on our builds server Apr 10 15:02:49 (let's say 6min :) ) Apr 10 15:03:16 I'll prepare a PR for my modifs Apr 10 15:05:50 Tofe: Great Apr 10 15:19:36 ka6sox: yes, good AFAIK Apr 10 15:20:18 in the next months we are starting a test programme with icecc. Apr 10 15:47:33 Herrie: https://github.com/Herrie82/android_device_hp_tenderloin-common/pull/1 Apr 10 15:49:58 Tofe: Ah thnx! Apr 10 15:50:06 Do you get anything more now in terms of booting? Apr 10 15:51:07 I'm going until the kernel crash Apr 10 15:51:22 I guess we could see more by deactivating the android-system service Apr 10 15:54:16 mmmh nope, still crashing, must be something else Apr 10 15:55:02 I still see "type=1701 audit(1608139536.219:5): auid=4294967295 uid=0 gid=0 ses=4294967295 pid=912 comm="storaged" reason="memory violation" sig=5" Apr 10 15:55:14 but android shouldn't have been started... Apr 10 15:55:47 storaged = webOS/LuneOS side already Apr 10 15:56:21 oh, ok, that explains a bit Apr 10 15:57:03 right, /usr/sbin/storaged Apr 10 15:57:42 I'll also deactivate its service then Apr 10 16:00:00 But that shouldn't be fatal normally Apr 10 16:00:26 depends if there's a flaw in the kernel Apr 10 16:02:04 "BUG: spinlock recursion on CPU#1, systemd/1" damn Apr 10 16:10:21 Tofe: I've seen this one too Apr 10 16:10:29 It could be the same issue you had on Hammerhead somehow? Apr 10 16:11:24 on hammerhead it was clearly related to loopback network Apr 10 16:11:31 Ah OK Apr 10 16:11:42 Which kernel are you using? Apr 10 16:12:06 yours; let me paste dmesg Apr 10 16:13:02 https://bpa.st/VP6A Apr 10 16:14:35 Tofe: well it says something about read only: [ 37.256408] FAT-fs (dm-6): Filesystem has been set read-only Apr 10 16:14:43 COuld be that FS got read only Apr 10 16:14:50 Also says: [ 61.280364] 8058_l10: Failed to create debugfs directory Apr 10 16:15:05 yes, but dm-6 seems to be android/auto, so quite useless Apr 10 16:15:14 Ah ok Apr 10 16:16:08 I don't understand where that comes from Apr 10 16:16:17 for the spinlock I mean Apr 10 16:18:07 Tofe: I found something for Exynos, where it seems to be related to serial port: https://linux.kernel.narkive.com/1dRv9hga/patch-arm-dts-exynos4-add-dma-support-for-serial-ports Apr 10 16:22:39 maybe... it's also frustrating that we don't have a usefull backtrace in our case. It looks like it comes from a syscall, so not so useful Apr 10 16:24:21 Tofe: Maybe enable CONFIG_DEBUG_SPINLOCK? https://cateee.net/lkddb/web-lkddb/DEBUG_SPINLOCK.html Apr 10 16:24:32 "Say Y here and build SMP to catch missing spinlock initialization and certain other kinds of spinlock errors commonly made. This is best used in conjunction with the NMI watchdog so that spinlock deadlocks are also debuggable." Apr 10 16:25:39 And maybe disable "CONFIG_WATCHDOG_NOWAYOUT" ? https://github.com/Herrie82/android_kernel_htc_msm8960-1/blob/ec64d94574097f2e9c6a541f7618734f96cbda72/arch/arm/configs/tenderloin_android_defconfig#L3644 Apr 10 16:26:35 maybe yes Apr 10 16:26:51 Just thinking out loud here, I'm always a bit trial & error with regards to this Apr 10 16:26:59 Or we could take 5.1 defconfig and add changes Apr 10 16:27:10 Differences seem quite significant probably Apr 10 16:35:48 That's what you did with Hammerhead, didn't you? Apr 10 16:37:09 yes; as it almost the same kernel, it can work Apr 10 16:38:27 Because I believe the defconfig changes between CM 12.1 and 9.0 were quite significant so could cause issues Apr 10 16:52:30 hammerhead is failing to build as well http://jenkins.nas-admin.org/job/LuneOS/view/halium/job/halium-luneos-9.0-build/11/console Apr 10 17:20:43 JaMa: I haven't built Hammerhead here locally yet Apr 10 17:20:44 Let me try that Apr 10 17:22:24 Should be quick enough Apr 10 17:22:30 Tofe did it at his end Apr 10 17:22:42 Sometime our kernel that works inside Yocto gives issues in Halium Apr 10 17:27:00 It's already @ 35% locally Apr 10 17:27:05 Should know in a few mins probably Apr 10 17:28:32 I notice my local build script (based on jenkins-job.sh) does: "build_device hammerhead lineage_hammerhead-userdebug" instead of "halium_build_device hammerhead aosp_hammerhead-userdebug" Apr 10 17:33:13 JaMa: Locally it succeeds Apr 10 17:34:14 JaMa: This is my build script: https://paste.ubuntu.com/p/gRSFy3M7Jx/ Apr 10 17:36:01 Seems very similar to what we have in jenkins-jobs Apr 10 17:42:02 yes it has to be lineage_hammerhead-userdebug Apr 10 18:00:15 Tofe/JaMa: Force pushed the change Apr 10 18:00:49 Kicked off a new try Apr 10 18:01:24 http://jenkins.nas-admin.org/job/LuneOS/view/halium/job/halium-luneos-9.0-build/12/console Apr 10 18:05:11 OK seems to continue now... 193/437 Apr 10 18:05:31 It can still fail at some point, but at least it will be further down the line :P Apr 10 18:08:48 So far so good, I guess we'll know in a few minutes Apr 10 18:33:43 Hammerhead OK, Rosy fails now :P Apr 10 18:38:36 drivers/staging/Kconfig:111: can't open file "drivers/staging/prima/Kconfig" ooh submodules, what a treat Apr 10 18:40:11 I have a kernel that builds with 5.1 defconfig (with a few tweaks, but less than 5) Apr 10 18:43:32 Same spinlock stuff. So I'll try the CONFIG_DEBUG_SPINLOCK and see if it helps Apr 10 18:56:31 nope, doesn't output anything more Apr 10 19:17:02 Herrie: found something, in fs/ioprio.c, looks like a merge mistake https://github.com/Herrie82/android_kernel_htc_msm8960/commits/halium-9.0/fs/ioprio.c look closely, there are now two times a "lock", and only once an unlock Apr 10 19:19:34 You mean: https://github.com/Herrie82/android_kernel_htc_msm8960/blob/halium-9.0/fs/ioprio.c#L147-L152 ? Apr 10 19:20:55 exactly Apr 10 19:21:04 yes, we can go a bit further Apr 10 19:22:39 Seems we don't have these commits in Hammerhead.... Apr 10 19:22:46 But Hammerhead of course is very old 3.4 Apr 10 19:22:55 it's a backport from 3.16 Apr 10 19:24:10 now I have a stable adb Apr 10 19:24:21 onto lxc now, which doesn't want to start Apr 10 19:24:41 Seems Onyx has the right one: https://github.com/shr-distribution/linux/blob/onyx/3.4.0/cm-14.1-los/fs/ioprio.c#L140-L154 Apr 10 19:24:50 Onyx is also 3.4 Apr 10 19:25:13 "cannot create directory '/var/lib/lxc/android/rootfs/build.prop'" ok, seems easy enough, let's see Apr 10 19:25:41 Yeah seems indeed a merge mistake Apr 10 19:27:14 I think I've seen this build.prop somewhere popping up in Halium channel in the past Apr 10 19:27:19 Should be easy enough to fix Apr 10 19:27:45 I think it's just that we are going the old way in the pre-start.sh script Apr 10 19:30:08 ah, here's the actual error: Script exec /var/lib/lxc/android/pre-start.sh android lxc pre-start produced output: mount: /mnt/vendor/persist: special device /dev/disk/by-partlabel/persist does not exist. Apr 10 19:32:21 We shouldn't be surprised that good old Tenderloin doesn't have a persist probably Apr 10 19:32:35 :) Apr 10 19:36:04 ah, library "/system/lib64/libselinux_stubs.so" not found Apr 10 19:36:17 I know the drill there... Apr 10 19:38:07 Tofe: Ah seems like solid progress :P Apr 10 19:39:52 and now I can push the new android image directly, it skips many steps Apr 10 19:41:13 Yeah and having ADB helps a lot Apr 10 19:46:58 luna-next[1332]: FATAL: 17:49:09.913: QPA-HWC: hw_get_module(HWC_HARDWARE_MODULE_ID, (const hw_module_t **)(&hwc_module)) in create returned -2 Apr 10 19:47:06 getting a bit nearer Apr 10 19:48:26 That could be the funny graphics driver stuff Apr 10 19:48:40 yes, it came to my mind too Apr 10 19:53:54 It's been a long time since I last looked into that part of libhybris Apr 10 19:55:05 Might this be something we need? https://github.com/mer-hybris/qt5-qpa-hwcomposer-plugin/commit/5f6b9d7b0796b8ef35a18b7600b5502909f6ab6d Apr 10 19:55:06 our ANDROID_HEADERS_DEFINES are different for tenderloin, but should it, now ? Apr 10 19:55:17 Tofe: Probably not anymore ? Apr 10 19:57:09 there isn't any QCOM_* or QTI_* in the android headers anymore... Apr 10 19:57:51 QCOM_DIRECTTRACK is gone for sure Apr 10 19:57:58 i checked that previously Apr 10 19:59:03 but the vendor gralloc hasn't been rebuilt... or maybe it has been ? or wrapped ? damn, I don't know :) Apr 10 20:07:38 of course, we're also missing stuff related to vibrator, memtrack and bluetooth; but I don't think it's fatal Apr 10 20:08:32 Tofe: On 5.1 we had some custom graphics stuff, we might need it on 9.0 as well? https://github.com/webOS-ports/android_hardware_qcom_display-caf/commits/tenderloin/halium-5.1 Apr 10 20:11:00 well, I'd like to find the exact repo used by DirtyUnicorns for tenderloin for Android 9.0 Apr 10 20:11:09 https://github.com/DirtyUnicorns/android_hardware_qcom_display/commits/p9x-caf ? Apr 10 20:13:03 maybe https://github.com/DirtyUnicorns/android_hardware_qcom_display/tree/p9x-caf-8960/libgralloc Apr 10 20:16:14 I don't see any weird #define there Apr 10 20:16:42 I'll leave it here for tonight Apr 10 20:21:54 if you want the same kernel as me, just remove https://github.com/Herrie82/android_kernel_htc_msm8960/blob/halium-9.0/fs/ioprio.c#L149 Apr 10 20:28:41 OK Apr 10 20:28:47 Also for tomorrow Apr 10 20:28:49 Heading to bed soon Apr 10 20:28:55 Didn't have great sleep last couple of nights ;) Apr 10 20:29:53 Tofe: And defconfig? Apr 10 22:15:29 Herrie: unchanged **** ENDING LOGGING AT Sun Apr 11 02:59:56 2021