**** BEGIN LOGGING AT Sat Feb 22 02:59:57 2020 Feb 22 06:57:01 New news from stackoverflow: Adding nativesdk-qt4-tools to my yocto SDK Feb 22 08:50:48 khem: I think the binutils change causes a selftest failure :/ Feb 22 09:07:29 khem: confirmed, the _x86-64 overridden line breaks oe-selftest -r sstatetests.SStateTests.test_sstate_32_64_same_hash Feb 22 10:58:20 how can I choose BRANCH based on MACHINE name in my own recipe? Feb 22 11:43:07 dv|2: BRANCH_machine1 = "A" BRANCH_machine2 = "B" and use ${BRANCH} ? Feb 22 11:43:25 RP, exactly! Feb 22 11:56:13 RP: I think I may have tracked down my extend_recipe_sysroot issue from yesterday. It was my mistake but a subtle one Feb 22 11:57:35 paulbarker: what was it out of interest? Feb 22 11:57:50 I had `do_image_wic[depends] += "u-boot-bbe:do_build"`, should have been `do_deploy` at the end there instead Feb 22 11:58:00 All works fine until you inherit rm_work Feb 22 11:58:12 paulbarker: ah, I can imagine how it could break Feb 22 11:58:56 Then you get a graph like `u-boot-bbe:do_build` -> `u-boot-bbe:do_rm_work_all` -> `glibc-initial:do_rm_work` -> `glibc-initial:do_populate_sysroot` Feb 22 11:58:58 the dependency code is all rather interesting :/ Feb 22 11:59:13 And suddenly extend_recipe_sysroot is trying to add both glibc and glibc-initial Feb 22 12:00:04 paulbarker: which then breaks since the code which blocks both together then doesn't trigger Feb 22 12:00:19 Exactly Feb 22 12:01:13 I wonder if we can make it drop do_rm_work & do_rm_work_all in the "Create collapsed do_populate_sysroot -> do_populate_sysroot tree" section of the function Feb 22 12:02:23 paulbarker: I think there would be other implications of the issue so that probably would create more confusion than it could solve :/ Feb 22 12:03:09 Ok. Maybe a warning if we see do_build in the dependencies of a task like that? Feb 22 12:03:48 paulbarker: maybe. I think only do_build should depend upon do_build Feb 22 13:58:34 RP:what does that test do, compare target binutils or cross Feb 22 14:00:29 khem: compares binutils-native on a 32 and a 64 bit host Feb 22 14:00:35 khem: I have a fix in -next Feb 22 14:01:04 ah yes was reading the testcase Feb 22 14:02:09 khem: fix mailed Feb 22 14:02:23 as well as fix for zeddii's kernel change and a fix for the psplash race issue Feb 22 14:03:56 looks good Feb 22 14:04:25 I was thinking to enable-targets=all for all kind of binutils as well Feb 22 14:04:52 khem: i was wondering about that. Could we even drop binutils-cross in favour of binutils-native and just symlink? Feb 22 14:05:29 which would make binutils build a bit longer and perhaps bigger but will account for Feb 22 14:05:44 khem: we build -native anyway though? Feb 22 14:05:52 RP: in that case perhaps yes Feb 22 14:06:20 yeah native is not always built I think but we can if we think of dropping cross Feb 22 14:07:09 khem: it'd be worth measuring the size/time difference Feb 22 14:07:19 in clang thats what I do, build one and symlimk for cross but clang driver supports it so any issues we can fix, not sure about binutils case, we might have to try Feb 22 14:11:24 RP: another problem I am seeing is that when I build musl/glibc in same tmpdir for same machine, it deletes content from core2-32-yoe-linux-musl when build is happening for glibc while it decides to do sysroot cleanup Feb 22 14:11:44 I thought core2-32-yoe-linux-musl will be unknown to glibc build Feb 22 14:12:03 so it should not even touch *-*-musl* dirs, no ? Feb 22 14:12:40 khem: It depends. Which dir exactly? Feb 22 14:13:02 work/core2-32-yoe-linux-musl Feb 22 14:13:16 khem: there is code which records what a given MACHINE built last time, then if its no longer reachable in a new build it can remove it Feb 22 14:13:21 packages under above are cleaned Feb 22 14:13:26 when doing glibc build Feb 22 14:13:42 khem: I'm slightly surprised it would clean musl from glibc Feb 22 14:13:53 I am too, Feb 22 14:14:53 easy to reproduce, TCLIBC=musl bitbake core-image-minimal && bitbake core-image-minimal now check musl/ dir under work/core2-32-yoe-linux-musl its removed Feb 22 14:15:31 and its surgical about it, not whole core2-32-yoe-linux-musl is removed Feb 22 14:15:52 khem: have a look at sstate-control/index- Feb 22 14:16:14 khem: I suspect something odd is going on with the indexes Feb 22 14:16:26 khem: my local ones don't look right to me at a quick glance Feb 22 14:16:36 khem: if those indexes aren't right it will remove things when it shouldn't Feb 22 14:17:37 khem: we have this behaviour so things like switching sysvinit to systemd work Feb 22 14:17:57 khem: it is there to clean up systemd ipks from the feed if the systemd recipe is disabled for example Feb 22 14:18:10 khem: I think it will also trigger to remove musl if you switch to glibc and vice versa Feb 22 14:19:09 but musl is different tuple unlike system/sysvinit case Feb 22 14:19:23 RP: is the wayland build issue caused by ninja update by any chance? Feb 22 14:19:36 I see that index names dont encode TARTGET_OS Feb 22 14:19:55 kanavin_home: possibly. Not sure if its deterministic yet or not Feb 22 14:20:36 RP: what I see in the failure is that it's trying to build tests ahead of everything else Feb 22 14:20:54 kanavin_home: so its racing with broken dependency information? Feb 22 14:21:38 RP: looking into source code now Feb 22 14:22:21 kanavin_home: I haven't really looked into that one, I decided to file it as its out my area of knowledge Feb 22 14:22:33 RP: index-core2-32 index-machine-qemux86 index-qemux86 they should also have TARGET_OS in their name IMO Feb 22 14:22:37 RP: let me try to build it with new ninja Feb 22 14:22:44 kanavin_home: there were three other issues for me to be going on with :/ Feb 22 14:23:11 khem: I'm not sure they should Feb 22 14:23:44 khem: where do the files end up under tmp/deploy/ipk ? Feb 22 14:24:51 khem: I suspect if we change this code you'll get both glibc and musl ipks together Feb 22 14:25:08 hmm thinking about it from feeds point of view it seems you are right Feb 22 14:25:12 khem: so the code is actually doing the right thing and preventing you hurting yourself Feb 22 14:25:17 :) Feb 22 14:25:48 so best is to use different tmpdir ? Feb 22 14:26:06 like oe-core defaults? Feb 22 14:26:26 khem: I think we'd have more work to do to make parallel glibc/musl builds work in the same tmpdir Feb 22 14:26:54 It does "work" only because we have code which detects and tries to handle it Feb 22 14:27:07 but to do that it has to clean up lots of stuff Feb 22 14:27:49 yeah i see that especially when you do world builds one after another you can see it cleaning up for long long time Feb 22 14:28:11 khem: separate tmpdir is simpler Feb 22 14:29:21 RP: with combined tmpdir, I have been able to catch some allarch bugs though Feb 22 14:29:41 so its not all in waste, but I guess I should change my distro defaults Feb 22 14:29:57 I have bigger fish to fry Feb 22 14:30:24 khem: right, I do see some value in it but I don't think its top of the agenda Feb 22 14:30:47 khem: In some ways I'm surprised it works at all! :) Feb 22 14:31:11 it works because I have been using it :) Feb 22 14:31:20 RP: nope, it's not ninja :-/ and the dependencies look right Feb 22 14:31:35 locally the tests build after everything else Feb 22 14:32:49 kanavin_home: hmm, so some kind of race? :/ Feb 22 14:34:17 or a malformed ninja file Feb 22 14:37:52 RP: I ran ninja -j 100 and the race still doesn't show up :-/ Feb 22 14:40:46 kanavin_home: I wonder if we still have that builddir, would that help? Feb 22 14:41:34 RP: yes, particularly the generated ninja file, but everything else too so I can compare to local build Feb 22 14:42:47 kanavin_home: we have it, one second and I'll share Feb 22 14:44:01 kanavin_home: https://autobuilder.yocto.io/pub/repro-fail/wayland.tgz is the cut down version Feb 22 14:44:20 kanavin_home: creating wayland2.tgz with recipe-sysroot and everything Feb 22 14:44:30 (also now done) Feb 22 14:44:55 200MB vs 1MB :) Feb 22 14:47:00 RP: right, let me see Feb 22 14:52:39 hmm, does anyone know systemd unit files? We're seeing psplash-systemd[134]: Error unable to open fifo even with my patch in -next :( Feb 22 14:56:32 RP: https://github.com/wayland-project/wayland/commit/0fc00fff3015edb70b865f126dc8f4c258fe2f56 Feb 22 14:56:45 I'll get some food now, then prepare the patch :) Feb 22 14:57:28 kanavin_home: cool, thanks! Feb 22 14:59:47 hmm, the dependencies just look wrong to me :/ Feb 22 15:08:16 RP which dependencies? Feb 22 15:14:07 kanavin_home: sorry, for psplash systemd Feb 22 15:31:04 RP patch for wayland is on the list Feb 22 15:36:37 RP: for sysvinit psplash fifo is export PSPLASH_FIFO_DIR=/mnt/.psplash Feb 22 15:36:50 is it different for dir for systemd Feb 22 15:44:14 khem: it is, should be in /run for systemd Feb 22 15:44:32 kanavin_home: thanks Feb 22 15:54:35 github has (temporarily?) disabled archive downloads, just in case anyone needed a better reason to stop using them: https://status.gitlab.com/pages/5b36dc6502d06804c08349f7 Feb 22 16:08:15 smurray: gitlab or github? it sounds temporary to address a DoS attack? Feb 22 16:09:51 RP: sorry, indeed gitlab. Yeah, hopefully temporary Feb 22 16:10:03 RP: I posted that pre-coffee Feb 22 16:10:06 RP: I see perhaps add ConditionPathIsReadWrite=/run as well to deps Feb 22 16:10:29 khem: won't that mean it just skips the unit ? Feb 22 16:10:41 The ConditionPath stuff sounds one shot Feb 22 16:11:00 A path unit file with PathExists sounds better but means a path unit file afaict Feb 22 16:11:15 * RP isn't systemd literate though, what do I know Feb 22 16:12:22 RP: if that happens then we know the problem is that its mounted but not writable yet Feb 22 16:12:29 I can't remember if systemd does watches so ConditionPath will trigger at runtime. It does seem more likely to be one shot Feb 22 16:13:04 khem: the problem is more likely the other service hasn't created it yet Feb 22 16:13:32 smurray: I found it hard to tell with google but one shot seemed more likely from the impression I got Feb 22 16:14:20 RP: yep Feb 22 16:14:36 khem: we might get to drop a patch eventually: https://gcc.gnu.org/git/gitweb.cgi?p=gcc.git;a=commitdiff;h=d6f420d98126ac51396b89fbe287a32287cd92ed :) Feb 22 16:17:17 RP: saw that, my gcc10 patch (when upgraded past this commit) will drop it Feb 22 16:17:30 how many years it took I wonder Feb 22 16:17:54 khem: filed in 2013 Feb 22 16:18:12 heh, so almost 7 Feb 22 16:18:27 Jan 2013 so over 7 years to get a simple fix in Feb 22 16:19:20 contributing to clang is so easy, write a patch and push it to fabricator or now github pull request Feb 22 16:19:57 khem: I should really find the time to go through our gcc patches and see if we can do something with some of them Feb 22 16:20:19 on systemd thing you added RequiresMountsFor so that should take care of /run being mounted isnt it Feb 22 16:21:09 RP: most of them are config related and some are relocatable install related noneof them are ustream worthy Feb 22 16:21:15 khem: right, which makes me thing its a dependency issue Feb 22 16:21:36 khem: we should talk to upstream about relocatable install though Feb 22 16:21:47 khem: or see if we can do it better Feb 22 16:22:12 RP: some spelunking in systemd source confirms that indeed it's only on unit start Feb 22 16:23:07 smurray: thanks. Do you happen to know if there is a "service is started when file X exists" parameter for a unit file? Feb 22 16:23:16 RP: perhaps 8 or so might be interesting to upstream Feb 22 16:23:31 khem: that would be 8 less to carry Feb 22 16:23:55 RP: yeah but who will sign legal papers which they never respond back Feb 22 16:25:23 RP: as a prerequisite for starting, or do you mean something that indicates it has been started? Feb 22 16:26:00 smurray: basically service X creates file A and I want service Y to start when file A exists Feb 22 16:26:31 smurray: X=psplash-start and Y=psplash-systemd Feb 22 16:26:43 A=/run/psplash_fifo Feb 22 16:29:32 RP: hrm, I think that ConditionPathExists=A might work if Y has After=X in its unit file, since the conditions are only checked on start Feb 22 16:30:13 smurray: there can be a delay between exec(X) and A existing though Feb 22 16:30:31 the irony being I thought systemd was designed to handle these races Feb 22 16:31:25 RP: interesting. What type of service is psplash-start set as? Just simple? Feb 22 16:32:03 smurray: http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/recipes-core/psplash/files/psplash-start.service?h=master-next Feb 22 16:36:53 RP: I always have to reread the systemd unit docs for this stuff. I could see how that might race, as for "simple" units it considers it started as soon as it forks to run whatever's in ExecStart Feb 22 16:37:16 smurray: that is my worry Feb 22 16:37:48 RP: I've not looked at psplash in a long while, does it slap up the image and exit, or stick around? Feb 22 16:38:08 smurray: sticks around with progress bar, the fifo is used to drive that Feb 22 16:39:10 RP: if it daemonizes, then Type=forking is more correct, and may fix the race Feb 22 16:39:26 smurray: don't worry too much, I was just hoping there was someone systemd literate around who could say "oh, just do ABC" Feb 22 16:39:40 smurray: I'd have to check to see if it does Feb 22 16:40:43 smurray: doesn't deamonize :/ Feb 22 16:42:03 This isn't funny, every new build has more failures we haven't seen before or don't understand :( Feb 22 16:44:06 RP: yeah, that's going to be impossible to keep from racing unless psplash-start gets patched to use sd_notify, or perhaps there's a game that can be played with ExecStartPost in the psplash-start unit to wait for the fifo Feb 22 16:45:50 RP: ouch. I'm about to step out, but I'll take a look at Bugzilla when I get in later and see if there's something I can help try to debug tomorrow Feb 22 16:46:45 smurray: thanks. I need to file them Feb 22 16:47:00 smurray: some I can't reproduce and am not sure its worth filing :( Feb 22 16:47:16 some we're seeing for the second time so they;re going to be problems Feb 22 16:54:08 smurray: an interesting one is executable bits disappearing from files in /lib/systemd/systemd-XXX :/ Feb 22 17:01:29 RP: yikes Feb 22 17:06:41 smurray: https://bugzilla.yoctoproject.org/show_bug.cgi?id=13810 Feb 22 17:06:42 Bug 13810: normal, Undecided, ---, unassigned, NEW , systemd file executable bits disappearing Feb 22 17:39:24 RP: if the psplash thing is a blocker, I can whip up a patch to try adding a sd_notify call in psplash-start when building for systemd Feb 22 17:42:01 smurray: thanks. We're lucky with that one in that its not in master yet. I'm now torn on merging it as my last fix appears to be working even if it still theoretically has a race Feb 22 17:43:01 smurray: also wondering about a unit path file and PathExists to avoid messing too much with it, the sd_notify call is probably going to be a pain to do Feb 22 17:43:40 * RP is now staring at a basehash changed error and wondering if hashequiv breaks it Feb 22 17:44:18 RP: I’ll take a look tonight/tomorrow Feb 22 17:44:47 hash equiv can't break a basehash though, so I should stop being silly :) Feb 22 17:48:49 I wonder if this is actually an inconsistent logging issue Feb 22 17:49:56 so its not bash the hash then Feb 22 17:50:50 armpit: hashequiv is the new sstate for any bug we don't understand :) Feb 22 18:00:14 Not logging but maybe data leakage between tests Feb 22 22:27:18 What is "WKS_FILES"? Feb 22 22:27:38 it's complaining that it is not set appropriately, but I can't find documentation about it Feb 22 22:28:16 (I'm trying to make an *.ubi image) **** ENDING LOGGING AT Sun Feb 23 02:59:57 2020