**** BEGIN LOGGING AT Wed Dec 11 02:59:58 2019 Dec 11 07:26:27 when i send an email to the list i see: "Trevor Woerner via Lists.Yoctoproject.Org" as the sender (instead of just, say, "Trevor Woerner". is that how everyone sees their own messages? or is my email not configured correctly? Dec 11 07:26:31 halstead: ^^ Dec 11 07:27:36 tlwoerner: everyone will see their own messages with the from line rewritten. Dec 11 07:28:02 halstead: ok, thanks :-) Dec 11 07:28:37 Depending on the DMARC setting for the domain you'll see more addresses rewritten. Dec 11 07:33:16 tlwoerner: gm! Dec 11 07:34:06 tlwoerner: are you still around for a second? Dec 11 07:59:26 New news from stackoverflow: Do_compile error for libpam package: Angstrom build Dec 11 08:05:59 good morning Dec 11 10:11:59 ok, lets try 3.1 M1 rc5 :/ Dec 11 10:29:53 New news from stackoverflow: Listing SRC_URI of all packages/files needed to build Yocto image Dec 11 10:44:45 RP: if you leave out one specific word, then the mail is pretty funny: "How about extending uninative with more bits?" :) Dec 11 11:35:54 RP: with the sstate changes from yesterday I'm seeing: Exception: bb.process.ExecutionError: Execution of '/OE/build/oe-core/tmp-glibc/work/x86_64-linux/quilt-native/0.66-r0/temp/run.sstate_create_package.25465' failed with exit code 1: Dec 11 11:35:58 mktemp: failed to create file via template ‘/OE/build/oe-core/sstate-cache/universal/5b/sstate:quilt-native:x86_64-linux:0.66:r0:x86_64:3:5b5ddc2dc26dbbdaa89060f194e5a6b56a048a7a6ad3b71bd290bd528355e70d_populate_sysroot.tgz.XXXXXXXX’: No such file or directory Dec 11 11:36:10 in clean build, any idea what went wrong? Dec 11 11:37:29 only .siginfo files were created in sstate-cache Dec 11 11:51:34 JaMa: fix is in master I think Dec 11 11:51:59 JaMa: I messed up the last fix :( Dec 11 11:55:21 RP: I do have this one http://git.openembedded.org/openembedded-core/commit/?id=0af4dae84099e8632a9ea6a4afdbea2f232bb170 Dec 11 11:55:47 JaMa: and its still breaking? :( Dec 11 11:56:42 JaMa: the patch is messed up, again. The mkdir needs to be about 8 lines higher :( Dec 11 11:56:53 yes, here is the trace if you can spot the code path leading to this https://paste.ubuntu.com/p/4TZP5w6dTN/ Dec 11 11:57:01 ok, let me check Dec 11 11:57:34 JaMa: sorry, these changes were meant to be straight forward small tweaks and have turned into a disaster ;( Dec 11 11:58:09 ah right, above mktemp Dec 11 11:58:25 strange that it's failing only on some hosts it seems Dec 11 12:00:18 JaMa: depends how many of the sstate dirs are present? Dec 11 12:00:23 JaMa: I've put a fix in master Dec 11 12:00:44 both builds were clean without sstate-cache Dec 11 12:01:07 and mktemp fails without existing directory on the builder where the build itself doesn't fail as well Dec 11 12:01:46 might be that only my local build where it was failing has hashequiv enabled Dec 11 12:02:18 * RP stops rc5 and goes to rc6 Dec 11 12:08:51 disaster? i hear disaster? Dec 11 12:16:11 RP: btw: that mkdir is using spaces for indentation and other lines around are tabs Dec 11 12:19:41 whitespace fix http://git.openembedded.org/openembedded-core-contrib/commit/?h=jansa/master&id=dd8620447448df890799e63a8cc300b44d99b3f4 Dec 11 12:21:19 anyone have experience making recipes for npm projects? Dec 11 12:24:15 stuom1: many have, and it universally considered a pain in the .... not only there, about everywhere. Dec 11 12:25:07 in my recipe (made with devtool) I have PV = "0.1.0+git${SRCPV}" but I get npm error for not finding file '/path/to/project-0.1.0+git999.tgz' because it is called '/path/to/project-0.1.0.tgz'. Where is that 999 coming to SRCPV and how to get it match Dec 11 12:25:38 LetoThe2nd oh good, im not alone Dec 11 12:27:36 I get it to build, if I remove the +git${SRCPV} part completely, but it builds only ONCE, next try fails with completely new problems, so I try to tackle them one at a time Dec 11 12:33:58 JaMa: ahhrg, right :/ Dec 11 12:34:15 with PV = "0.1.0" I can build it once using "devtool build" but if I try second time without any changes, or try to bitbake an image, I get "npm ERR! Cannot read property 'replace' of null" Dec 11 12:42:56 LetoThe2nd: hey! it was off to bed seconds after i wrote that; i didn't even see the DMARC reply :-) Dec 11 12:43:50 tlwoerner: hehe Dec 11 12:44:38 tlwoerner: hope you had a good night then! Dec 11 12:45:14 LetoThe2nd: it was a little shorter than i would have liked; i'll probably need a nap this afternoon ;-) Dec 11 12:46:06 tlwoerner: happends, that! just had one remark, and one idea/question. Dec 11 12:47:17 tlwoerner: remark: once you find (or even create) a good outline on the topic you put on the ML, i'd be glad to see it! Dec 11 12:50:51 does anyone know how to increase loglevel or verbosity for qemu-wrapper? I'm trying to give more information on that gobject-introspection + icecc bug on >= warrior and khem asked for more info Dec 11 12:51:42 LetoThe2nd: sounds good. it's not enough to simply read a recipe top-to-bottom, one has to consider *when* various parts (e.g. tasks) will run. it's like reading VHDL or verilog: the order doesn't matter! Dec 11 12:53:12 tlwoerner: the other is, like an idea. what do you think about preparing a bootable thumb drive that brings a fully prepared yocto/oe setup for a specific board, including hot sstate and all? Dec 11 12:54:10 i've been looking into ubuntu-based stuff here, but no real clue yet. Dec 11 12:56:43 i mean, it would be rather huge, hard to make it downloadable - but not exactly unheard of, but drives are cheap and shipping them trough mail is easy. Dec 11 13:00:30 sneakernet! or carrier pigeon Dec 11 13:00:38 bth! Dec 11 13:03:15 a long time ago ndec and myself were giving a tutorial at a Linaro Connect (in Dublin, if I remember correctly?) and we brought a thumb drive with a hot cache, sstate-cache, tmp etc. the only *gotcha* is that the host distro version has to match for the sstate-cache to work Dec 11 13:04:11 i think this was in the days before the amazing build accounts halstead sets up for devdays Dec 11 13:04:40 we handed out the drives and people copied them to their machines before starting the tutorial Dec 11 13:05:45 in sstate-cache is a host-distro-specific directory, if the host distro version doesn't match, the hot sstate is ignored and the machine ends up building everything from scratch Dec 11 13:06:21 but you can take the same sstate-cache directory to several hosts and the directory will contain multiple host-distro-specific directories Dec 11 13:06:27 tlwoerner: eactly, there's still a couple of gotches. Dec 11 13:08:59 You could also maybe do builds in a docker container, to make sure "host distro" is the same regardless of the users distro choice. Dec 11 13:10:50 erbo: and ship a 50GB docker container? Dec 11 13:11:50 i'm rather envisioning a setup that boots, and has the absolutely highest chances of supporting one target instantly. without downloads, without big rebuilds. Dec 11 13:13:42 tlwoerner: uninative helps with that, doesn't it? Dec 11 13:14:18 LetoThe2nd: ooh, i hadn't thought of making it bootable. that would solve the "host distro version" matching thing. but would mean you'd need drives for everyone in the class (which isn't too onerous) Dec 11 13:14:35 tlwoerner: i'm not thinking about classes. Dec 11 13:15:26 JaMa: isn't uninative about having the same host (native) tools? uninative doesn't setup sstate does it? Dec 11 13:16:54 LetoThe2nd: I usually bind mount the sstate-dirs, downloads, meta-data and tmp-dir etc into the container. So that the container doesn't ever change, it's basically just a poky-container with a few extra packages for conveniece. Dec 11 13:17:14 But sure, a bootable USB solves it too Dec 11 13:17:35 erbo: yeah. we do that too, and its all nice and such for people who know their way. Dec 11 13:18:02 i want something that i can give to somebody who literally has *no* *clue* Dec 11 13:18:20 Yeah then adding a layer of docker might not be the best thing to do :) Dec 11 13:19:24 bingo Dec 11 13:19:42 anything that requires manual interaction/setup is too much Dec 11 13:27:18 LetoThe2nd: https://dilbert.com/strip/1995-06-24 Dec 11 13:30:18 tlwoerner: yeah Dec 11 13:47:24 Hello there! I'm trying out YP through the use of Build Appliance.. I then ran a "bitbake core-image-minimal" as a simple test and ran into an error while building /home/builder/poky/meta/recipes-devtools/gcc/gcc-cross_9.2.bb.. well, it seems it does not build out of the box.. where should I look into, first ? Dec 11 13:47:42 build appliace? Dec 11 13:49:46 LetoThe2nd: yes, Build Appliance - Zeus 3.0 Dec 11 13:50:09 PaowZ_: can you pointme to your source? Dec 11 13:50:24 The Build Appliance is a virtual machine which enables you to build and boot a custom embedded Linux image with the Yocto Project using a non-Linux development system Dec 11 13:50:28 https://www.yoctoproject.org/software-overview/downloads/ Dec 11 13:51:06 I just ran that image into VirtualBox Dec 11 13:52:20 you see me surprised that the build appliance even is seeing new releases Dec 11 13:52:54 new package releases, you mean ? Dec 11 13:53:03 new yocto releases, i mean Dec 11 13:53:03 PaowZ_: what was the error? Dec 11 13:55:37 RP: https://snipboard.io/wQByfq.jpg Dec 11 13:56:52 PaowZ_: if you "bitbake gcc-cross-x86_64 -c clean" and rerun the build do you get the same error? Dec 11 13:57:56 by "rerun the build", there is a specific command or I just "bitbake core-image..." ? Dec 11 13:58:31 I'm cleaning the way you advise, though.. Dec 11 13:58:50 PaowZ_: just bitbake core-image... again Dec 11 13:59:18 ok.. I keep you posted.. thanks.. Dec 11 13:59:40 PaowZ_: its unusual for gcc-cross to break that like. The times I've seen a VM do this before were often memory issues on the system :( Dec 11 14:00:09 PaowZ_: if it does exactly the same thing again its probably not memory, if it breaks randomly somewhere else each time, it probably is Dec 11 14:00:34 PaowZ_: but its the best place to start in figuring out what is wrong... Dec 11 14:01:31 interesting point, RP.. here the settings for the VM: 4cores, 2048MB for ram and 54GB for disk.. Dec 11 14:02:20 aside, I have IO-APIC enabled and PAE disabled (don't need it, I think..) Dec 11 14:02:35 2048mb ram is a serious bummer Dec 11 14:02:51 too short ? Dec 11 14:03:14 yup Dec 11 14:03:31 ok.. well.. I gonna change the settings for upper amount.. Dec 11 14:03:50 I keep you posted, guys, thanks for the responsiveness :) Dec 11 14:04:08 it hopefully shouldn't hit that hard, but "GB including the whole building host system is really, really tiny :P Dec 11 14:05:00 2GB is really short in my guess, when it comes to build toolchain, indeed.. I should have thought about it.. Dec 11 14:07:52 PaowZ_: I'm more worried about whether there is an issue with the underlying memory hardware or the virtualisation mechanism. We've seen both give errors a bit like that... Dec 11 14:08:13 The virt issue was many years ago when it was much newer technology Dec 11 14:10:01 this used to be true, for virtualization mechanism, back then.. but now it tends to be quite reliable Dec 11 14:10:48 PaowZ_: see how the build goes, that should give a hint Dec 11 14:11:04 yup.. it's currently ongoing.. Dec 11 14:11:16 thanks to the cache.. Dec 11 14:12:39 PaowZ_: has it gotten past gcc-cross? Dec 11 14:22:16 RP: currently being built :) Dec 11 14:28:54 tlwoerner: maybe I misunderstood what you meant with "sstate-cache is a host-distro-specific directory, if the host distro version doesn't match, the hot sstate is ignored" but with uninative the ssate isn't specific to host distro and the path to sstate-cache or sstate mirror isn't included in the signatures, so it should be used just fine (as long as the metadata are matching) Dec 11 14:29:38 JaMa: ooh, i didn't know that! interesting Dec 11 14:30:01 tlwoerner: without uninative all native sstate will be in host distro specific LSB directory inside sstate-cache and indeed not reused in different distros and because target recipes now include native signatures, it will make whole sstate-cache host-distro specific Dec 11 14:32:11 yes, that's what I've seen. i got around that by mounting the same sstate directory into different hosts and running each build on each host but with one sstate. it inflated the sstate, but then ensured the cache was "hot" for anyone using it on their machine Dec 11 14:32:28 but your suggestion would be much better. noted! thanks :-) Dec 11 14:35:46 RP: do_compile ok for gcc-cross-x86 Dec 11 14:41:22 PaowZ_: I'd probably run memtest on the hardware that VM is running on Dec 11 14:41:39 PaowZ_: also possible it was some kind of parallel make race but less likely Dec 11 14:42:17 I'm wondering, what's the rationale behind having MACHINEOVERRIDES loosely set to MACHINE, forcing us to use prepends. We're leveraging this prepend stuff in some of our includes as well, but I discovered today that we should put those MACINESOVERRIDES =. *before* all `require` prepending to [e=13]pike [r=14]NickServ [t=15]#brootlin [y=16]alex [u=17]#guy [i=18]NamNam [o=19]atenart [20]paulk-leonov Dec 11 14:42:30 ugh Dec 11 14:42:56 I'm wondering, what's the rationale behind having MACHINEOVERRIDES loosely set to MACHINE, forcing us to use prepends. We're leveraging this prepend stuff in some of our includes as well, but I discovered today that we should put those MACHINESOVERRIDES =. *before* all `require` prepending to the said variable Dec 11 14:42:59 RP: Using "top" command I could estimate how heavy the threads build are.. It really needed 8GB at least.. Dec 11 14:43:39 qschulz: MACHINEOVERRIDES is what is added to OVERRIDES, not MACHINE Dec 11 14:43:56 PaowZ_: right, but that should have given an OOM error, not an assembler failure Dec 11 14:46:09 RP: MACHINE is in MACHINEOVERRIDES: http://git.yoctoproject.org/cgit.cgi/poky/tree/meta/conf/bitbake.conf#n740 Dec 11 14:46:46 qschulz: exactly, so what is your question? Dec 11 14:47:10 it has a default, a machine can override it to what it needs if the default doesn't work? Dec 11 14:48:08 qschulz: I guess the answer you probably are looking for is some machines may want to override the whole thing and not have MACHINE in there Dec 11 14:52:13 RP: In my mind, one would absolutely want MACHINE to be the rightmost part of MACHINEOVERRIDES Dec 11 14:52:37 RP: "god knows.." ;) Dec 11 14:53:05 qschulz: not if you're making some clone of a machine to change one setting and want it to otherwise behave as the original Dec 11 14:53:16 qschulz: we even test that Dec 11 14:53:30 RP: we require the original machine conf file for that Dec 11 14:53:57 and MACHINEOVERRIDES the name of the required machine conf file Dec 11 14:54:09 which means we need this MACHINEOVERRIDES before the require Dec 11 14:55:05 I'm not saying it does not work. It just feels weird to me for this particular variable to "be forced" to have it prepended before doing requires Dec 11 14:55:36 qschulz: well, I'm saying that people do use it like I describe and its very hard to undo an append/prepend so this is why its like this Dec 11 14:56:32 qschulz: I suspect its expected that most machines would just force a new value and include MACHINE Dec 11 14:56:40 then no prepending required Dec 11 14:59:38 RP: that means that we need to force MACHINEOVERRIDES for each similar machine, error-prone if you have many different includes and MACHINEOVERRIDES. Dec 11 15:00:09 qschulz: so build it carefully with prepends and appends Dec 11 15:00:12 RP: Is there any known use case for not wanting MACHINE in MACHINEOVERRIDES? Dec 11 15:00:23 qschulz: yes, I described it above Dec 11 15:00:58 qschulz: I create a machine called "qemux86copy", I don't want "qemux86copy" in overrides Dec 11 15:01:46 qschulz: mentioned in meta/lib/oeqa/selftest/cases/sstatetests.py and used by several people as a way of testing various pieces of the system Dec 11 15:01:57 RP: if we don;t leverage qemux86copy anywhere, it does not hurt to have it MACHINEOVERRIDES Dec 11 15:02:06 RP: ok, will read. Thanks ,anyway was just being curious :) Dec 11 15:02:29 qschulz: extra overrides slow down the datastore and have a habit of being dangerous IMO Dec 11 15:02:45 * RP is suitably scared of overrides but I know how the code works :( Dec 11 15:03:33 qschulz: you are correct in that it could have been done differently however it was done this way and it has pros/cons Dec 11 15:03:48 well, I'm scared too now (again) because I thought we already fixed our disordered MACHINEOVERRIDES but no :) forgot this one Dec 11 15:05:47 RP: I just wanted to hear if there was a very specific use case which warranted this behavior or if it just turned out to be the chosen implementation. I.e. I wanted to know if I should say to people in my company "it is how it is" or "it is like that because of..." Dec 11 15:06:23 your qemu example looks like a very specific use case so I'll read that file. Thanks for having taken the time to answer me, much appreciated Dec 11 15:06:23 qschulz: the principle use case is to allow overriding when MACHINE isn't what you want in overrides Dec 11 15:06:52 changing an append/prepend is near impossible so that is why we don't use them here Dec 11 15:06:53 RP: understood, I was failing to see the "when MACHINE isn't what you want in overrides" use case Dec 11 15:07:30 RP: yeah and we don't want to use _remove as much as possible Dec 11 15:07:40 qschulz: I'd prefer never ;-) Dec 11 15:09:47 qschulz: using _remove in the middle of overrides construction would also be asking for trouble/performance problems Dec 11 15:12:36 qschulz: sorry, I think I'm feeling jaded today :/ Dec 11 15:13:18 * RP has an inbox of email about how performance sucks and we have all these other problems :/ Dec 11 15:13:40 RP: pipe to /dev/null and let people screw themselves :) Dec 11 15:14:24 LetoThe2nd: perhaps you could take care of bug triage this week :) Dec 11 15:15:58 RP: i can declare it as a yocto bof? Dec 11 15:23:16 LetoThe2nd: not really :) Dec 11 15:23:26 RP: no deal then. Dec 11 15:25:19 RP: Here's my PoC on how I think the AB hash equiv failure could be fixed: http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=jpew/reproducible&id=2030f68d109a1f7e9c3fd4635b5a95724e18ce22 I don't think it's doing the "uncover" in the correct location, and it royal screws up the build stats reporting (but I have an idea for that). Dec 11 15:30:30 JPEW: Allowing multiple "normal" task exection? Dec 11 15:30:51 JPEW: that breaks a ton of assumptions all over the system :( Dec 11 15:31:10 RP: Ya, basically, but only when they were previously covered by setscene Dec 11 15:31:54 RP: e.g. were previously skipped because they were covered by setscene Dec 11 15:33:22 JPEW: ah, right. I thought the code could already handle this, we just disabled it due to problems? Dec 11 15:33:57 RP: Maybe... but that might of been before we had the "force tasks to be equivalent" Dec 11 15:34:12 RP: Without that, stuff was reexecuting all the time Dec 11 15:34:20 (unnecessarily) Dec 11 15:34:43 I think we need it back for the specific (and rare) case where the server can't give us the hash we want Dec 11 15:36:20 JPEW: I don't like it, we disabled this for a reason :/ Dec 11 16:08:08 RP: Hmm.... Ok. I'll keep thinking about it. Dec 11 16:12:21 JPEW: sorry, not trying to be negative, I appreciate the thought. I just worry about the problems we had with this previously. Perhaps it would be ok in the rare case this happens... Dec 11 16:12:32 JPEW: I'll ponder it some more too Dec 11 16:13:07 RP: It's fine :) Dec 11 16:13:35 * RP wonders why buildperf is breaking but only for release builds Dec 11 16:13:47 * RP suspects need to file another bug Dec 11 16:14:13 reproducibile builds failed the selftests for 3 out of 4 in the release build too :( Dec 11 16:17:35 RP: Do you have a log on that one? Dec 11 16:19:06 RP, the buildperf systems are Union Dec 11 16:19:37 RP, are the failures saving perf results? Dec 11 16:20:12 * armpit should just go look Dec 11 16:20:53 JPEW: selftests from https://autobuilder.yoctoproject.org/typhoon/#/builders/83/builds/570 Dec 11 16:21:00 JPEW: will be the perl issue I expect Dec 11 16:21:27 armpit: /home/pokybuild/yocto-worker/buildperf-ubuntu1604/yocto-autobuilder-helper/scripts/upload-error-reports: line 11: cd: /home/pokybuild/yocto-worker/buildperf-ubuntu1604/build/build/../: No such file or directory Dec 11 16:21:30 whatever that means Dec 11 16:21:50 two build dirs ?? Dec 11 16:22:38 armpit: normal Dec 11 16:22:45 armpit: one is buildbot, one is ours Dec 11 16:22:48 there have been a range of issue. your master-next build where lookin for /master Dec 11 16:23:03 I opened a bug on that one Dec 11 16:23:14 armpit: master-next would look for master to compare against Dec 11 16:23:30 it was looking for /master dir to save Dec 11 16:24:18 RP: One of them looks like libgcc, I don't think thats one of the usual perl ones Dec 11 16:24:41 JPEW: this is the problem with failures, that mask things Dec 11 16:24:47 s/that/they/ Dec 11 16:26:08 RP: Ya :( Dec 11 16:26:24 * JPEW really needs to get the diffoscope reporting into the test result Dec 11 16:30:16 should unzip be in HOSTTOOLS? (there is gunzip already), today I've started seeing do_unpack failures "/bin/sh: 1: unzip: not found Dec 11 16:40:08 JaMa: doesn't the fetcher code add an unzip-native depends (or zip-native or whatever?) Dec 11 16:40:43 JaMa: gzip is common for sources and our bootstrap, zip, less so on linux Dec 11 16:42:44 JaMa: yes, it does in base.bbclass Dec 11 16:56:46 Anyone know offhand if oe.package/package_manager/sdk/rootfs provide methods to produce the list of files for an installed package (ideally, all installed packages) in a rootfs/sysroot/sdk? Dec 11 16:57:04 looking to provide a recipe/files mapping in a sysroot in a non-package-manager-specific way Dec 11 17:02:18 how is TARGET_ARCH set? i cant find the history for it in `bitbake -e` Dec 11 17:10:40 hi all, i want to support two versions of the kernel, built from the same source, dev and rel Dec 11 17:11:19 i am wondering what's the best approach to this, without involving yocto-kernel-cache Dec 11 17:12:05 one thing that pops up in the manuals is to have config fragments that enable features... so with this scheme i'd have a stripped down rel kernel, with features adding the dev stuff on top Dec 11 17:12:17 marble_visions: do i understand correctly that you have your own kernel tree, that is you dont want/cannot use the default yocto one Dec 11 17:12:56 milloni: the kernel comes from a semi vendor, namely nxp, via the meta-fsl-bsp-* layers Dec 11 17:13:27 and i'm not sure how much of the upstream yocto kernel dev practices they follow, i need to doublecheck Dec 11 17:13:36 and you want to continue using their recipe, pointing to the same git commit, just use a different config? Dec 11 17:13:51 exactly, different configs* Dec 11 17:14:02 and i guess i would select the configs on the image-level Dec 11 17:14:30 although i'd like all of them to be build at the same time.. since there won't be any complex dependencies between kernels and userspace Dec 11 17:14:47 i suppose you dont want to keep two different configs Dec 11 17:15:22 because you want to avoid having an overlap between (and then having to be careful that they dont become out of sync)? Dec 11 17:15:22 actually i'd like to Dec 11 17:15:41 oh yeah.. Dec 11 17:15:57 what's the alternative? kernel feature selection? Dec 11 17:16:20 there are config "fragments" Dec 11 17:16:49 so you would have your defconfig for your rel build Dec 11 17:17:02 and then assume for your dev build you want to enable additional options Dec 11 17:17:14 you would put those options in a config fragment Dec 11 17:17:45 yep, that was the cleanest i could figure out Dec 11 17:18:05 but first, you want to be able to build rel and dev builds Dec 11 17:18:25 it seems to me that the best way to implement this is this Dec 11 17:19:08 you'd create two distro configs Dec 11 17:19:22 one for rel the other dev Dec 11 17:19:59 and that's going to tweak the options you want for your rel/dev images Dec 11 17:20:25 so it would be one distro, two images? Dec 11 17:20:45 two distros Dec 11 17:20:58 or actually, i suppose a two machine configs would make more sense Dec 11 17:21:13 so, do you understand the distinction between distro, machine, and image? Dec 11 17:21:48 i know they're distinct, but i don't know the details Dec 11 17:21:58 if it's in the manuals i will find it Dec 11 17:22:13 but if there is any arcane knowledge please share :) Dec 11 17:22:30 it's not really arcane knowledge, it's just a bit fuzzy... Dec 11 17:22:46 hopefully someone will correct me if im wrong, but here's my understanding Dec 11 17:23:12 first, image is defined by your image recipe, and it's just a regular recipe Dec 11 17:23:24 this is important and a lot of people miss this Dec 11 17:23:55 an image recipe defines how an image is built, it will use the artifacts built from other recipes etc, but the way it is implemented it's just a regular recipe Dec 11 17:24:19 this is important because it means any option set in the image recipe only affects the image recipe Dec 11 17:25:28 okay Dec 11 17:25:42 it's a common mistake to set let's say PREFERRED_VERSION_virtual/kernel in the image recipe and think it's going to affect the entire build Dec 11 17:26:16 so in your specific case we're going to leave the image reciple alone Dec 11 17:26:25 the other two set global build options Dec 11 17:26:34 i.e distro and machine Dec 11 17:26:52 which means that any option you set there will affect every recipe in the build Dec 11 17:27:17 this is perfect for making build-wide tweaks Dec 11 17:27:30 RP: looks like unzip issue might be caused by broken sstate for it (run out of disk space during the build last night), the interpretrer doesn't look right sysroots-components/x86_64/unzip-native/usr/bin/unzip: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /jenkins, for GNU/Linux 3.2.0, BuildID[sha1]=1c1124606c58a4945ee41e0a453376b9827bb5d7, stripped Dec 11 17:27:56 i said the concept is a bit fuzzy because to a degree it's a matter of convention what should go into distro and what should go into the machine Dec 11 17:28:10 both are build-wide Dec 11 17:28:18 make interpreter also doesn't look very complete sysroots-components/x86_64/make-native/usr/bin/make: ELF 64-bit LSB shared object, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/l, for GNU/Linux 3.2.0, BuildID[sha1]=9b552261fb11831ed9c357668beae0962cb967d4, stripped - actually it's in all native binaries I've checked now Dec 11 17:28:35 milloni: right Dec 11 17:28:58 marble_visions: but basically, i assume you've got your bsp for the board that you're using? Dec 11 17:29:04 does it define a machine config? Dec 11 17:30:11 it does Dec 11 17:30:29 so lets say your machine config is called machine.conf Dec 11 17:30:41 in your layer, i would create a new machine config Dec 11 17:30:56 conf/machine/myproject-dev.conf Dec 11 17:31:03 put Dec 11 17:31:16 require conf/machine/machine.conf Dec 11 17:31:33 and then below that, anything you want to set for you machine Dec 11 17:32:11 now we just need to figure out how to conditionally include a kernel config fragment based on the machine Dec 11 17:33:05 i guess the quick and dirty way would be in local.conf with the VAR[var] syntax? Dec 11 17:33:42 i see uboot does this with UBOOT_CONFIG[mfg/nand/etc] Dec 11 17:33:59 im not familair with that syntax Dec 11 17:34:02 i was thinking Dec 11 17:34:15 SRC_URI += "debug-config.cfg" Dec 11 17:34:19 sorry Dec 11 17:34:24 SRC_URI += "file://debug-config.cfg" Dec 11 17:34:37 expect you would only append that for your debug machine Dec 11 17:35:01 not sure but i think this may be the syntax: Dec 11 17:35:17 SRC_URI_myproject-dev += "file://debug-config.cfg" Dec 11 17:35:30 means only append to SRC_URI for MACHINE myproject-dev Dec 11 17:36:04 and i select MACHINE when setting up the build Dec 11 17:36:19 when sourcing yocto setup Dec 11 17:36:25 okay, that looks reasonable Dec 11 17:36:42 yeah, either in local.conf or by setting MACHINE env variable Dec 11 17:37:28 the sstate cache will be valid betwen dev and rel builds (most of it) Dec 11 17:38:08 so you won't have to rebuild too much, and you wont have to maintain separate layers for your debug build Dec 11 17:38:31 we went for a separate layer approach on our project and it kind of proved to be a pain in the ass :) Dec 11 17:38:59 i hear zeus has support for building multiple machines at the same time. not sure how that works but it makes it even simpler Dec 11 17:39:09 potentially you can build both with one command (?) Dec 11 17:39:13 zeus? Dec 11 17:39:19 the latest version of yocto Dec 11 17:39:22 ooh Dec 11 17:39:23 nice Dec 11 17:40:20 milloni: thanks for the info, it puts things into perspective Dec 11 17:40:30 i'll bet on two mach confs Dec 11 17:40:35 see where that takes me Dec 11 17:40:54 im happy to help, unfortunately i've not been able to test that out myself Dec 11 17:41:09 so i dont know how good it's going to be in practice Dec 11 17:41:38 i'd be interested to hear how it works out for oyu Dec 11 17:41:58 indeed, i might share in due time Dec 11 19:23:40 Looks like we've destroyed the performance of world builds with hashequiv :( Dec 11 19:23:59 https://autobuilder.yoctoproject.org/typhoon/#/builders/52/builds/1327 - 7 hours and task 1806 of 23848 Dec 11 19:31:13 huh, thats not really the desired effect Dec 11 19:32:56 Lots of "checkins sstate mirror object availibilty..." Dec 11 20:26:09 JPEW: I need to reproduce this under profiling, see where the bottleneck is. Has to be something silly like the log messages Dec 11 21:07:26 RP: dare i say, turn off hash equiv for m1? Dec 11 21:09:25 rburton: I'm starting to think we may have to :( Dec 11 21:12:52 Running setscene task 287565 of 11921 Dec 11 21:13:04 Hah! That's impressive Dec 11 21:13:50 JPEW: for 23000 tasks in total, very Dec 11 21:14:23 Oof Dec 11 21:16:14 I think its an accounting error rather than the real number of tasks it ran FWIW Dec 11 21:16:22 but still looks insane Dec 11 21:16:33 Hmm.. 23000 tasks doesn't seem all that large.. but ya, 287k of 23k is a problem.. ;) Dec 11 21:16:55 NOTE: Running setscene task 222166 of 11921 Dec 11 21:16:55 NOTE: Running setscene task 222246 of 11921 Dec 11 21:16:55 NOTE: Running setscene task 223302 of 11921 Dec 11 21:17:15 that is it incrementing Dec 11 21:17:22 lol Dec 11 21:17:41 JPEW: I think its the sstate rescans which are slow Dec 11 21:17:58 RP: Ya, that's what I was thinking. Dec 11 21:18:00 halstead: this comes back to the NAS not responding very quickly :( Dec 11 21:18:29 Why are there so many each time? Dec 11 21:18:49 JPEW: I'd guess a world build has lots of endpoints which means lots of rehashes Dec 11 21:19:52 RP: I'm not too familar with the "checking for object availablilty"; is it just looking for a single file for each message (e.g. just a LOOKUP) Dec 11 21:20:07 JPEW: in this case probably Dec 11 21:20:26 JPEW: that message comes from the hashvalidate function in sstate.bbclass Dec 11 21:20:59 RP, I think fixing this will require something other than optimizing the NAS. We could remove more objects or change the directory structure. Dec 11 21:21:03 JPEW: its from the update_scenequeue_data([tid] call in runqueue Dec 11 21:21:21 halstead: right, we talked about changing the directory structure Dec 11 21:22:14 halstead: I think ext4 as a filesystem may be faster for this btw but I know that isn't feasible Dec 11 21:22:21 RP, or I suppose we could have a dedicated mount point for sstate with caching enabled. Caching lead to races in the past so it's disabled. Dec 11 21:23:12 halstead: wish I could remember the details of the race issue. I wonder if we fixed that somehow] Dec 11 21:23:13 RP: Hmm, we could batch them all together in a single call to update_scenequeue_data. That would allow them to at least happen in parallel Dec 11 21:23:27 halstead: is it easy to turn on the caching to test? Dec 11 21:23:57 JPEW: yes, we should do that Dec 11 21:24:07 * JPEW writes a patch Dec 11 21:24:14 * RP clearly isn't thinking straight as that wasn't clear to me :/ Dec 11 21:24:33 RP, I don't know the details but there were multiple workers trying to write to the same file because it didn't exist in the cached dir list. Dec 11 21:25:52 halstead: easy to turn on? Dec 11 21:26:05 that has been reported by other users as well.. I'm wondering if there needs to be some kind of a lock file for things Dec 11 21:26:42 fray: you need to be specific about what exactly Dec 11 21:27:03 fray: I remember what michael is talking about in this case and I know we changed the code Dec 11 21:28:08 I wonder if we change the sstate write code to do a mv -n, how nfs copes with that. Is the -n a syscall/nfs preserved operation (no-clobber) Dec 11 21:28:22 sstate-cache directory Dec 11 21:28:24 is there an easy way to debug why 'apt-get install ...' fails with 'unmet dependencies' in my do_rootfs task for my image? Dec 11 21:28:45 I've been told more then once at the ELC / ELC-E in the last 3 years that people are seeing multiple writes when the original didn't exist.. often nearly at the same time Dec 11 21:29:27 (lesser I've also been told that about the dwonloads directory.. but that was before we fixed a bunch of lock files prior to Zeus) Dec 11 21:29:28 RP, requires remounting nfs. Simple to do between builds. Dec 11 21:30:38 halstead: server or client side? Dec 11 21:30:49 halstead: client I'd assume? Dec 11 21:30:56 RP client, yep Dec 11 21:31:15 halstead: could we remount mid build, see if it speeds up? Dec 11 21:31:40 risk taking is a skill at times, right? :) Dec 11 21:32:23 looks like we'd want the RENAME_NOREPLACE flag of renameat2 Dec 11 21:33:49 RP, We can try. Shall I go for it? Dec 11 21:35:25 halstead: yes, which machine/build are you going to change? Dec 11 21:35:33 halstead: just so I can watch the log output Dec 11 21:36:41 halstead: looks like the world builds are on centos7-ty-1 and debian8-ty-1 Dec 11 21:36:55 RP, I was going to push to all of them. Starting with just on is a better idea. :) Dec 11 21:37:06 halstead: yes, lets just try one Dec 11 21:37:46 RP, I'll change debian8-ty-1 Dec 11 21:38:08 halstead: ok Dec 11 21:43:00 Looking at the processes on the autobuilder I'm doubting this is NFS bound, its burning 95% cpu in cooker Dec 11 21:46:58 halstead: hmm, I think that has dropped cpu usage Dec 11 21:47:45 has anyone else tried to build a static libcrypt ? Dec 11 21:47:52 I'm trying and failing at the moment. Dec 11 21:48:06 RP, I've lazy unmounted /srv/autobuilder and mounted with the new options at the same location. Testing to see if it's working as expected. Dec 11 21:49:32 halstead: thanks, I think it may be helping, we'll see... Dec 11 21:49:49 zeddii: we disable static libraries so I assume you're disabling that? Dec 11 21:51:04 JPEW: I think its spending so much CPU in the rehashing code, the rest of runqueue is simply sitting waiting :( Dec 11 21:51:26 nfs is part of it but only a small part :( Dec 11 21:51:50 rp: hmm. that might be it, I've never had to worry about this before. so I've never looked. I'll see if I can find the right toggle. Dec 11 21:52:06 I'm just trying to build a static busybox and it is chaining to needing a static libcrypt .. Dec 11 21:52:19 I'll go check the initramfs layers to see if they have something for this already. Dec 11 21:53:17 zeddii: meta/conf/distro/include/no-static-libs.in Dec 11 21:53:22 +c Dec 11 21:53:59 ahah. will go poke around. Dec 11 21:55:10 zeddii: enabling those wastes so much more build time ;-) Dec 11 21:55:36 RP: Also possible Dec 11 21:55:40 yah. and I just want it for this one package + its depenendencies .. so I'll see if I can figure out another way Dec 11 21:56:03 zeddii: we used to special case sqlite-native for pseudo-native Dec 11 21:56:52 I can see about forcing it on for the libxcrypt build in my distro. Dec 11 21:59:52 JPEW: I think I can see how we could unroll the loops at the top of process_possible_migrations() too, at least to have one big recursive rehash rather than multiple Dec 11 22:03:41 RP, Caching is now in effect. Dec 11 22:04:04 RP: Ah, ya! thats probably causing the server to switch in and out of the fast streaming mode Dec 11 22:04:35 If we can make all the get_unihash() calls occur sequentianlly without intermixing report_unihash_equiv() that would be good Dec 11 22:05:44 JPEW: We're not seeing any _equiv() reports I see in the logs so that isn't the issue Dec 11 22:06:09 RP: I saw a few in the log I was looking at Dec 11 22:06:11 RP, I have the sstate cache clean disabled. Do we need a way to inform the hashserv about deleted objects before enabling it? Dec 11 22:06:36 Maybe not otfen enough to be the culprit though Dec 11 22:06:38 halstead: I think we've worked around it Dec 11 22:06:51 JPEW: pretty sure its runqueue's code given the pattern Dec 11 22:07:20 JPEW: lines 2264-2309 Dec 11 22:07:40 RP, Nice. So it's fine to turn back on now? I'll also start saving logs to deleted object names. Dec 11 22:07:56 halstead: yes, thanks Dec 11 22:08:18 JPEW: I think we can take 2275-2309 out of the top for loop Dec 11 22:08:34 but I doubt its the only problem Dec 11 22:09:31 JPEW: The local reproducer is to do a big build, then make a change to sstate.bbclass which doesn't change output but does force a complete rebuild where the output will always match Dec 11 22:09:53 * RP suspects an O2 or worse slowdown Dec 11 22:10:28 Ok, I'll give that a try. I think I have the batching of the sstate object check sorted out Dec 11 22:11:13 JPEW: I'm talking out loud btw, not saying you need to do this! :) Dec 11 22:11:48 I think I need to reproduce under bitbake -P and check where the time in the loop is spent Dec 11 22:12:02 RP: Ya, that's fine :) I figured moving the update out looked easy enough. I wasn't going to look at the rest yet Dec 11 22:12:30 could be as simple as commenting the logger.debug() Dec 11 22:12:45 we've had this before Dec 11 22:13:17 JPEW: fixing the mirror check would help a lot as it clearly crazy atm :) Dec 11 22:18:25 RP: Patch sent if you want to take a look. I have to go to a Christmas program now Dec 11 22:26:27 JPEW: thanks! Dec 11 22:32:00 JPEW: its the get_taskhash and get_unihash calls Dec 11 22:32:09 JPEW: I managed to get a profile Dec 11 22:56:45 JPEW: you won't believe what the problem is :/ Dec 11 23:14:31 RP: g'wan tell Dec 11 23:58:23 rburton_: well, it turned out to be complicated Dec 11 23:59:26 heh, he ran away :) Dec 12 00:00:35 http://git.yoctoproject.org/cgit.cgi/poky-contrib/commit/?h=rpurdie/t222&id=5d0281806dacbb28529001a837fa32a3f4c8bdf5 does fix it, the question is why Dec 12 00:00:55 and the reason is self.unitaskhashes is 192MB locally Dec 12 00:01:18 local access to that rather than abstracted via a function is much faster due to its size Dec 12 00:01:56 get_unihash gets called around 600,000 times in my tests :/ Dec 12 00:02:06 and that isn't a big build Dec 12 01:34:03 Is anyone familiar with the use of qemu build through yocto? I was wondering if it was possible to specify the exact qemu CPU to use. **** ENDING LOGGING AT Thu Dec 12 02:59:58 2019