**** BEGIN LOGGING AT Thu May 16 03:00:30 2019 **** BEGIN LOGGING AT Thu May 16 06:54:52 2019 May 16 07:26:20 what clean commands do people use? I often do rm -rf tmp/ because it never has issues and is pretty fast on my computer May 16 07:26:59 ernstp: depends on what i want to archieve May 16 07:27:54 I know deleting tmp/deploy/ can lead to issues because the recipies don't expect that... May 16 07:29:34 I saw rm -rf tmp/work/ suggested by someone, is that picked up correctly by the buildsystem? May 16 07:29:44 ernstp: well *why* are you cleaning? what do you want to archieve May 16 07:31:19 right! my Jenkins server right now nukes the whole tmp/, to be able to handle any kind of change May 16 07:32:04 but it's on the slow side so parsing recipes and setting up tools takes a lot of time every time May 16 07:32:40 so I was thinking if there was something in between that was a good compromise May 16 07:32:58 there's also tmp/stamps/ that's interesting... May 16 07:34:50 ernstp, have you seen rm_work.bbclass ? May 16 07:36:10 kroon: interesting... is that to save disk space mostly or? May 16 07:37:54 ernstp, yes May 16 07:38:18 ernstp, it wipes most of the build artefacts after a recipe build, but keeps log files around May 16 07:41:13 ernstp: the point is, what do you want to test? if you nuke tmp including sstate, then a new rerun actually will probably just exercise the signatures and image creation, but no source rebuilds May 16 07:42:09 ernstp: so if you want to make sure your ci exercises rebuilds of your applications, you either have to wipe sstate (globally), or bitbake -c cleansstate $YOURRECIPE (specifically) May 16 07:42:20 LetoThe2nd: you mean nuke tmp and save sstate? yes, and that's pretty close to what I want May 16 07:42:50 ernstp: nuking tmp and saving sstate exercises image creating, but no source rebuilds. May 16 07:43:04 LetoThe2nd: but let's say there's one recipe that's been updated May 16 07:43:37 LetoThe2nd: then you need to setup the native sysroot and the sysroot, from sstate. and that takes some time on a slow computer May 16 07:47:07 all the setscene stuff... May 16 07:49:39 ernstp: i would maybe agree in a dev workflow, but on a ci system? those are basically meant to check your work asynchronously anyways. May 16 07:50:57 LetoThe2nd: somewhat of a microoptimization yes... just incredibly starved on HW at the moment May 16 07:51:27 nuking tmp/ is extremely stable, so that's nice. May 16 07:51:32 ernstp: consider spinning up cloud instances :) May 16 07:51:45 no matter what kindof silly patches people throw at it... May 16 07:52:09 LetoThe2nd: I've heard about this cloud :-) May 16 07:55:19 "rm -rf tmp/stamps/" runs all the setscene like "rm -rf tmp/" but your recipe parsing cache seems to not be invalidated... May 16 08:01:32 hmm, are patches which reduce GPLv3 dependecies ok for poky? May 16 08:01:50 for example dropping GNU readline support from sqlite3? May 16 08:02:14 they have been fine in the past.. May 16 08:02:16 this kind of changes don't fit to meta-gplv2 either May 16 08:02:36 could there be gplv3 distro feature which could be disabled? May 16 08:02:42 there is a meta-gplv2 repository that may be used for some of these things, but in the main oe-core -- usually we want this implemented as an optional PKGCONFIG/PACKAGECONFIG setting.. May 16 08:03:25 If there were a gplv3 distro feature, it should be implemented in the meta-gplv2 layer IMHO. The general OE community is not interested in a GPLv3 limited or free system. However they are interested in reducing dependencies (or alternative dependencies) May 16 08:03:30 ah, so I could create readline PKGCONFIG setting and disable RDEPENDS based on that May 16 08:03:34 yes May 16 08:04:14 if reducing / changing dependencies meets your goals -- then I'd say oe-core. If it's specifically just reducing dependency on GPLv3, then it likely belongs in the meta-gplv2 layer May 16 08:04:55 currently meta-gplv2 provides old readline version so the changes don't fit there either.. May 16 08:05:39 then again the bbappends to drop readline dependency in sqlite3, lua etc are really trivial for me now May 16 08:05:40 even in that case, linking against readline is something that is frowned upon -- since it introduced a GPL 'library'. (vs LGPL). So that dependency drop is reasonable in any of the layers in my opinion.. May 16 08:06:10 ya, dropping readline dependencies belong in the originating layer in my opinion. This isn't a GPLv* issue, but a general dependency thing May 16 08:06:18 are you just dropping it, or replacing it with something like editline? May 16 08:06:32 atm just dropping it. May 16 08:06:46 ok.. ya, that makes sense for a PACKAGECONFIG switch to me May 16 08:33:33 mcfrisk: adding a PACKAGECONFIG for it certainly should be fine May 16 08:33:50 mcfrisk: we could then put an example set of PACKAGECONFIGs into meta-gplv2 May 16 09:28:35 Is there an easy way to mark a package as machine independent? The recipe has one package that includes a file depending on the machine, but the other package it produces is completely independent May 16 09:29:16 fl0v0: https://www.yoctoproject.org/docs/latest/mega-manual/mega-manual.html#ref-classes-allarch May 16 09:33:32 LetoThe2nd: the allarch class makes the whole recipe architecure independent if I understood correctly, but i just want to make one package machine independent. Maybe i have to create a own recipe then for the dependent package May 16 09:33:50 fl0v0: exacty. sounds like you want to split the recipe. May 16 11:23:02 RP: is DL_DIR safe to share over r/w NFS? I couldn't find a definitive answer. May 16 11:23:16 kanavin: yes May 16 11:23:26 kanavin: the autobuilder does it May 16 11:26:08 is there some documentation about what is published in http://sstate.yoctoproject.org/ ? and how (e.g. from which builds)? May 16 11:27:12 RP: thanks. I thought the autobuilder fetches things separately prior to starting the builds? May 16 11:37:37 kanavin: it once did, these days it only does a sanity check on what is there :/ May 16 11:47:55 the big issue with any sharing over NFS is that many NFS servers and clients are 'slightly broken'. So replacing files causes issues.. May 16 11:48:10 if you can do the file replacement in an atomic fashion, that usually resolves the problem except on the really broken systems May 16 11:48:59 doesn't bitbake fetcher locks handle that well enough? May 16 11:49:22 no.. because locking doesn't always work on NFS fileservers.. :P May 16 11:49:31 cause NFS is usually slightly broken May 16 11:49:48 none of this is anything wrong with OE/bitbake.. it's all due to NFS itself being horribly broken.. May 16 11:50:00 usually result is failures reading files (as they're being written to by other sessions) May 16 11:50:17 the only issue I've seen related to this in last 6+ years (with 20+ builders sharing NFS from quite old RHEL) is when the lock file wasn't removed when bitbake was forcibly killed at wrong moment (happended about 3 times) May 16 11:50:23 i.e. you start reading a file.. the file then changes and the NFS server pukes and you get the equivalent of an I/O error on the original client May 16 11:50:49 I have customers who see this all the time.. much of the time they are use netapps and such to serve NFS to Linux clients.. May 16 11:51:01 * JaMa surprised that RHEL 5.6 isn't broken enough for us to see this :) May 16 11:51:30 but I've even seen it Linux client to Linux client.. IF they have problems with this kind of things, I always recommend moving to an rsync 'push', with an http pull for mirrors... takes more local space, but seems to resolve the issues May 16 11:52:10 fray: we're quite careful about not changing files unless we have locks so file issues tend not to be a problem. We assume the locking works well enough for our use cases May 16 11:52:19 but also re-downloads the same thing many times when multiple builders are triggered for the same change at the same time (our typical scenario) May 16 11:52:33 ya, like I said.. the problem is NOT OE/bitbake.. the rpoblem is 100% in the NFS servers and clients May 16 11:52:43 they're just plain broken in many cases May 16 11:53:53 for our builders we have dedicated processes that fetch from the network.. all of the other build process then only fetch from the local download cache.. that avoids the multiple builders trying to fetch the same thing issue (and ensures our local cache is always complete) May 16 11:54:45 --runall=fetch FYI May 16 11:55:09 and then you share that local download cache with PREMIRROR? May 16 11:55:13 yes May 16 11:55:29 (we also disable external network acccess on those builders) May 16 11:56:00 limits it to one controlled environment that accesses outside network resources.. helped us control network traffic when a major upgrade happened May 16 11:56:55 ok, makes sense, the only downside I see is that all the builders are blocked until this separate process is done updating all needed pieces May 16 11:57:19 correct.. but you can run that process in parallel with the regular builds, since it's addative May 16 11:57:27 which might not be big issue for you if you don't need so many updates May 16 11:58:17 just have to make sure you do the download fetch before running a series of builds.. in our environment we have ways to do this -- but due to the volume of builds, there may be short periods of time that the featcher hasnt caught up.. but these are generally "quick" failures.. May 16 11:58:26 not in the way how we're using it, we have a developer which creates a review in gerrit somewhere and might trigger CI in completely different location few minutes after that May 16 11:58:41 ya that is different then us.. May 16 11:59:11 for those types of builds we start with the premirror from the fetch resource, and then allow network access for the rest.. but that 'rest' is not cached, since the change hasn't been comitted to the repository yet.. May 16 11:59:22 with shared DL_DIR first builder which gets to do_fetch, downloads new reference from remote location to local NFS in shared DL_DIR, other builders if they reach do_fetch as well, will wait a bit until the lock is lifted and then they finish all at almost the same time May 16 11:59:23 once it's committed hten the regular process kicks off and actually stores the download longer term.. May 16 11:59:38 and the shared DL_DIR is then used by developers in given location as local PREMIRROR as well May 16 12:02:53 for the committed changes it's the same for us, someone will create annotated submissions/NN tag in the repository somewhere and almost immediately trigger verification build with metadata change incrementing the submission number for his component. May 16 12:05:46 and the same with shared SSTATE_DIR, except that the builds started in parallel cannot take much advantage of one of them finishing the task a bit sooner (because the setscene tasks were already finished and they all already decided to run the actual task itself) May 16 12:08:33 Ya the shared SSTATE_DIR, I'm not sure how exactly that is setup... the build guys have figured it out, but I know at one point rsync was involved to ensure atomic updates... not sure fi it's still being used.. but NFS was a real problem and we got away from it as far as R/W, and moved to read-only.. May 16 12:08:54 hopefully the task queue will be rewritten to consider cache after every step instead of only once at the beginning May 16 12:08:55 and every time we chased down a problem, it was a bug in the NFS server itself.. OE was doing the right thing already.. May 16 12:09:48 kanavin that is really the only deficiency I see with the current way of doing things.. (other then initial parse performance, but we've looked at that and I'm not sure it can be optimized any further without a possibly radical change) May 16 12:10:09 kanavin: that would be nice especially with the sstate equivalence thing which would compare the build output of each step May 16 12:11:32 about to give an "intro to yocto" presentation to folks who were formerly application engineers and never had to deal with recipes or running the build system.. :) May 16 12:11:49 good luck! :) May 16 12:12:56 (biggest trick will be to get a taxi when I'm done with this back to Helsinki.. I'm some 35 minutes west of Helsinki right now.. kind of the middle of nowhere) :) May 16 12:13:30 fray, may I ask whome are you visiting, or is that ultra secret? I lived in Helsinki for 15 years May 16 12:13:51 think large telco company.. :) headquartered in Espoo May 16 12:13:58 worked for Nokia, then Intel May 16 12:14:10 ya this is the training facility to the west May 16 12:15:45 in Helsinki area anything outside of the smallish core center feels like middle of nowhere, it's basically an overgrown village May 16 13:19:22 New news from stackoverflow: How to build a customized linux kernel image for sama5d27 som1 ek1 board? May 16 15:00:44 RP: khem: it looks like there is still some missing PREFERRED_VERSION for one of gcc recipes (maybe only with multilib enabled) May 16 15:01:28 glibc just failed for me with: 9.1.0/ld.bfd: cannot find -lgcc while using the current default GCCVERSION ?= "8.%" May 16 15:01:53 NOTE: recipe lib32-gcc-cross-arm-9.1.0-r0: task do_populate_lic_setscene: Started May 16 15:01:53 NOTE: recipe lib32-gcc-runtime-8.3.0-r0: task do_populate_lic_setscene: Started May 16 15:02:13 NOTE: recipe gcc-cross-canadian-aarch64-8.3.0-r0: task do_populate_lic_setscene: Succeeded May 16 15:02:16 NOTE: recipe gcc-cross-canadian-arm-9.1.0-r0: task do_populate_lic_setscene: Started May 16 15:03:56 it's set only for cross-${TARGET_ARCH}/cross-canadian-${TRANSLATED_TARGET_ARCH} not the other arch from multilib May 16 15:04:41 maybe using negative D_P while the newest one isn't the default one would make sense as well May 16 15:27:30 The do_image_complete function removes the old images. Is there a way to avoid this (that is keep the old images)? Thanks! May 16 15:30:54 JaMa: I wonder why we don't see that on our multlib tests. I kind of left this fuzzy to track down some of these issues May 16 15:31:11 JaMa: I did think I had them all after sorting fortran though May 16 15:31:28 well, there is a better way to sort fortran but... May 16 15:35:41 RP: is there build log from one of multilib builds on AB? maybe it also builds 2 different versions, just without triggering the failure in glibc May 16 15:36:19 JaMa: https://autobuilder.yoctoproject.org/typhoon/#/builders/44/builds/611 May 16 15:36:25 I'm debugging the code in multilib_global.bbclass and I'm not seeing TARGET_ARCH to be expanded with multilib (only the MLPREFIX added) May 16 15:38:03 JaMa: you're probably right about it working and not triggering the failure May 16 15:40:23 JaMa: had a quick grep over the step*b logs and can't see gcc 9 anywhere May 16 15:41:53 neither did I, will try to replicate the multilib setup from AB and compare the debug output from multilib_global.bbclass May 16 16:46:00 multilib magic continues ERROR: Nothing PROVIDES 'lib32-gcc-cross-i586'. Close matches: lib32-gcc-cross-i686 lib32-go-cross-i586 lib32-gdb-cross-i686 May 16 16:48:42 and I don't mean the i586 in lib32-gcc-cross-i586 I've used in bitbake -e, but go-cross is using i586 arch, while P_V is set only for lib32-go-cross-i686 May 16 16:52:17 ah, that's proabably because go-cross sets PN = "go-cross-${TUNE_PKGARCH}" and PREFERRED_VERSION_go-cross-${TARGET_ARCH} May 16 16:59:09 Is there a way to create more than an image(wic) from same recipe(bb)? May 16 17:02:16 JaMa: ah, that starts to explain more about what is going on May 16 17:05:05 RP: but this only on go-cross (not gcc-cross), so it might be separate issue, in my build it seems to work for x86_64 (and i686 multilib), but not aarch64 (and arm multilib) with multilib enabled in both machines very similarly May 16 17:06:24 I've sent RFC about go-cross (in case someone remembers why TUNE_PKGARCH is used there) May 16 17:12:45 RP: I took a look at bitbake/lib/bb/runqueue.py... yeah it's not easy :) there are two separate executor classes for setscene tasks and 'real' tasks, first one runs to completion then the other May 16 17:13:59 the rewrite would have to somehow combine both into one mega-loop, where each completed real task would trigger another pass at setscene tasks May 16 17:23:23 kanavin: That was my conclusion also May 16 17:55:50 Hi, it's probably a dumb question but if I call `bitbake` by setting first an environment variable `MY_VAR=1 && bitbake image`. How can I read the value of MY_VAR from the recipe ? xx = ${MY_VAR} ? May 16 17:58:01 erakis: I think bitbake only imports a small number of variable from the environment May 16 17:58:34 erakis: https://www.yoctoproject.org/docs/current/bitbake-user-manual/bitbake-user-manual.html#var-bb-BB_ENV_WHITELIST May 16 17:59:25 erakis, environment variables are generally a very difficult thing to debug, I would recommend you control your recipe behaviour through other means May 16 17:59:25 erakis: But, you want to be careful about doing that because it can cause a lot of unecessary rebuilding May 16 18:00:36 If I had a time machine, I would prevent them from happening at all. May 16 18:03:44 Suppose I want to launch a build from a CD/DI system like Gitlab. I would like to pass the job/build ID to yocto and retrieved it from a recipe. The only way I found so far is to declare a variable into the `local.conf` and then lunch the bitbake from a script that use `sed ....` to replace the variable's value by the one coming from an environment variable or the script's parameters. May 16 18:04:15 kanavin: What do you propose? May 16 18:05:03 JPEW: Thank you. May 16 18:06:07 erakis, where would you use the job id? May 16 18:07:00 if you want to include it in images' filenames, you can rename them after bitbake completes May 16 18:12:33 kanavin: The job id is automatically generated on Gitlab when build is triggered via web api or web interface. We use it as a build id. But once the image is build and flash on the device we have no way to read the original build number. I recently discovered the recipe `distro-feed-configs` and then I would override it to include the gitlab job id in the `/etc/build` file. When gilab runner trigger a build it open a shell and then launch bitbake so this is May 16 18:12:34 way I though passing the job id using an environment variable was a good idea. May 16 18:15:35 erakis: bitbake will respect an auto.conf file in addition to local.conf. The auto.conf file is designed to be written by CI. I think if you make your CI job write out this as auto.conf it will do what you want: https://pastebin.com/JxdELz6s May 16 18:17:35 erakis: FWIW we're using BB_ENV_WHITELIST exacly for this to pass BUILD_ID from jenkins to bitbake environment May 16 18:18:05 which is then used to construct consistent versioning of all build artifacts (images, kernel images, modules, etc) May 16 18:18:36 the problem with environment variables is that they live in memory, so when things go wrong, it's difficult to establish what they were at any given point May 16 18:18:59 for this reason I always prefer passing data via files May 16 18:19:23 for us they are e.g. written at the beggining of build in the Build Configuration, so it's quite clear what they were May 16 18:19:38 unlike some files which most people cannot ever access on the build slaves filesystem May 16 18:20:47 files however remain in place after something has failed, so you can go and inspect them May 16 18:21:05 for environment vars, you have to insert printouts at strategic points in code May 16 18:21:09 What I'm doing is not far from this, I use a script to start the build. This script open the file `local.conf` and overwrite the value of the variable `MY_BUILD_NUMBER` for `MY_BUILD_NUMBER=4434`. May 16 18:21:09 after which the variable is available in the recipes. May 16 18:21:40 not for us, when something fails then there is probably many builds in queue which will happily remove all the left-over files and start their own script May 16 18:22:49 JaMa, that's for debugging failures locally. I know from experience that env vars are a pain in the butt. Just the other day I wanted to upgrade apt in yocto (because the one we have has a major security hole). May 16 18:23:18 I decided to do something else after finding out the new apt sanitizes the environment, and I have no idea where to start looking for where it happens. May 16 18:23:42 had it passed configuration via a file, it'd be far easier to deduce May 16 18:24:05 as long as you don't export that BB_ENV_WHITELISTed variable, then what's wrong with that? May 16 18:29:42 kanavin: JaMa: Thank you for you advise. May 16 18:47:13 RP: the multilib/gcc P_V issue is really strange, if I add another localdata.expand(v) before http://git.openembedded.org/openembedded-core/tree/meta/classes/multilib_global.bbclass#n38 then it works for my aarch64 (with arm multilib) case as well, will dig a bit more May 16 19:21:41 JaMa: that does seem odd :/ May 16 19:22:55 kanavin: your summary is correct, it is basically a rewrite of runqueue :/ May 16 19:23:53 kanavin: still thinking about that packagegroup rebuild thing May 16 19:24:31 kanavin: I worry 7298 is related and it'd be the second time that bug has come up recently May 16 19:40:20 RP: this however is pretty clear-cut special case - the packagegroup explicitly refers to packages with TUNE_ARCH in their name (smth like binutils-cross-canadian-$TUNE_ARCH). If TUNE_ARCH changes, packagegroup must be rebuilt. May 16 19:51:11 RP you still awake ? May 16 19:51:47 bug 10653 is in Stephen's name May 16 19:51:48 Bug https://bugzilla.yoctoproject.org/show_bug.cgi?id=10653 normal, Medium, Future, sjolley.yp.pm, NEW , Improve Node.js developer workflow. May 16 19:53:29 * armpit thats freaky May 16 20:21:41 Is there any way to override/modify the CONFIGUREOPTS from autotools? I'm trying to remove the --target and replace it by --enable-targets= for gdb-cross, it is mostly to access the threaddb functionnality of gdb (a big hack). May 16 20:25:42 easily that is May 16 20:28:34 worst case just use _remove. May 16 20:30:09 armpit: yes, kind of May 16 20:32:00 psrcode: not sure if you saw it but I've been testing more ptest configurations and lttng-tools-ptest isn't working so well on a core-image-minimal. I suspect there are some "obvious" dependencies we're missing but its back to the timeouts :/ May 16 20:32:41 kergoth: that could actually do it, not sure why I did not think about thatr May 16 20:33:13 RP: yep, will look at it. I should be able to reproduce this with a simple conf with lttng-tools-ptest installed? May 16 20:33:45 psrcode: yes, its a core-image-minimal with lttng-tools-ptest and openssh added (plus some space and memory for qemu) May 16 20:33:47 RP: what tree are you testing this against? juste to be sure I have the same tree May 16 20:34:03 psrcode: basically the exact config lines in my email. It was with current master May 16 20:34:24 perfect May 16 20:34:31 i'll give it a try May 16 20:35:10 psrcode: I'm sure its something obvious missing, I've not looked into what yet as there are a few issues that test highlighted... May 16 20:35:39 kergoth: will the _remove be scoped to only the current recipe? May 16 20:36:01 if it's defined in the recipe or bbappend, then by definition yes, a recipe can't affect configuration or other recipe metadata May 16 20:36:06 otherwise use the _pn- override May 16 20:37:10 kergoth: good, sorry for those stupid question, I need to find some time to do a proper reading of the manual. May 16 20:50:35 RP: I found the reason why the multilib_global.bbclass didn't work with our tune files, it's this line: OVERRIDES_append_class-target = "${@bb.utils.contains('TUNE_FEATURES', 'webos-minsize', ':webos-minsize', '', d)}" May 16 20:52:00 RP: so the OVERRIDES used by localdata in multilib_global.bbclass were: "${TARGET_OS}:${TRANSLATED_TARGET_ARCH}:pn-${PN}:${MACHINEOVERRIDES}:${DISTROOVERRIDES}:${CLASSOVERRIDE}${LIBCOVERRIDE}:forcevariable${@bb.utils.contains('TUNE_FEATURES', 'webos-minsize', ':webos-minsize', '', d)}:virtclass-multilib-lib32" May 16 21:00:27 RP: after replacing _append with .= it works as well May 16 21:08:16 * armpit wonders if overrides is shooting? May 16 21:09:15 * armpit or maybe overrides is German May 16 21:11:41 shooting Germans? May 16 21:31:56 Err, whats the Upstream-Status for an unmaintaned project (zip)? May 16 21:51:09 New news from stackoverflow: Unable to parse Machine ?= raspberrypi? May 16 22:21:13 New news from stackoverflow: How To Determine Which Recipe Built A File In The Target SDK? May 16 22:26:33 JPEW, Inappropriate [dead project] ? May 16 22:27:00 ah,, nope May 16 22:27:04 no upstream (the upstream is no longer available -- dead project) May 16 22:27:23 its in https://www.openembedded.org/wiki/Commit_Patch_Message_Guidelines May 16 22:51:19 New news from stackoverflow: How to tell bitbake to calculate variable's basehash value after specific task? **** ENDING LOGGING AT Fri May 17 02:59:57 2019