**** BEGIN LOGGING AT Mon Jul 08 02:59:58 2019 Jul 08 06:51:44 good morning Jul 08 08:03:43 New news from stackoverflow: Copy Yocto Project to other PC by tar Jul 08 09:33:49 odd, I'm seeing lots of do_populate_sdk failures from my build after switch from sumo to master branch. any ideas why images do_populate_sdk task dependency to recipes do_package_write_ipk and do_packagedata would not be enough? packages fail to install in do_populate_sdk due to "opkg_prepare_url_for_install: Couldn't find anything to satisfy 'binary-package-name'" Jul 08 09:34:12 these errors only happen after wiping tmp, first build fails. second passes. Jul 08 09:47:15 is populate_sdk_base.bbclass maybe missing dependencies: do_populate_sdk[recrdeptask] += "do_populate_lic do_package_qa" ? Jul 08 09:47:30 because do_rootfs tasks have these but do_populate_sdk doesn't Jul 08 09:50:54 <__angelo> for a new ditribution, where is the proper place to keep the local.conf ? Jul 08 09:51:32 <__angelo> oe-init-build-env complains about templateconf: No such file or directory Jul 08 11:23:18 local.conf is - by definition - not your distribution Jul 08 11:23:35 the distro config is called .conf Jul 08 11:24:09 the distro layer can also ship a setup script and template to seed a local.conf which sets DISTRO correctly, just like poky does Jul 08 11:24:50 or if you want a really lean example, https://github.com/rossburton/customdistro Jul 08 11:34:24 what could be an approach to have a legacy grub integrated into the build process? There's a doc about doing it manually https://wiki.yoctoproject.org/wiki/TipsAndTricks/Running_YP_Image_On_AWS, so i guess one would need to implement a wic plugin or a custom class to automate this? Jul 08 12:07:09 Has anyone seen "-mfloat-abi=hard" get set for qemuarm on thud? I've dumped my environment but do not see where it's being set. I'm not seeing it in temp either. Very weird. Jul 08 12:34:19 New news from stackoverflow: Yocto Warrior Bitbake Python3 Matplotlib 3.1.1 JQuery Download Error on do_install Jul 08 12:43:34 RP: Do you know if it is possible to add a task hash dependency to another task? Jul 08 12:43:34 I was thinking about how to fix the do_stash_locale() patch problem for the future, as well as the ramification to others who update or try to bisect a tree with a glibc change. Jul 08 12:43:34 I was wondering if the glibc do_compile rule could depend on the hash of the do_stash_local (), such that if it ever changes, we have to go all the way back to running do_compile, since that is where bin/localedef comes from? Jul 08 13:00:01 jwessel: isn't that like a normal task dependency? Jul 08 13:00:45 Well when the do_stash_locale () function gets changed, the do_compile and do_install never runs again for the glibc Jul 08 13:01:00 Which causes the files that get mv'ed to be gone forever. Jul 08 13:01:14 At least until the next time you cleansstate on the glibc. Jul 08 13:01:32 jwessel: so some task isn't idempotent ? Jul 08 13:02:01 I believe so. Jul 08 13:02:24 jwessel: tasks are supposed to be so if its not, that is the real bug Jul 08 13:03:14 I am not entirely sure how the glibc bits were designed, but it is a problem with the way the sstate is "cheated". Jul 08 13:05:08 A number of folks in the community ended up seeing the problem building from the sstate, such that when the do_stash_locale runs (because it was changed by a patch I submitted), it runs the mv operation again, but there is nothing to move. It is long gone. Jul 08 13:05:08 jwessel: just to be clear and check I understand, the real issue is that do_stash_locale() reruns but do_install doesn't, so files that were moved don't exist any more Jul 08 13:05:20 Correct. Jul 08 13:05:47 The do_stash_local was setup to run before the sstate_save of the glibc. Jul 08 13:05:50 jwessel: do_stash_locale is the problem then, its meant to be able to be rerun and can't be Jul 08 13:06:11 Sure. To be clear, it wasn't a problem from my patch. Jul 08 13:06:27 This is a long standing problem, I just wasn't sure how to fix it, but it seems like it should get fixed. Jul 08 13:06:32 jwessel: right, you just exposed the problem Jul 08 13:07:13 The obvious choice to me, at the time was to ask if there was a way to have this function be a part of the hash computation for the do_compile rule. Jul 08 13:07:38 have an image for which want to build SDK by default. Thus it has d.appendVarFlag("do_build", "depends", " {}:do_populate_sdk".format(pn)) and d.appendVarFlag("do_rm_work", "depends", " {}:do_populate_sdk".format(pn)). With latest master it also seems to need d.appendVarFlag("do_packagedata", "depends", " {}:do_populate_sdk".format(pn)). Is that expected or am I doing something silly? Jul 08 13:07:40 The glibc has another separate problem in that bin/locale_def only comes from the do_compile rule. Jul 08 13:07:54 jwessel: that doesn't fix the problem. The task just needs to be rewritten so it can be reexecute Jul 08 13:08:32 RP: That would imply you need a "double super secret stash", because there are no other copies of the files. Jul 08 13:08:39 jwessel: what happens if we just change the mv to cp and remove the rm ? Jul 08 13:08:55 I think who ever invented this, wanted to save some space or something. Jul 08 13:09:16 jwessel: perhaps, I think we need to unravel that problem though Jul 08 13:09:19 I don't understand why glibc-locale is a separate recipe in the first place. Jul 08 13:09:28 jwessel: going down the "more task hash magic" route won't help, trust me Jul 08 13:09:38 jwessel: that I do know, speed Jul 08 13:09:48 That is why I thought I'd ask you first. Jul 08 13:09:55 jwessel: building and packaging locales takes an age and you want the rest of the build to continue Jul 08 13:12:42 jwessel: I'd guess a package preprocess func and a populate_sysroot preprocessing func which deleted the files they shouldn't see should have the same effect Jul 08 13:13:05 Perhaps I am look at it the wrong way then. different question... Jul 08 13:13:25 Why can't the stash function which does the mv stuff be in the do_install rule? Jul 08 13:13:47 The bit that packages up the results shouldn't have to change. Jul 08 13:13:51 jwessel: do_stash_locale is an sstate task and do_install is not Jul 08 13:14:57 jwessel: I suspect that it will have to change Jul 08 13:16:00 Trying to simplify it in my mind... The glibc (not locale) sstate depends on the do_install() implicitly already. Jul 08 13:16:42 jwessel: it comes down to a simple underlying principle in the buildsystem ignoring sstate Jul 08 13:17:10 jwessel: we allow arbitrary tasks to be re-executed Jul 08 13:17:33 jwessel: right now if you do "bitbake glibc -c stash_locales" and then rerun that, it breaks Jul 08 13:17:37 so sstate involved Jul 08 13:17:42 no sstate involved Jul 08 13:17:52 I hadn't tried that, but I would imagine that would break things. Jul 08 13:18:02 In the same manner. Jul 08 13:18:11 so yes, sstate gets broken but that isn't the root problem Jul 08 13:18:37 I was thinking about it more from the perspective of the code changes. Jul 08 13:19:01 So that day to day it builds. The exec problem you just mentioned probably exists in a lot of places. Jul 08 13:19:15 jwessel: no, it doesn't or at least shouldn't Jul 08 13:19:34 jwessel: we did used to test for this but haven't for a while as its a long/painful test Jul 08 13:20:29 jwessel: if tasks can't rexecute, the whole system is just totally broken as we *rely* on that to be true Jul 08 13:20:48 The tasks absolutely can re-execute so long as the bits they depend on re-run. Jul 08 13:21:10 For example, if I ran by and the 3 temp files, it certainly still works. Jul 08 13:21:10 jwessel: no, absolutely not Jul 08 13:21:31 jwessel: tasks things depend on do not rerun untless they too have changed Jul 08 13:22:18 So you understand why I asked the question how to make the do_compile() depend on the do_stash_locale. :-) Jul 08 13:22:39 jwessel: I understand your question, yes but it means you don't understand how the system works Jul 08 13:22:44 and its not the correct fix Jul 08 13:24:09 I am not sure how to fix it, other than to use the cp vs the mv, and then glibc has all the extra files. Jul 08 13:24:32 Perhaps add to the glibc another package called glibc-locale-noinstall ? Jul 08 13:24:35 jwessel: That is basically the only way to fix it Jul 08 13:24:37 Which collects the files? Jul 08 13:24:48 but never gets installed by anything? Jul 08 13:25:02 jwessel: right, we change the mv/rm to cp and then have to fix the fallout "somehow", there are a few ways for that Jul 08 13:25:18 I'd rather not create it than create something we then have to "hide" Jul 08 13:27:07 jwessel: there are bascially two ways. One is cp instead of mv, the other is to mv into some special location and then check for the existence of that so you know not to redo parts of the operation Jul 08 13:27:23 jwessel: I prefer the cp approach and fix other fallout as I think it will be cleaner Jul 08 13:28:02 Well with a "cp" approach, some kind of glibc collector of the files needs to be added. Jul 08 13:28:33 Else we'll have "glibc created XXX files which are not packaged..." Jul 08 13:28:38 jwessel: or it removes the files Jul 08 13:29:24 jwessel: see PACKAGE_PREPROCESS_FUNCS += "gi_package_preprocess" Jul 08 13:29:24 gi_package_preprocess() Jul 08 13:29:36 and SYSROOT_PREPROCESS_FUNCS_append = " gi_ldsoconf_sysroot_preprocess" Jul 08 13:29:36 gi_ldsoconf_sysroot_preprocess () Jul 08 13:30:07 jwessel: just the first examples I could come across but you get the idea Jul 08 13:30:14 I'll have a look. Jul 08 13:30:28 jwessel: they're operating on copies of the files so this is safe Jul 08 13:33:34 How does one get at the copy that didn't get erased? Jul 08 13:34:16 It will still be in ${D} ? Jul 08 13:34:33 It is probably easy enough to try to see what happens... Jul 08 13:41:52 RP: Thanks for the insight. I'll setup a test with the PACKAGE_PREPROCESS_FUNCS to purge the rm, and see where it goes. If it works I'll submit a patch to fix it. Jul 08 13:45:26 jwessel: ${D} is the copy created by do_install Jul 08 14:12:02 adding a new package to the SDK, dnf is failing as the last step when assembling it, with error "package foo does not have a compatible architecture" Jul 08 14:12:21 the rpm content seems fine and not particularly different from other RPMs that are built Jul 08 14:12:27 any tip? Jul 08 14:15:16 the binary inside the pkg seems to be fine Jul 08 14:18:32 duh, it was a typo in the spelling of the pkgname in the local.conf -_- sorry for the noise Jul 08 14:42:05 Has anyone had trouble around thud with not all runtime dependencies being installed? For example, I have a package that rdepends on python-numpy, however if I try to import it on the target, I get an error that future_builtins doesn't exist. Jul 08 14:48:04 which is more common / correct? foo__any.ipk or foo__all.ipk for arch independent packages? Jul 08 14:58:29 tgoodwin: sounds like the rdepends for numpy are not complete Jul 08 15:11:25 rburton: I'm chipping through it now; it caught me off guard since originally I had a slew of python packages in my image_install, which I pulled so that nothing "extra" was being required (save space, etc.) and then this broke. Jul 08 15:15:31 tgoodwin, that is usually hiw we find bad DEPNDS :) Jul 08 15:22:32 yep :) Jul 08 15:25:50 ALways a good test to build an image with only gnuradio and see if it does something useful Jul 08 15:27:20 I had to add python-modules to my image_install fix it (without extending my package's rdepends to include other ones). Jul 08 15:27:34 *to fix it... accidentally a "to" Jul 08 15:43:18 tgoodwin, do need to track the root fix though and submit a patch Jul 08 16:12:07 Crofton: indeed, but I'm not sure what the appropriate fix would be if this impacts any python module, for example. As I was debugging this, it also impacted pydoc until I added python-modules to the image. Jul 08 16:12:11 Then everything was fine. Jul 08 16:50:47 Crofton: no "fix" necessary maybe? https://www.yoctoproject.org/docs/current/mega-manual/mega-manual.html#migration-2.6-packaging-changes says you have to install specific ones or "all" with python-modules, but it makes me wonder if it's a subset of the modules that makes this work correctly. Jul 08 16:56:40 is python-modules a meta-package that installs opthers? Jul 08 16:59:40 Crofton: yes, its all modules Jul 08 17:00:01 RP: is that "all modules" as in everything that can be built or "all" that the image has included? Jul 08 17:00:14 (I'm reading over that python function in python_2.7... right now) Jul 08 17:02:29 For your afternoon enjoyment: https://www.youtube.com/playlist?list=PLD4M5FoHz-TxMfBFrDKfIS_GLY25Qsfyj Jul 08 17:10:01 python-modules (and python3-modules) is a meta-package that depends on all the modules Jul 08 17:13:41 so it sounds like numpy shoudl RDEPENDS on python-modules, or prefeably, the one it really needs? Jul 08 17:14:58 Well even when I manually added python-futures to my image, I couldn't import it from the python shell despite the egg, etc., being installed. Jul 08 17:15:25 It's like there's something missing from the python package since I would think "import" should work if the package is installed. Jul 08 17:16:19 So where I'm off the rails is figuring out which package it is that helps importing work (since it impacted pydoc too) Jul 08 19:11:17 Thanks RP! I'll send a patch out for review. I was able to fix the do_stash_locale() implementation with your guidance. Jul 08 21:19:40 jwessel: thanks, at a quick glance it looks right. Appears we already had half a fix there for the cp vs mv :) Jul 08 21:20:07 Indeed it is true. Your assertion about the issue was spot on. Jul 08 21:23:47 jwessel: Its good to have it fixed, I suspect there are others like it. We have a script somewhere which basically runs bitbake X -c Y -f to test for issues like this Jul 08 21:24:06 jwessel: its a lot of combinations though Jul 08 21:24:10 I am not even sure that would have uncovered it though in this case. Jul 08 21:24:40 It isn't until you try to build glibc-locale after running it a second time, that it chokes, because it had catches for the existence of the files. Jul 08 21:25:11 jwessel: I guess you'd need "bitbake glibc -c stash_locales -f; bitbake glibc-locales" :/ Jul 08 21:25:41 jwessel: that is indeed harder :( Jul 08 21:26:14 Finding that is a bit more by accident. That is why I wanted to get it fixed for the next guy that comes along. Jul 08 21:26:31 Plus, I didn't want things to break when I push the next set of patches to our end user base. Jul 08 21:26:50 I never want a situation where an incremental build fails for some odd ball like this. Jul 08 21:27:23 Plus the error generated non-sense, not pointing you to any obvious root cause. :-) Jul 08 21:37:17 jwessel: totally agreed on fixing it. I'm just trying to think about better detection Jul 08 21:37:37 I guess force running and sstate hash equivalence testing would be the test here Jul 08 21:38:09 sstate content comparison for hash equivalence to be clear Jul 08 21:41:46 Detection is interesting. If you wanted to per package check it, that would have been caught. Jul 08 21:42:03 e.g. Build package glibc Jul 08 21:42:38 Run a subtask again and let it go all the way through packaging and check sstate. Jul 08 21:42:52 It would definitely have been different the second time because there was nothing left to move. Jul 08 21:44:05 I am guessing that would probably catch most of these that don't just blow up when you run it a second time. Jul 08 21:45:55 jwessel: I guess buildhistory would detect this. The question is how much of a build to rerun Jul 08 21:47:03 I doubt you want to check a whole build the whole way through as it gets expensive, but per package seems reasonable. Jul 08 21:47:16 My question is how many packages have the sub tasks that need checking? Jul 08 21:47:59 jwessel: well, in theory you can break any task so it doesn't rexecute without some side effect Jul 08 21:48:50 jwessel: This isn't something anyone likely has time to look at right now, its just interesting to remember what may have helped or worked better Jul 08 22:26:36 <__angelo> first steps into custom distro, i am getting "FileNotFoundError: [Errno 2] file conf/bitbake.conf not found" Jul 09 00:35:45 __angelo: What command were you running that got you that error? What steps have you completed to get to this point? Jul 09 01:53:17 Hi guys, I'm going to build poky with command `bitbake core-image-minimal`. The error is occurred with output: `ModuleNotFoundError: No module named '_sqlite3'`. After searching the google, someone suggested to re-install python. What is the root cause? Jul 09 02:46:17 huynq: I think that is not an error you should be able to receive unless your python installation is somehow broken, hence the recommendation Jul 09 02:50:11 bluelightning: hmm, I think the python in my system is work as well. There are two Python application (Restful API + web) which are running on the system. Jul 09 02:57:50 huynq: well, nowhere in the bitbake code do we try to import _sqlite3; that implies that it's somewhere in your python libraries that does (and fails) Jul 09 02:59:16 we do "import sqlite3", so it would be interesting to see what happens if you tried that in a python (and python3) shell **** ENDING LOGGING AT Tue Jul 09 02:59:57 2019