**** BEGIN LOGGING AT Wed Jun 14 03:00:03 2017
Jun 14 04:16:32 <rick1> hello, everyone, good afternoon...
Jun 14 04:20:10 <rick1> I have a question related with meta-web-kiosk, does anyone have a knowledge about it ? thanks
Jun 14 05:23:17 <khem> rick1: what questions do you have specific to that layer
Jun 14 05:56:06 <phako[m]> Hi, I'm trying to convince the qtbase recipe to put the bearer plugins into a separate package... so what I did in a .bbappend is https://paste.gnome.org/phzqqnnjw - I see the folders being created in packages-split, but the .so files still end up in the generic -plugins package...
Jun 14 05:58:31 <CostinC> morning all
Jun 14 06:32:43 <rick1> I want to add chromium browser into core-image-sato...The step : 1 .I had made core-image-sato-custom.bb
Jun 14 06:33:09 <rick1> 2. Add add bblayer.conf ==> meta-browser
Jun 14 06:33:52 <rick1> 3. Add IMAGE_INSTALL += "chromium" inside core-image-sato-custom.bb ?
Jun 14 06:34:09 <sandsmark> sounds about right?
Jun 14 06:34:11 <sandsmark> doesn't it work?
Jun 14 06:34:46 <rick1> correct me if I'm wrong, I can't do that yet because I 've been compiling something else here..
Jun 14 06:35:34 <sandsmark> just make sure the meta-browser layer is before your own layer, iirc that matters
Jun 14 06:36:34 <rick1> I don't have much layer, only default in bblayers.conf..I'll keep posted if I can or can not that..
Jun 14 06:36:43 <rick1> many layers..
Jun 14 06:36:54 <rick1> I don't have many layers
Jun 14 06:37:36 <rick1> I'm learning so I'm not sure...
Jun 14 06:37:49 <LetoThe2nd> sandsmark: nah that actually not *that* important.
Jun 14 06:38:10 <LetoThe2nd> way more important is that the release branches are coherent. e.g. all morty, all krogoth, you get it.
Jun 14 06:38:42 <sandsmark> ah, my bad
Jun 14 06:39:18 <LetoThe2nd> sandsmark: no bad. AFAIK there can be edge cases where the order matters, and its good practise :)
Jun 14 06:40:01 <sandsmark> ok, so I wasn't misremembering completely :p
Jun 14 06:40:50 <LetoThe2nd> sandsmark: not at all. its just in most cases not important when adding some package that does not already see appends or such.
Jun 14 06:41:06 <rick1> I'm following the book 5.2. Customizing Images yoctoproject 2.3 mega manual
Jun 14 06:41:10 <sandsmark> ah, of course
Jun 14 06:42:02 <sandsmark> so yeah, keep it after the meta-browser layer, you're probably going to bbappend to the chromium recipe
Jun 14 06:42:13 <sandsmark> rick1: btw, the chromium recipe in the meta-browser layer was very outdated last I looked
Jun 14 06:42:16 <sandsmark> it used gyp for example
Jun 14 06:42:45 <LetoThe2nd> rick1: all is well. just for completeness, your core-image-sato-custom recipe should go into your custom layer, not into one of the existing upstream ones.
Jun 14 06:43:02 <sandsmark> rick1: this is up to date: https://github.com/rakuco/meta-crosswalk/tree/chromium59/recipes-browser/chromium
Jun 14 06:43:33 <rick1> thanks sandmark...
Jun 14 06:43:51 <rick1> leto: how can I use custom layer ?
Jun 14 06:44:31 <sandsmark> rick1: there's a command that creates the basic directory structure for it
Jun 14 06:44:57 <LetoThe2nd> rick1: poky comes with a script called yocto-layer. so, "yocto-layer create mystuff" should create a layer where you are, in a directory called "meta-mystuff"
Jun 14 06:45:10 <rick1> my experiment now, put custom-image-sato-custom.bb, inside poky/meta/recipes-sato/images
Jun 14 06:45:18 <LetoThe2nd> rick1: add that to bblayers.conf, and use it just like the existing ones.
Jun 14 06:45:29 <rick1> and bitbake it.....it's running now with
Jun 14 06:46:00 <rick1> IMAGE_FEATURES += "splash package-management x11-base x11-sato ssh-server-dropbear hwcodecs ssh-server-openssh" and IMAGE_INSTALL += "strace nano"
Jun 14 06:46:20 <LetoThe2nd> rick1: for a first test tinkering in poky is all fine. its just once you modify more than one place, want to keep your changes, want to pull updates, then it becomes nasty. so basically, in all real-world usage cases.
Jun 14 06:46:55 <rick1> Leto : I see...I don't have your skill yet
Jun 14 06:47:16 <sandsmark> rick1: it's very straightforward to use a custom layer
Jun 14 06:47:20 <LetoThe2nd> rick1: all is well as long as you're willing to learn :)
Jun 14 06:47:34 <sandsmark> and it gets harder to move stuff over later, so better start with it right away (from experience)
Jun 14 06:47:54 <sandsmark> but another unrelated issue that I've been stuck with half a day (and a night); I added a wpa-supplicant_%.bbappend and put in «SYSTEMD_SERVICE_${PN} = "wpa_supplicant@wlan0.service"» and «SYSTEMD_AUTO_ENABLE = "enable"», to try to get it to auto-enable wifi on boot, but for some reason my image ends up with «wpa_supplicant@%i.service» in the multi-user.target.wants folder
Jun 14 06:48:39 <LetoThe2nd> sandsmark: were all those underscores intended?
Jun 14 06:48:40 <sandsmark> the pkg post_inst script seems to do the right stuff («systemctl $OPTS enable wpa_supplicant@wlan0.service»), and if I run that manually after booting it enables it correctly
Jun 14 06:48:43 <rick1> sandmark : yup...I'll make my "meta-stuff" and get back again...
Jun 14 06:48:56 <sandsmark> LetoThe2nd: yeah, they are from the original wpa_supplicant_foo.bb
Jun 14 06:49:15 <sandsmark> eh, the .bb is called wpa-supplicant_foo, but all the scripts and service files are wpa_supplicant
Jun 14 06:49:19 <rick1> I've read yocto-layer create mystuff before, but not yet in practice...
Jun 14 06:49:48 <LetoThe2nd> sandsmark: it its really call "_SYSTEMD_SERVICE_${PN}"? that sounds weird. let me have a look
Jun 14 06:50:05 <sandsmark> thank you
Jun 14 06:50:31 <rick1> anyway...what time is it there ?
Jun 14 06:50:31 <sandsmark> I'm fairly sure it is the correct way to do it, but I think I've been looking too long at it, probably overlooking something simple
Jun 14 06:51:09 <sandsmark> rick1: 08:51
Jun 14 06:51:09 <LetoThe2nd> sandsmark: i don't see none of the lot of underscores that were in your snippets here: http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-connectivity/wpa-supplicant/wpa-supplicant_2.6.bb?h=master
Jun 14 06:51:52 <sandsmark> LetoThe2nd: these?: http://cgit.openembedded.org/openembedded-core/tree/meta/recipes-connectivity/wpa-supplicant/wpa-supplicant_2.6.bb?h=master#n18
Jun 14 06:52:21 <sandsmark> this is my entire wpa-supplicant_%.bbappend: http://ix.io/xsc
Jun 14 06:52:55 <sandsmark> erh, ignore the rm -f, I was testing if it worked if I removed the extra .service files
Jun 14 06:53:14 <rick1> sandmark : Let me guess, you're somewhere in europe ...
Jun 14 06:53:20 <sandsmark> rick1: norway :)
Jun 14 06:53:59 <rick1> few thousands K away from me..:)
Jun 14 06:54:07 <rick1> I'm in Aus
Jun 14 06:54:09 <sandsmark> bah, it's probably easier to just ignore the systemd.bbclass and ln -s manually
Jun 14 06:54:11 <sandsmark> ah, nice
Jun 14 06:54:44 <sandsmark> oceania is one of the few regions of the world I haven't been in
Jun 14 06:56:09 <rick1> I've never been in Norway as well...
Jun 14 06:56:38 <LetoThe2nd> sandsmark: the replacement failure rings a bell, but i donÄt know which. maybe just put it onto the ml and do something else today, waiting for a response?
Jun 14 06:57:31 <sandsmark> I suck at putting a half-solved problem away, hard to concentrate on something else then :p
Jun 14 06:57:44 <sandsmark> but yeah, I should probably do that
Jun 14 06:57:53 <LetoThe2nd> sandsmark: :-)
Jun 14 06:59:45 <rick1> sandmark ==> any debug message from your WPA stuff ? I don't exactly but if you have debug stuff..echo "something" ?
Jun 14 07:00:30 <rick1> I don't exactly know..
Jun 14 07:01:25 <sandsmark> not sure where it would help to put debug output, the «wpa-supplicant/latest.pkg_postinst» shows the correct stuff
Jun 14 07:01:38 <sandsmark> maybe in the systemd.bbclass code that tries to parse stuff
Jun 14 07:01:49 <rick1> auto-enable wifi ==> do you have the driver running already ?
Jun 14 07:02:05 <sandsmark> yup
Jun 14 07:02:25 <rick1> and if you manually call / ping it's working isn't it ?
Jun 14 07:02:26 <sandsmark> I managed to get our odm to use a chip that's well-supported by the mainline kernel, luckily
Jun 14 07:02:45 <sandsmark> yeah, it works well if I start or enable wpa_supplicant manually on the device
Jun 14 07:03:01 <sandsmark> but I want an image from yocto I can just stuff into delta_generator without having to do manual stuff
Jun 14 07:04:36 <rick1> it's just my idea...can you create a service calling that device on /etc/systemd/system ..manually ?
Jun 14 07:04:56 <rick1> I can see some services in that directory...what I
Jun 14 07:05:16 <sandsmark> well, I could just ln -s the file manually in do_install_append(), but I wanted to do it The Right Way (TM)
Jun 14 07:06:08 <rick1> I see..
Jun 14 07:07:03 <rick1> so the image should have a script running that service ( auto wifi )
Jun 14 07:07:11 <rick1> correct me if I'm wrong..
Jun 14 07:08:13 <rick1> I'm thinking to put that "auto script" inside the image...but I don't know exactly what you want..
Jun 14 07:10:40 <rick1> sysctl status "auto wifi script" ?
Jun 14 07:11:48 <sandsmark> ah, systemd is basically like a script like that
Jun 14 07:12:46 <rick1> yup
Jun 14 07:14:01 <LetoThe2nd> sandsmark: to me it just sounds like the script name replacement doesn't work correctly. you can always bitbake -e the recipe to see what going on internally.
Jun 14 07:15:33 <rick1> it will be something :
Jun 14 07:15:39 <sandsmark> LetoThe2nd: ah, good idea
Jun 14 07:16:04 <rick1> [Service]
Jun 14 07:16:04 <rick1> ExecStart=your autowifi script
Jun 14 07:16:07 <rick1> [Install]
Jun 14 07:16:08 <rick1> WantedBy=default.target
Jun 14 07:16:53 <rick1> inside /etc/systemd/system
Jun 14 07:17:11 <sandsmark> rick1: yeah, I'm just using the default .service files that wpa_supplicant ships with, the problem is that for some reason the bitbake script that's supposed to automatically set them up for me fails to install it with the correct name
Jun 14 07:19:03 <rick1> ok, if you put a hardcode on your bitbake script, will it do it correctly ? probably you miss a variable ?
Jun 14 07:19:34 <rick1> it's getting complicated as it gets :)
Jun 14 07:20:46 <sandsmark> yeah, it will work correctly if I hardcode the ln -s, but it feels wrong :p
Jun 14 07:21:07 <rick1> scratch my head....
Jun 14 07:58:59 <Dvorkin> did anybody ever saw the message "[log_check] Failed to set locale, defaulting to C" while building rootfs ?
Jun 14 07:59:35 <Dvorkin> can't find the source of the problem. my image won't build at the final step
Jun 14 08:04:08 <Dvorkin> i've got message "[log_check] Failed to set locale, defaulting to C" at the final step of do_rootfs. anybody can help?
Jun 14 08:07:46 <bluelightning> Dvorkin: what host distro is this on? is it inside a container?
Jun 14 08:08:51 <Dvorkin> bluelightning, it's Mageia 4
Jun 14 08:08:59 <Dvorkin> my host locale is not EN
Jun 14 08:09:09 <bluelightning> Dvorkin: what's LANG set to?
Jun 14 08:09:33 <Dvorkin> ru_RU.UTF-8 I tryed en_US.UTF-8 without success
Jun 14 08:10:04 <bluelightning> is the en_US.UTF-8 locale installed? I'm not sure how that's handled on Mageia to be honest but it may be packaged separately
Jun 14 08:10:26 <bluelightning> it may be a sub-package of glibc perhaps
Jun 14 08:13:08 <bluelightning> (we ought to be checking to ensure whatever is needed is installed, so there is definitely a bug here, just a matter of figuring out where exactly)
Jun 14 08:17:53 <Dvorkin> bluelightning, yes, en_US is in stalled on my PC
Jun 14 08:18:41 <Dvorkin> as I said, I tryed to set LANG and LC_* to en_US without success
Jun 14 08:19:56 <bluelightning> kanavin: ^ google suggests this might be dnf complaining, any ideas?
Jun 14 08:20:05 <Dvorkin> I even can't find who sais that? dnf-native? maybe I should set some host packet to build with DNF?
Jun 14 08:20:54 <bluelightning> Dvorkin: just checking, is that the only failure message or are there other errors?
Jun 14 08:21:50 <Dvorkin> there are one more error, but it doesn't shown in console. I see it in the logfile:
Jun 14 08:22:28 <Dvorkin> NOTE: /etc/host.conf: line 3: bad command `nospoof on' /etc/host.conf: line 4: bad command `spoofalert on'  NOTE: /etc/host.conf: line 3: bad command `nospoof on' /etc/host.conf: line 4: bad command `spoofalert on' Added oe-repo repo from /home/dv/workspace/tpp.17/poky/x.tpp/tmp/work/tpp-tps-li Last metadata expiration check: 0:00:00 ago on Wed Jun 14 07:42:46 2017 UTC. Metadata cache created. Failed to set locale, defaulting to C   
Jun 14 08:23:05 <Dvorkin> but it doesn't shown in console. I see it in the logfile of do_rootfs
Jun 14 08:24:54 <Dvorkin> "Failed to set locale" appeared right after   Verifying        : kbd-keymaps-2.0.4-r0.cortexa8hf_neon               814/814    Installed: ....   wireless-regdb.cortexa8hf_neon 2017.03.07-AUTOINC............................    Complete! Failed to set locale, defaulting to C
Jun 14 08:25:59 <Dvorkin> It says Complete! after verying packages. and then immediately says "Fail" about locale.
Jun 14 08:30:50 <bluelightning> Dvorkin: I think the operation did succeed, it's just our code is assuming something failed and is failing do_rootfs as a result
Jun 14 08:31:42 <bluelightning> Dvorkin: if you run export LC_ALL="en_US.UTF-8" does it give any error?
Jun 14 08:32:16 <Dvorkin> looks like! trying...
Jun 14 08:32:50 <Dvorkin> the other strange thing - where NOTE: /etc/host.conf: line 3: bad command `nospoof on' comes from?
Jun 14 08:33:32 <Dvorkin> why it come from MY /etc/host.conf?
Jun 14 08:34:08 <bluelightning> that's in your /etc/host.conf?
Jun 14 08:34:40 <Dvorkin> definitely!
Jun 14 08:35:12 <Dvorkin> it's host PC's file.
Jun 14 08:36:03 <Dvorkin> after export LC_ALL="en_US.UTF-8" it goes the same way
Jun 14 08:38:26 <bluelightning> Dvorkin: yes but does setting that give an immediate error? that's what I was after
Jun 14 08:38:32 <bluelightning> or does it just set as normal?
Jun 14 08:39:06 <Dvorkin> no, I have en_US locale installed in my host PC
Jun 14 08:39:15 <Dvorkin> it doesn't give any error
Jun 14 08:39:20 <bluelightning> ok, thanks
Jun 14 08:39:27 <bluelightning> not sure where to go from here to be honst :(
Jun 14 08:39:33 <bluelightning> *honest
Jun 14 08:39:45 <bluelightning> kanavin: you around/
Jun 14 08:39:47 <bluelightning> ?
Jun 14 08:42:40 <Dvorkin> I comented out "nonspoof on" and "spoofalert on" in my host PC's /etc/host.conf, spoofalert error doen't appeare in my logfile but "Failed to set locale, defaulting to C" still exists
Jun 14 08:42:56 <bluelightning> right, the two appear unrelated
Jun 14 08:46:50 <bluelightning> RP: ^ any ideas about "Failed to set locale, defaulting to C" ?
Jun 14 08:53:59 <Dvorkin> the complete logfile: https://paste.fedoraproject.org/paste/Cr9IxtQWZ~DL6huZFoC8-A
Jun 14 08:58:19 <ant_work> Dvorkin, the error is not with locales
Jun 14 08:59:05 <ant_work> check line #8617 of your paste
Jun 14 08:59:56 <Dvorkin> ant_work, oh!
Jun 14 09:03:17 <Dvorkin> cheking...
Jun 14 09:05:36 <ant_work> maybe s/no/disable/ ?
Jun 14 09:08:29 <Dvorkin> ant_work, no. set to "disable" and problem still exists
Jun 14 09:12:00 <ant_work> the syntax is enable or disable, see couple of lines above
Jun 14 09:12:20 <Dvorkin> I know, thanks. I fixed, but problem still exist
Jun 14 09:12:22 <Dvorkin> https://paste.fedoraproject.org/paste/iKCW88IyuBqBizYm6QaSvg
Jun 14 09:12:28 <Dvorkin> the new logfile
Jun 14 09:13:52 <RP> bluelightning: no, sorry :(
Jun 14 09:14:23 <RP> I have to wonder which locale its trying to set...
Jun 14 09:15:23 <Dvorkin> I have ru_RU.UTF-8 locale on my host PC, but setting it to en_US.UTF-8 doesn't help
Jun 14 09:41:26 <lukma> Dear All,
Jun 14 09:41:26 <lukma> I'd like to ask for your opinion - what is the preferred way to add binary program (a few KiB) to yocto?
Jun 14 09:41:26 <lukma> Just treat it as other source? and properly install ?
Jun 14 09:42:50 <LetoThe2nd> lukma: here you go: http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#packaging-externally-produced-binaries
Jun 14 09:43:11 <LetoThe2nd> lukma: its a common usecase and properly documented :)
Jun 14 09:44:08 <lukma> LettoThe2nd: Thanks :-)
Jun 14 09:52:29 <bluelightning> Dvorkin: the only thing I would suggest at this point would be to file a bug and we'll try to investigate it
Jun 14 10:36:51 <Dvorkin> bluelightning, thanks. I'll try
Jun 14 10:48:07 <Dvorkin> the other question: how to get rid of busybox in my image completely ?
Jun 14 10:55:56 <sveneisenberg> Does anyone use 1.1.0 with yocto or openembedded? I'm looking for a recipe.
Jun 14 11:16:57 <Dvorkin> any ideas why do_rootfs failed? : https://paste.fedoraproject.org/paste/s~SuKagYWMH2wmJLB6ckFQ
Jun 14 11:25:40 <LetoThe2nd> Dvorkin: no immediate idea. but please no thread hijacking - your problem is absolutely not realted to libraries in the image/sdk
Jun 14 11:54:43 <kanavin> bluelightning: ?
Jun 14 11:56:47 <bluelightning> kanavin: any ideas regarding Dvorkin's locale issue within do_rootfs? it seems to be dnf that's complaining
Jun 14 11:56:48 <bluelightning> kanavin: http://www.yoctoproject.org/docs/current/dev-manual/dev-manual.html#new-recipe-writing-a-new-recipe
Jun 14 11:56:50 <bluelightning> er
Jun 14 11:56:53 <bluelightning> wrong paste
Jun 14 11:56:57 <bluelightning> kanavin: https://paste.fedoraproject.org/paste/iKCW88IyuBqBizYm6QaSvg
Jun 14 11:58:52 <kanavin> bluelightning: I do not have specific ideas; grep the source code to see where that error is coming from, and take it from there
Jun 14 11:58:58 <kanavin> Dvorkin: ^^^
Jun 14 12:05:53 <kanavin> Dvorkin: you need to check dnf/i18n.py and dnf/pycomp.py - they seem to do a straigforward call to locale.setlocale() though, but they obscure the actual error in an exception handler
Jun 14 12:06:57 <kanavin> so you should run the natively built python3 interpreter, issue the setlocale calls from the prompt, and see why they fail
Jun 14 12:07:20 <kanavin> that's all I can help you with, I'm afraid, as I'm not getting the same issue here
Jun 14 12:07:26 <kanavin> bluelightning: ^^^
Jun 14 12:08:02 <bluelightning> kanavin: ok, that is helpful, thanks
Jun 14 12:13:29 <Dvorkin> kanavin, thanks! already checked!
Jun 14 12:13:45 <kanavin> Dvorkin: so what was it?
Jun 14 12:14:14 <Dvorkin> the reason of the problem was at tmp/work/cortexa8hf-neon-tps-linux-gnueabi/dnf/2.3.0-r0/image/usr/lib/python3.5/site-packages/dnf/i18n.py
Jun 14 12:14:39 <kanavin> yeah, but why is setlocale failing?
Jun 14 12:14:43 <Dvorkin> it throws an exception while calling setup_locale()
Jun 14 12:16:01 <Dvorkin> in poky DNF-2.3.0 is used, althrough 2.5.1 is out.
Jun 14 12:17:06 <wooo> does anyone else have troubles fetching meta-arago via git? "git clone http://arago-project.org/git/meta-arago.git" gives me "fatal: read error: Connection reset by peer"...
Jun 14 12:17:39 <bluelightning> Dvorkin: what's the exception?
Jun 14 12:18:01 <Dvorkin> and it differs in this part: function setup_locale() sets locale.LC_ALL to '', instead of LC_ALL to '' and LC_TIME to "C" in youcto DNF version
Jun 14 12:18:36 <Dvorkin> I don't see the message of exception yet, need to debug more...
Jun 14 12:19:38 <kanavin> the need for an explicit setlocale call is the dumbest idea ever, codified in POSIX
Jun 14 12:23:52 <Dvorkin> it want to set explicit LC_TIME for logging, afaik. but why it escalates rather "soft" error to do_rootfs ?
Jun 14 12:24:28 <bluelightning> Dvorkin: we have some logic that just looks for things like "error" "fail" etc. in the log, that's log_check
Jun 14 12:25:05 <Dvorkin> I understand that... Is there a way to exclude such message from checker?
Jun 14 12:25:22 <bluelightning> if you hack the code, yes... exclusions are hardcoded unfortunately
Jun 14 12:25:32 <bluelightning> triggering in this case is probably overkill, but I'd rather just have people fix their systems if that's possible
Jun 14 12:26:34 <bluelightning> if you do bitbake -c devshell your-image, run python3 and then try to call setlocale() like the code does, do you get the same issue?
Jun 14 12:32:56 <Dvorkin> empty locale: locale.Error: unsupported locale setting
Jun 14 12:33:22 <Dvorkin> locale.setlocale(locale.LC_ALL, 'C') is OK
Jun 14 12:34:05 <Dvorkin> locale.setlocale(locale.LC_ALL, '') : locale.Error: unsupported locale setting
Jun 14 12:34:47 <Dvorkin> is something wrong in my image settings?
Jun 14 12:37:00 <kanavin> Dvorkin: something is wrong on your host machine. Can you check what are the values of environment variables LANG, and LC_* (all of them)?
Jun 14 12:37:31 <kanavin> env | grep LANG
Jun 14 12:37:35 <kanavin> env |grep LC_
Jun 14 12:37:47 <Dvorkin> env | grep LC : LC_SOURCED=1 LC_ALL=ru_RU.UTF-8
Jun 14 12:38:10 <Dvorkin> env | grep LANG : LANG=ru_RU.UTF-8 LANGUAGE=ru_RU.UTF-8:ru
Jun 14 12:38:27 <kanavin> Dvorkin: unset LC_ALL
Jun 14 12:39:09 <Dvorkin> [dv@localhost x.tpp (master)]$ export LC_ALL="" [dv@localhost x.tpp (master)]$
Jun 14 12:39:14 <Dvorkin> no errors
Jun 14 12:39:38 <kanavin> setting LC_ALL is a bad idea in general...
Jun 14 12:42:08 <Dvorkin> export LC_ALL="" then running bitbake gives same error, as I expected.
Jun 14 12:42:32 <Dvorkin> something is wrong with python locale setting in image. I won't set empty locale
Jun 14 12:42:41 <Dvorkin> something is wrong with python locale setting in image. It won't set empty locale
Jun 14 12:42:52 <kanavin> Dvorkin: does the same call succeed from the host python3?
Jun 14 12:43:31 <Dvorkin> it does:
Jun 14 12:43:35 <kanavin> 'setting empty locale' is actually 'setting locale to what the locale environment variables specify' :-)
Jun 14 12:43:36 <Dvorkin> >>> locale.setlocale(locale.LC_ALL, '') 'ru_RU.UTF-8'
Jun 14 12:44:12 <Dvorkin> at my host PC python I got success reply from python3
Jun 14 12:44:17 <Dvorkin> 'ru_RU.UTF-8'
Jun 14 12:44:48 <Dvorkin> so... it means, I need to set default locale in the image somehow?
Jun 14 12:45:42 <Dvorkin> or I need to add some localization package to my image ?
Jun 14 12:45:53 <Dvorkin> for python, i mean
Jun 14 12:47:05 <kanavin> Dvorkin: nope, this is a problem with native python, and is not about target images
Jun 14 12:47:56 <Dvorkin> but native python runs ok...
Jun 14 12:48:07 <kanavin> Dvorkin: host python and native python are two different things :)
Jun 14 12:48:14 <Dvorkin> ah, sorry :)
Jun 14 12:48:15 <kanavin> Dvorkin: host python runs ok, native python does not
Jun 14 12:48:48 <Dvorkin> looking at this https://ask.fedoraproject.org/en/question/87599/hi-i-have-a-problem-which-is-failed-to-set-locale-defaulting-to-c/
Jun 14 12:48:56 <kanavin> so you can make a tiny script with just that one line, and run it like this:
Jun 14 12:48:57 <kanavin> strace python3 <script>
Jun 14 12:49:01 <Dvorkin> I understand that it could be package problem
Jun 14 12:49:17 <kanavin> then try to find in the output which locale definition files the two versions of python try to access
Jun 14 12:49:20 <kanavin> and where is the difference
Jun 14 12:49:39 <kanavin> looks like native python is not able to access the locale definition for the ru_RU
Jun 14 12:52:43 <Dvorkin> kanavin, strace says, python tryes to find default locale in host PC.
Jun 14 12:53:45 <Dvorkin> at start
Jun 14 12:54:05 <kanavin> at some point it should try to find the ru_RU stuff
Jun 14 12:55:19 <Dvorkin> it does not
Jun 14 12:55:47 <Dvorkin> at startup it searches for en* in host PC
Jun 14 12:59:47 <kanavin> Dvorkin: but one version of python gives you an error on this script, and another does not
Jun 14 12:59:57 <kanavin> Dvorkin: so there has to be a difference in what strace outputs
Jun 14 12:59:58 <Dvorkin> Native: print(locale.getdefaultlocale()) ('en_US', 'UTF-8')
Jun 14 13:00:08 <kanavin> Dvorkin: the tricky part for you is to find it :)
Jun 14 13:00:28 <Dvorkin> looks like I have to install en_US locale to my PC
Jun 14 13:03:00 <Dvorkin> I have everything... omg. why python can't find it?
Jun 14 13:04:38 <Dvorkin> why it searches for locales in /usr/lib/locale/, while I have it in /usr/share/locale ?
Jun 14 13:07:13 <kanavin> Dvorkin: I have to go now :) keep digging!
Jun 14 13:07:44 <Dvorkin> oh, I just made a symbolic link from /usr/share/locale to /usr/lib in my host PC and it built.
Jun 14 13:08:04 <Dvorkin> that's the subject to note somewhere :)
Jun 14 13:08:41 <kanavin> Dvorkin: I do believe it should look in /usr/share though
Jun 14 13:08:57 <kanavin> would be appreciated if you find out why it doesn't
Jun 14 13:18:46 <yohboy> Hi there, someone uses CLion from JetBrains with poky sdk ?
Jun 14 13:45:19 <fl0v01> is there a simple way to run a command whenever X dpms is disabled or enabled?
Jun 14 13:46:21 <fl0v01> i want to unmap a window so it doesnt receive the first touch event when the screen is turned back on again by a touch
Jun 14 13:48:23 <rburton> fl0v01: i'm surprised apps are seeing the first event
Jun 14 13:49:16 <fl0v01> im just using the bare X11 dpms with xset
Jun 14 13:49:47 <fl0v01> i wondered if just installing a screensaver would help too
Jun 14 13:50:14 <fl0v01> but i thought that the blanking actually works like a screensaver
Jun 14 14:09:41 <rburton> not really
Jun 14 14:09:46 <rburton> blanking is different
Jun 14 14:13:16 <MarcWe> hi, my qt framwork is not handeling touch events. but libinput-debug-events shows them
Jun 14 14:28:42 <MarcWe> when i run a qt Quick application ( default generated by qt ) im getting file:///usr/lib/qt5/qml/QtQuick/Controls/MenuBar.qml:155: Error: Cannot assign [undefined] to int but when i mous click in the application the event is handled
Jun 14 14:43:14 <lukma> Is there any convenient way to
Jun 14 14:43:14 <lukma> make the arm-poky-gnueabi to become HF version (arm-poky-gnueabihf )?
Jun 14 14:43:14 <lukma> Is it enough to just extend TOOLCHAIN_OPTIONS += "--with-float=hard --with-fpu=vfpv3-d16 --with-mode=thumb --with-tune=cortex-a9 --with-arch=armv7-a"
Jun 14 14:43:22 <lukma> or is there any easier way?
Jun 14 14:45:49 <LetoThe2nd> lukma: FWIW the machine configuration should set that how it wants, e.g. the DEFAULTTUNE
Jun 14 14:46:07 <lukma> DEFAULTTUNE = "cortexa5thf-neon"
Jun 14 14:46:12 <lukma> This is what I do have
Jun 14 14:46:18 <lukma> and then
Jun 14 14:47:49 <lukma> DISTRO_VERSION    = "2.3"
Jun 14 14:47:49 <lukma> TUNE_FEATURES     = "arm armv7a vfp thumb neon callconvention-hard cortexa5"
Jun 14 14:47:49 <lukma> TARGET_FPU        = "hard"
Jun 14 14:48:07 <LetoThe2nd> so whats the problem?
Jun 14 14:48:09 <lukma> And it looks like we have it correct
Jun 14 14:48:27 <LetoThe2nd> vfp, neon, callvention hard. exactly, looks all good.
Jun 14 14:50:07 <lukma> LetoThe2nd: https://pastebin.com/F3gEpXw3
Jun 14 14:50:30 <lukma> The toolchain built with meta-toolchain lacks following switches:
Jun 14 14:50:54 <lukma> --with-float=hard --with-fpu=vfpv3-d16 --with-mode=thumb --with-tune=cortex-a9 --with-arch=armv7-a
Jun 14 14:51:28 <LetoThe2nd> lukma: its mainly about the vfpv3-d16
Jun 14 14:52:02 <lukma> (probably by default the toolchain has --with-fpu=neon)
Jun 14 14:54:53 <LetoThe2nd> lukma: although i'm actually not seeing the direct point. it looks like the source is trying to compile some asm (-> the lowlevel_init.S, part of u-boot? or what?), and the compiler is declining it. IIRC i had something similar with the valgrind test suite quite a while back: it worked for arm-vfp/neon, but not for vfpv3-d16 on cortex-a5
Jun 14 14:59:12 <lukma> populate_sdk is responsible for building toolchain
Jun 14 14:59:40 <lukma> LetoThe2nd: I will add TOOLCHAIN_OPTIONS to my distro.conf file
Jun 14 14:59:49 <lukma> and I will see if it works
Jun 14 15:00:21 <LetoThe2nd> lukma: no, populate_sdk is not responsible for the toolchain. populate_sdk is responsible for packing things up into a sdk.
Jun 14 15:04:59 <lukma> LetoThe2nd: Hmm.... so there is any other recipe which is building gcc......
Jun 14 15:05:14 <lukma> but this is strange since meta-toolchain only inherit populate_sdk
Jun 14 15:06:10 <LetoThe2nd> lukma: unless your sdk machine and the build host do not differ, the toolchain should be built anyways for the desired image, and then just reused for the sdk
Jun 14 15:07:09 <LetoThe2nd> what is it good for anyways? just do bitbake -c populate_sdk $YOURWANTEDIMAGE
Jun 14 15:07:32 <rburton> meta-toolchain predates proper SDKs and we should remove it
Jun 14 15:07:59 <LetoThe2nd> rburton: my gut feeling said that too but i wasn't certain enough to actually say it ;-)
Jun 14 15:08:07 <rburton> :)
Jun 14 15:08:17 <lukma> So where should I look for the toolchain
Jun 14 15:08:28 <LetoThe2nd> 17:07 < LetoThe2nd> what is it good for anyways? just do bitbake -c populate_sdk $YOURWANTEDIMAGE
Jun 14 15:08:40 <LetoThe2nd> there you go :)
Jun 14 15:08:40 <lukma> to modify both - SDK and the one which builds the actual package (like u-boot)?
Jun 14 15:13:25 <lukma> LetoThe2nd: Ok so I'm sure:
Jun 14 15:13:25 <lukma> export CC="arm-poky-linux-gnueabi-gcc  -march=armv7-a -marm -mfpu=neon  -mfloat-abi=hard -mcpu=cortex-a5 --sysroot=
Jun 14 15:13:34 <lukma> the -mfpu=neon
Jun 14 15:14:16 <LetoThe2nd> lukma: where do you see that? you really are lacking context.
Jun 14 15:14:46 <MarcWe> lukma: sources/oe-core/meta/conf/machine/include/
Jun 14 15:15:10 <lukma> bitbake -e virtual/bootloader | grep "arm-poky-linux-gnueabi-"
Jun 14 15:15:15 <lukma> this the output from it
Jun 14 15:17:59 <LetoThe2nd> i'm on a cortexa5 too, it looks totally the same here and i don't have any problems with it.
Jun 14 15:18:27 <LetoThe2nd> plus, a few minutes ago you mentioned: --with-tune=cortex-a9
Jun 14 15:18:53 <LetoThe2nd> which certainly is wrong then. so my guess is that your machine file or its use is buggy in some way.
Jun 14 15:19:13 <lukma> cortex-a9 was from the original linaro toolchain
Jun 14 15:19:16 <lukma> which works
Jun 14 15:19:47 <lukma> (I used linaro toolchain -> arm-linux-gnueabihf) with the source code to see if this is a toolchain issue
Jun 14 15:20:39 <LetoThe2nd> lukma: and what is the actual hardware in the end?
Jun 14 15:21:04 <lukma> The HW is cortex-a5
Jun 14 15:22:25 <lukma> The difference between toolchains:
Jun 14 15:22:26 <lukma> --with-fpu=vfpv3-d16 ->working
Jun 14 15:22:26 <lukma> -mfpu=neon  -> built by OE
Jun 14 15:24:55 <khem> RP: sent the PIE rework to list
Jun 14 15:27:13 <LetoThe2nd> lukma: i can't put the finger on it, but i still don't believe that its the toolchain itself that causes the problem.
Jun 14 15:29:40 <fray> you need a tune for what you are doing.  with -with-fpu=vfpv3-d16 will avoid neon instructions and instead use only d16 style vfpv3 instructions..
Jun 14 15:29:58 <fray> the -mfpu=neon will use neon and the assumed set of vfpv4? and related instructions..
Jun 14 15:30:36 <fray> if your CPU is lacking instructions (vendor removed say NEON, when normally that processor would have it), then you will need to define a new tune for your BSP to use, or you will need to pick a different (existing tune) that is compatible with the options on your system
Jun 14 15:39:31 <stephano> y
Jun 14 15:49:27 <RP> khem: thanks. We merged gcc7 (many thanks for the work on that!)
Jun 14 15:51:12 <kergoth> Hmmm
Jun 14 15:52:35 <tlwoerner> chromium isn't happy with gcc-7 :-(
Jun 14 15:55:26 <georgem> tlwoerner: it seems to break anytime there is a gcc change
Jun 14 15:56:52 <tlwoerner> georgem: it likes to keep me busy
Jun 14 16:02:14 <Crofton|work> it keeps tlwoerner from spending all his time on horses
Jun 14 16:02:30 <tlwoerner> Crofton|work: :-)
Jun 14 16:04:20 <Saur> RP: I now have a set of patches which hopefully fix the problems you got on the autobuilders due to my change for automatic file dependencies. However, it is hard to tell if I have fixed them all since many failures are the same between the various builds. So I was hoping that if I send the stack of changes to the list, that you could take them through another run on the the autobuilders.
Jun 14 16:04:46 <RP> Saur: sounds good thanks. The patch won't make M1 but happy to try testing it once we get that sorted out
Jun 14 16:05:56 <kergoth> bluelightning: Hmm, thoughts on how to get a full task graph out of tinfoil? you can generate the .dot files from it, or fire a generated graph *event*, but not just retrieve the graph data, and we don't have direct access to either taskdata or runqueue, only the original data from the recipecache. guess i could try registering an event handler that references a global to get the data, or just patch command.py to add a new command for it
Jun 14 16:05:57 <Saur> RP: Ok, I'll send them of as soon as I can. I am still trying to find a more generic solution to one of the problems...
Jun 14 16:06:48 <kergoth> well, clearly the issue is more with the available commands for interacting with the server than with tinfoil itself, but you get the point
Jun 14 16:06:49 <kergoth> hmm
Jun 14 16:06:49 <bluelightning> kergoth: kind of sounds like we'd need a command to get that really
Jun 14 16:07:10 <kergoth> seems like a pretty basic thing, i want to take some targets, with or without :<task>, and be able to examine the tasks that would be run to build them
Jun 14 16:07:12 <bluelightning> I certainly wouldn't object to adding one
Jun 14 16:07:41 <bluelightning> kergoth: BTW I am in the process of adding some interfaces to tinfoil to allow running tasks and more easily enumerate recipes
Jun 14 16:07:49 <kergoth> k, will give that a go, thanks for the thoughts
Jun 14 16:07:55 <bluelightning> one sec and I'll push it somewhere just so you can see what I am up to
Jun 14 16:07:57 <kergoth> cool, sounds useful
Jun 14 16:08:56 <bluelightning> http://git.yoctoproject.org/cgit/cgit.cgi/poky-contrib/log/?h=paule/fixes2d
Jun 14 16:09:50 <bluelightning> as you can probably tell the main motivation was removing some of the performance-impairing evility in the devtool code due to the lack of the ability to run a real build
Jun 14 16:11:27 * kergoth nods
Jun 14 16:11:37 <kergoth> looking very promising, nice work so far
Jun 14 16:14:14 <kergoth> bluelightning: only thought so far is might also want to add variants or a new method to interact with non-preferred providers, i.e. to gather info about the universe rather than the world
Jun 14 16:14:20 <kergoth> but that can always be a future enhancement, of course
Jun 14 16:18:43 <kergoth> Having interacted with taskdata on a number of occasions for bb commands and the like, I have to say I hate how late in the process the bitbake code converts the countless pieces of dependency info into task dependencies. having to spawn a runqueue just to map i.e. build targets to task deps w/ BB_DEFAULT_TASK for example.. i'd like to see an intermediate method that takes all the bits of recipecache data and emits a task graph, ideally with an option
Jun 14 16:18:44 <kergoth>  to record the origin of each task dep, and then make runqueue use that
Jun 14 16:18:57 <kergoth> also, the taskdata api is really weird
Jun 14 16:20:43 <kergoth> Hmm, maybe I'll take a gander at prototyping such a thing as part of adding the new cooker command i want
Jun 14 16:21:00 <kergoth> would be ideal for a more capable whatdepends command/tool as well
Jun 14 16:21:03 <bluelightning> yeah, I agree FWIW
Jun 14 16:21:21 <kergoth> and with the origin info, we could add notes to the nodes/edges of the dot graph with info on why something was pulled in
Jun 14 16:21:25 * kergoth ponders
Jun 14 16:28:35 <jcelerier> hello :)
Jun 14 16:29:03 <jcelerier> does someone know if it's easy to get a yocto project running on "commodity hardware" tablets
Jun 14 16:29:08 <jcelerier> e.g. samsung galaxy tab, etc...
Jun 14 16:29:50 <kergoth> writing a bitbake-layers command to take a list of targets, with or without :<task> suffixes, generate the full task dependency info, gather up all the do_fetch tasks that will be run, use that to get a list of recipe and bbappend filenames, then parse all of them with variable tracking enabled, examine the history of SRC_URI using the convenience method on VariableHistory, and use that to map all the urls / downloaded files in DL_DIR to the layer that
Jun 14 16:29:51 <kergoth>  added them to SRC_URI
Jun 14 16:30:35 <bluelightning> kergoth: interesting... certainly a good use case for tinfoil
Jun 14 16:31:25 <kergoth> that sort of thing is useful for constructing installers that ship downloads, depending on the installer in question and how the artifacts are broken up
Jun 14 16:32:40 <kergoth> I wanted to do it with a task, which would simplify matters, since the task could do the work for the recipe it's run in, but it's non-trivial to re-parse the recipe and its appends with variable tracking enabled from inside a task :)
Jun 14 16:32:44 <kergoth> but this should do
Jun 14 18:57:56 <machinehum> Does anyone have experiance with the iMX6?
Jun 14 19:37:00 <seebs> so it turns out my clever idea of "just increase openfiles limit" won't work, because fd_set objects are capped at, by default, 1024 anyway.
Jun 14 19:38:01 <seebs> Next up: Benchmarking for the FASTOP case because I think it may need to go if we don't want continued reliability problems.
Jun 14 19:38:30 <fray> I seem to remember FASTOP was a huge win.. the round-trip case was very slow
Jun 14 19:38:34 <fray> (or is this different)
Jun 14 19:39:04 <seebs> That is the case, but I don't actually have concrete numbers.
Jun 14 19:39:20 <fray> yuo did at one point, but it's long enough ago definitely worth it again
Jun 14 19:39:32 <khem> RP: I pushed 11th patch into the pull request which upgrades and fixes webkit for ppc with hardening flags
Jun 14 19:39:36 <seebs> I do know that when I added actual profiling, I found that the wall-clock time for round trips with actual database access was *smaller* than the cost of some of the path processing.
Jun 14 19:39:47 <fray> can you do the equiv of the fast-op on the -server- side?  ACK the connection then just immediately send the reponse showing you got it?
Jun 14 19:40:03 <khem> machinehum: sure, do you have questions ?
Jun 14 19:40:11 <seebs> Not really. The reason fastop is a performance win is that you don't even have to wait for the transmission to have completed at all.
Jun 14 19:40:29 <seebs> You don't even care whether the kernel has successfully made the data available to the other side yet, you're already done.
Jun 14 19:40:38 <seebs> The actual time to process things is comparatively trivial, I believe.
Jun 14 19:40:43 <fray> k
Jun 14 19:41:04 <seebs> We've had Lots Of Events which is why a couple of these things have been sitting.
Jun 14 19:41:10 <fray> my memory was that the data 'did something' round trip was more signifciant that the message sent and response received part
Jun 14 19:41:23 <fray> but I know there have been a lot of changes in that area since you did the original work..
Jun 14 19:41:27 <fray> may not be much of an issue any longer
Jun 14 19:41:42 <seebs> Also, on reflection... That would still have the same basic problem of data reliability.
Jun 14 19:41:59 <seebs> The point of the round trips was to have an actual assurance that either the operation had *actually* succeeded, or you'd know to retry.
Jun 14 19:42:10 <fray> yup
Jun 14 19:44:16 <khem> rburton: http://git.openembedded.org/openembedded-core-contrib/log/?h=kraj/hardening-fixes more trouble for you :)
Jun 14 19:44:26 <machinehum> khem: thanks! I think I might have found my solution
Jun 14 19:45:00 <khem> machinehum: very good cool
Jun 14 19:54:52 <seebs> fastop is definitely noticeable on things that are heavily reliant on a ton of operations.
Jun 14 19:57:15 <seebs> 769.8070 msec wrapper time, 413.7500 msec op time, 399.4210 msec IPC time.
Jun 14 19:57:19 <seebs> 1281.6980 msec wrapper time, 961.4040 msec op time, 948.2140 msec IPC time.
Jun 14 19:57:23 <fray> I guess another option would be to batch the messages to the other side 'in a fastop' replacement.. and process them all as a unit?  (might not actually help)
Jun 14 19:57:26 <seebs> so, yes, quite a lot slower internally.
Jun 14 19:57:53 <seebs> Although... It's significantly less noticeable as more and more of the workload isn't actually *specifically* testing pseudo heavily.
Jun 14 19:57:53 <fray> ya, and it's all clearly in the IPC side there.. (which presumably includes backend processing)
Jun 14 19:58:19 <seebs> yup.
Jun 14 19:59:05 <fray> I'm curious how much is IPC and how much is backend processing..  (usually IPC is quick)
Jun 14 20:00:06 <fray> it's too bad you can't do something like send IPC to server.. keep processing (client) and before the client terminates (atexit) get responses and deal with retry's.. but that might not resolve any behavior issues.. :/
Jun 14 20:00:28 <fray> it's the blocking behavior more then anyting during the op that I suspect is the 'noticable' piece
Jun 14 20:01:23 <fray> I know at one point we were considering using xattrs (when possible) to avoid the backend DB processing..
Jun 14 20:01:32 <fray> is that something that could be resurrected?
Jun 14 20:05:40 <machinehum> Does anyone use Yocto on Cygwin?
Jun 14 20:06:34 <seebs> the xattr feature exists but I don't think it turned out to be worth it, because xattrs were slow.
Jun 14 20:06:50 <seebs> And it looks like just acking immediately about splits the difference:
Jun 14 20:06:59 <seebs> fastop: 708.7490 msec wrapper time, 391.6180 msec op time, 380.1340 msec IPC time.
Jun 14 20:07:07 <seebs> full round trip: 1281.6980 msec wrapper time, 961.4040 msec op time, 948.2140 msec IPC time.
Jun 14 20:07:17 <seebs> instant ack for fastops: 1008.8720 msec wrapper time, 682.4250 msec op time, 668.4860 msec IPC time.
Jun 14 20:07:28 <seebs> that might be worth it to get the reliability back.
Jun 14 20:08:43 <seebs> so, untar of /usr/include over empty space, 0.49s fastop, 0.55s fakeop, 0.65s full round trip.
Jun 14 20:08:58 <seebs> With an existing copy, it's 0.61, 0.76, 0.91.
Jun 14 20:09:38 <seebs> But if we do that, we can just go ahead and randomly shut down live connections on clients and not expect things to get lost, which *should* handle the openfiles thing.
Jun 14 20:09:50 <seebs> Although I still really want to know how we're ending up with >1k simultaneous connections to a single pseudo server.
Jun 14 20:10:08 <seebs> That's >1k processes on just one recipe, basically, and that seems high.
Jun 14 20:10:20 <fray> ya, it has to be some massively parallel set of operations all running under pseudo control..
Jun 14 20:10:37 <fray> each one has a smaller set, but you end up with 1000 connects at a time.. (most likely the clients have already shut down by that point) etc..
Jun 14 20:11:07 <fray> think of something like kernel or toolchain build on a machine with unbounded CPU/threads.. I could see it happen..
Jun 14 20:11:27 <fray> and wouldn't it be 500 connections, two fds one send, one recieve?
Jun 14 20:13:26 <seebs> no, fds are bidirectional.
Jun 14 20:13:39 <seebs> Pseudo has a single fd per client.
Jun 14 20:13:48 <seebs> And in *theory* if they've shut down, the connections should be closed.
Jun 14 20:14:03 <fray> maybe with the fastop the client doesn't get the shutdown and it zombies?
Jun 14 20:14:25 <seebs> huh
Jun 14 20:14:36 <seebs> no, server doesn't do it. but.
Jun 14 20:14:42 <seebs> i just remembered a comment from many years ago
Jun 14 20:15:04 <seebs>                                 if (FD_ISSET(clients[i].fd, &events)) {
Jun 14 20:15:05 <seebs>                                         /* this should happen but doesn't... */
Jun 14 20:15:05 <seebs>                                         close_client(i);
Jun 14 20:16:03 <seebs> so... what if the real problem isn't that there's 1k processes open, but that for some reason we weren't hitting this?
Jun 14 20:16:14 <seebs> ... Actually that raises an obvious question.
Jun 14 20:16:24 <seebs> When a build hangs, do we *actually* have 1k processes which are waiting on pseudo?
Jun 14 20:17:50 <seebs> Because it could be that what's really wrong is that I never fixed this.
Jun 14 20:18:15 <seebs> ...
Jun 14 20:18:16 <seebs> dammit.
Jun 14 20:18:23 <seebs> so something like 10 years ago I misread a man page.
Jun 14 20:19:05 <seebs> Hmm. But it looks like select() should regard them as "readable", but with zero bytes available.
Jun 14 20:19:08 <seebs> So my comments are wrong.
Jun 14 20:22:52 <RP> seebs: I did the numbers and we can end up in a situation where 1k isn't so crazy. My server has 88 cores which means BB_NUMBER_THREADS=88*2=176. In a single recipe, do_package_write_{ipk|deb|rpm}|do_package_qa can all run together, each building or checking packages. For something like perl that could be 176 packages in parallel. 176*4 = 704 so you don't have much headroom against 1k
Jun 14 20:23:27 <RP> seebs: now, yes, this is a bit pathological and perhaps we shouldn't do this but its certainly possible it can run into trouble...
Jun 14 20:24:05 <seebs> yeah
Jun 14 20:24:30 <seebs> When you get a chance, could you check to see whether that's actually what's happened?
Jun 14 20:24:43 <seebs> Because I'm suspicious that my close_client logic is actually wrong, although I think it should work anyway.
Jun 14 20:24:58 <seebs> But the event thing is just plain wrong, I somehow got the impression that a closed socket was an exception, and it's not.
Jun 14 20:32:49 <seebs> Hmm. So it looks like it should be successfully closing sockets. But just in case, I'm checking more carefully.
Jun 14 20:33:21 <seebs> ... ahh, it turns out that some Unixes, but not Linux, flag "closed" in the exception set in a select.
Jun 14 20:34:16 <seebs> Anyway, I think more data would be needed to be sure, but I think the workaround of having FASTOP still send things, just not waiting for database ops before responding, will allow us to deal with Too Many Clients more gracefully.
Jun 14 20:39:12 <RP> seebs: I will try and reproduce those hangs again, see if I can get more info
Jun 14 20:44:03 <seebs> Because it does look like, if things are actually shutting down, they should indeed be successfully getting noticed.
Jun 14 20:44:42 <seebs> hmm.
Jun 14 20:54:15 <RP> 22G of virtual memory for rpmbuild when building python3. I suspect we may need to look at that...
Jun 14 20:54:41 <RP> heh, then top switched to display in TB
Jun 14 20:59:32 <seebs> oh, heh. i have had a bug roughly forever wherein the shutdown requests from pseudo -S were producing an "uh-oh" message which turned out to be harmless, but it revealed a flaw.
Jun 14 21:04:38 <fray> RP that 22G, is that after the openMP change?
Jun 14 21:05:17 <fray> I know that RPM (and especially bdb that it uses) mmaps files.  So it turns out that it can report a TON of virtual memory used, but in reality very little is actually allocated.
Jun 14 21:05:43 <fray> the mmap to 'write out files', turns out to be a lot faster then lots of open/writes/seeks/etc..
Jun 14 21:05:48 <fray> (and is more memory efficient as well)
Jun 14 21:25:29 <RP> seebs: a hung set of processes: https://pastebin.com/NDiZJtMS
Jun 14 21:26:44 <RP> seebs: a bit more detail of a subset https://pastebin.com/imQumx7Y
Jun 14 21:27:08 <RP> seebs: at ~500 instances each spawning pipelines, I can believe we deadlocked :/
Jun 14 21:27:26 <RP> fray: good question
Jun 14 21:28:00 <RP> fray: yes, its after openmp
Jun 14 21:28:52 <fray> ya, so my guess is that all of the open rpms it's writing are mmaped so they can be written out.. causing the larger memory space..
Jun 14 21:29:08 <fray> (does YP even support 32-bit hosts anymore, if it does -- this could be a problem)
Jun 14 21:29:35 <RP> fray: entirely possible. And yes, in theory we do support 32 bit still
Jun 14 21:30:01 <denix> what happened to $ORIGIN-relative RPATHs in nativesdk binaries? did I miss something couple releases back? :)
Jun 14 21:30:35 <fray> be interesting to test that
Jun 14 21:30:59 <fray> in the past rpm operations were setup that they would only mmap for the memory SPACE avaialble, but I don't know how (or if) the openmp changes affects that
Jun 14 21:32:55 <RP> denix: they should have them? we do relocate the toolchain at install time though
Jun 14 21:33:32 <RP> seebs: since I have this hanging here, anything else it would be good to know?
Jun 14 21:36:25 <denix> RP: so, should the relocated toolchain binaries have full path in RPATH or $ORIGIN/../bin?
Jun 14 21:37:19 <khem> it should be relative to ORIGIN
Jun 14 21:42:36 <denix> oops, I meant $ORIGIN/../lib not bin
Jun 14 21:43:43 <denix> khem: exactly. I don't see those any more for some reason...
Jun 14 21:44:45 <denix> this is one of the old toolchain binaries as reported by readelf - (RPATH) Library rpath: [$ORIGIN/../lib:$ORIGIN/../../lib]
Jun 14 21:47:06 <denix> and I don't see RPATH in new toolchain binaries... hence the question - is that expected?
Jun 14 22:01:34 <seebs> RP: Nothing obvious, but it occurs to me in retrospect that it would be neat to run with PSEUDO_DEBUG=v, and look at pseudo.log and see how many clients it's reporting. But it does look like we are actually hitting 1k processes, so.
Jun 14 22:01:49 <seebs> The interesting thing is that it doesn't appear to be able to resolve *any* of them, though?
Jun 14 22:01:51 <seebs> And that's very odd.
Jun 14 22:02:20 <machinehum>  Does anyone have any good references for cross-compiling using Yocto?
Jun 14 22:02:36 <machinehum> Or just a good jumping off point
Jun 14 22:03:32 <seebs> Like, it's not enough for *one* chain to be deadlocked. *all* the chains have to be deadlocked, or at least one of them ought to be continuing to process things until it finishes.
Jun 14 22:05:01 <seebs> *thinks* Something is weird here. I can't figure out how this would end up with *nothing* progressing. And if even one chain is able to progress, it should be able to free up file descriptors.
Jun 14 22:05:21 <seebs> It might be interesting to attach to various such processes with a debugger and see where they are.
Jun 14 22:05:45 <seebs> Are they all stuck either on a pipe read/write to another thing, or waiting on pseudo to open a connection?
Jun 14 22:45:04 <RP> seebs: I have wondered why it can't resolve any :/
Jun 14 22:45:30 <RP> denix: We started removing pointless RPATHS oif that helps explain things?
Jun 14 22:48:23 <RP> seebs: seems so far to be a lot in connect() or read from pipes
Jun 14 22:49:30 <RP> seebs: in fact they're all blocked on connects or reads from other processes in their subtrees :/
Jun 14 22:51:01 <RP> seebs: I suspect its the way I've parallelised the packaging code, it creates "the perfect storm"
Jun 14 22:51:47 <RP> seebs: many things which create long pipelines each of which use a connection and end up doing heavy IO followed by new processes. Eventually they all end up in new processes
Jun 14 23:01:44 <seebs> Hmm. That makes some sense; if they're all starting stuff and filling up the tree, and then they all try to start new processes.
Jun 14 23:02:01 <seebs> But it still seems odd that we'd have that many simultaneous clients.
Jun 14 23:02:18 <seebs> Might be instructive to have pseudo have something that causes it to dump a client list.
Jun 14 23:09:48 <fray> seebs, think of a make that spawns processes, that spawn processes.. (repeat) a few iterations and you quickly get to 1000 processes
Jun 14 23:09:58 <fray> since the connection opens and doesn't close.. you get stuck there
Jun 14 23:18:44 <RP> seebs: could pseudo favour taking new connections compared to servicing existing ones?
Jun 14 23:24:37 <RP> seebs: one other thought. In the EMFILE case, perhaps we could start expiring connections in the "no activity for say 1s and hit EMFILE"?
Jun 14 23:25:58 <RP> seebs: If we're seeing no activity on the connections for a period of time, it becomes very unlikely there would be any data loss since we'd see data in select() and whilst its not perfect, I think that maybe good enough to keep fastop?
Jun 14 23:26:28 * RP -> Zzzz
Jun 14 23:27:12 <denix> RP: thanks
Jun 14 23:28:12 <denix> RP: would that cause binaries to use host libraries instead?
Jun 14 23:35:40 <seebs> It can't favor taking new connections to serving existing ones, because the existing ones already exist and are taking up files.
Jun 14 23:35:59 <seebs> I'm currently just trying to figure out whether something more obvious is wrong.
Jun 14 23:36:13 <seebs> And I actually sort of think we should possibly kill fastop because it's been behind a number of actual data loss issues now.
Jun 14 23:39:16 <denix> RP: btw, I assume you are referring to http://cgit.openembedded.org/openembedded-core/commit/?id=6b94569d5d8e8bdd575d12e260abf10d3ac11cfd
Jun 14 23:43:40 <khem> denix: whats the issue you are seeing with SDK ?
Jun 14 23:44:25 <denix> khem: leaking of system libs into sdk binaries
Jun 14 23:44:58 <khem> is it for nativesdk packages or target packages ?
Jun 14 23:45:07 <denix> khem: nativesdk
Jun 14 23:46:06 <khem> it should be using nativesdk-glibc loader
Jun 14 23:46:14 <khem> is that happening or no ?
Jun 14 23:47:28 <denix> khem: well, it uses the loader from sdk, as far as I can tell - PT_INTERP is set properly
Jun 14 23:48:06 <khem> ok
Jun 14 23:49:00 <khem> do we have the equivalent library in nativesdk sysroot ?
Jun 14 23:49:13 <khem> the one its loading from host
Jun 14 23:49:16 <denix> khem: I remember in the past RPATH was always set to $ORIGIN/../lib so sdk libraries will be used instead of hosts one. now it's not the case
Jun 14 23:51:07 <denix> khem: yes. I got a report that nativesdk-cmake complains about some missing openssl symbols, which point to the host lib. cmake links with libcurl and that one links with libssl and all those deps are available in sdk.
Jun 14 23:51:09 <khem> we patch ld.so for nativesdk to reverse the search order
Jun 14 23:52:10 <seebs> RP: https://gist.github.com/seebs/8cf9bc902e8b9467512330bb17bb2ad6
Jun 14 23:52:23 <seebs> i probably broke this somehow, but i'd be super interested in the results this produces.
Jun 14 23:52:47 <seebs> To use: (1) apply patch, (2) produce hang, (3) send SIGUSR2 to pseudo daemon, (4) wait a second or so for a select to timeout or an operation to happen, (5) check pseudo.log
Jun 14 23:52:54 <seebs> it should have ~1k lines of output listing clients.
Jun 14 23:53:02 <seebs> listing clients [1 through 1]:
Jun 14 23:53:02 <seebs> client    1: fd    8, pid 17322, state -, program sh
Jun 14 23:53:02 <seebs> done.
Jun 14 23:53:10 <khem> denix: do you have the failing sysroot around >
Jun 14 23:53:14 <seebs> "state" should be R for any client which was reported readable by select()
Jun 14 23:53:16 <denix> khem: ok, that is becoming more clear why RPATH got removed - what's the commit that reversed the search order?
Jun 14 23:54:09 <denix> khem: not at the moment, but I can get it
Jun 14 23:55:06 <denix> khem: well, actually it was failing for someone else...
Jun 14 23:57:39 <denix> khem: cmake: /lib/x86_64-linux-gnu/libssl.so.1.0.0: version `OPENSSL_1.0.2d' not found (required by /tmp/sdk-devkit/sysroots/x86_64-arago-linux/usr/lib/libcurl.so.4)
Jun 14 23:59:20 <denix> khem: that was the error someone reported. they claim to have sourced the environment setup script and the sdk does contain the correct libssl.so library
Jun 15 00:02:04 <denix> khem: the thing is that I cannot reproduce the issue locally, but I noticed missing RPATH and got suspicious. sounds like that might not be the problem... so I'd have to dig deeper. let me know if you have anay suggestions
Jun 15 00:11:46 <khem> denix: actually we still prefer RPATH over nativesdk only cache order is sent to last
Jun 15 00:12:50 <khem> denix: is it pyro release ?
Jun 15 00:14:21 <denix> khem: morty
Jun 15 00:18:48 <khem> I think a reproducer will help to understand the case
Jun 15 00:20:31 <khem> it seems to be jumping the search order after libcurl.so.4 I Wonder if it has encoded wrong shlib version
Jun 15 00:21:07 <khem> which might force it to ignore the libssl version which is in sysroot
Jun 15 00:21:59 <khem> in anycase running with LD_DEBUG=versions ./yourbinary
Jun 15 00:22:06 <khem> will reveal all version deps
**** ENDING LOGGING AT Thu Jun 15 03:00:03 2017