**** BEGIN LOGGING AT Wed Jan 25 02:59:56 2012 Jan 25 08:26:11 kergoth`, RP__: strace from bitbake hanging today: http://paste.pocoo.org/show/540394/ Jan 25 13:41:53 JaMa: Any idea of the timeout frequency? Jan 25 13:42:04 very fast Jan 25 13:42:06 walters: Its in meta-oe Jan 25 13:42:16 like every second or less Jan 25 13:42:46 JaMa: I'd say that is the threading code, not the sqlite code but I'm going by instinct Jan 25 13:43:32 I've switched to pysqlite just to rule out this option and will report if it happens again Jan 25 14:22:41 RP__: got another hang.. strace looks the same Jan 25 14:23:02 RP__: and timeout is every 1/3s cca Jan 25 14:23:20 RP__: should I try something more before I kill it? Jan 25 14:47:29 JaMa: no, I'm out of ideas on it Jan 25 14:48:30 if it becomes even more often I'll revert that futures changes and try without Jan 25 14:57:12 I wonder if we could hook up something that allows for remote python debugging to a signal handler or something Jan 25 14:57:25 strace is so low level we can't tell what's really happening :\ Jan 25 15:20:29 kergoth`: it would be good to have some way of getting in to debug this kind of thing... Jan 25 15:20:46 kergoth`: I was hoping strace might have revealed any issue with sqlite3 at least Jan 25 15:21:08 * kergoth` nods Jan 25 15:21:39 well, it'd help if we had previous data about the open() of that file descriptor. guess he could try running the whole build under strace until he hits a failed one :) Jan 25 15:21:40 * kergoth` shrugs Jan 25 15:21:55 kergoth`: could we get that info from /proc? Jan 25 15:21:59 course, i'm not sure how well strace supports forking nowadays Jan 25 15:22:10 ah, good point, worth checking out anyway Jan 25 15:22:35 JaMa: if you do see that again, it could help to see if we can identify the file handle from /proc... Jan 25 15:23:13 i've started an app at least twice to parse and summarize strace output, but never finished it Jan 25 16:51:35 * nitink looks like otcchat server is down Jan 25 16:51:47 Good Morning All! Jan 25 16:52:18 hi nitink Jan 25 16:52:29 hi josh Jan 25 16:52:50 s/josh/paul ;) Jan 25 18:05:39 ungh, if I introduce a parser error bitbake wedges Jan 25 18:05:58 Ctrl-C interrupts are detected but the thing doesn't die Jan 25 18:05:59 still? we already fixed the hang due to parse errors in recent master Jan 25 18:06:13 within the past couple days, though. you should update Jan 25 18:06:27 kergoth`: we did? I have everything as of 20mins ago Jan 25 18:07:15 see https://github.com/openembedded/bitbake/commit/d104f29871c04a5a36600a35b2568b49e5b21ca0 Jan 25 18:07:26 that should have taken care of it, and did for all the tests i did, anyway Jan 25 18:07:36 aye, I have that Jan 25 18:08:18 kergoth`: do you have any pointers to debug this? Jan 25 18:08:58 it's the CookerParser in cooker.py. beyond that, no. if it's a hang it's almost certainly a deadlock, which is never fun to deal with Jan 25 18:09:24 \o/ Jan 25 18:12:40 the main issue is the clean or unclean shut down of the parser processes. when using a multiprocessing pool, there's no way to shut down the pool without terminating the processes, which can corrupt the queues that we use for things like sending messages to the UI. at least, that was the theory, hence the switch to futures, which has a method of shutting down without process SIGTERM Jan 25 18:14:34 right, that all makes sense Jan 25 18:15:07 I guess I need to figure out the code and what I'm doing to upset it Jan 25 18:15:14 if its still hanging, i really have no clue where we're going wrong Jan 25 18:16:15 what's really annoying is that python internally uses a feeder thread when you put objects onto a Queue. if you try to shut down that thread before it finishes feeding, without calling cancel_join_thread() on the queue in that process, it will block trying to join the thread until it finishes feeding. if, of course, the other end doesn't drain the queue, it'll never finish feeding. deadlock Jan 25 18:16:25 have to watch for that sort of thing. I thought we had that nailed down, though. Jan 25 18:16:44 wow, that's nasty Jan 25 18:17:16 s/shut down that thread/shut down that process/ Jan 25 18:17:18 * kergoth` nods Jan 25 18:18:39 looks like an exception in one of the parser threads causes said deadlock Jan 25 18:18:54 so first thing is to figure out why that exception is being thrown Jan 25 18:20:03 exceptions are supposed to be caught by the futures code and passed back via the exception() method on the future. for base exception, we turn it into an exception and raise that, as otherwise you can get things like an uncaught SystemExit prematurely shutting down a parser thread :) Jan 25 18:21:41 I'm assuming that because this is all multi-thread there's no easy way to figure out what caused the exception? Jan 25 18:24:37 oh great, and $EDITOR is hanging too Jan 25 18:24:44 * incandescant should go back to bed Jan 25 18:27:54 just go into the parse function which is what gets dispatched to the parser processes. it should be simple enough to emit a message to somewhere there in the exception handler Jan 25 18:29:51 yup, thanks for the tip Jan 25 18:32:29 np, and good luck. assuming I find a spare minute, I'll see if I can find a combination of circumstances which reproduces it (unlikely, but you never know) Jan 25 18:37:51 thanks Jan 25 19:08:50 RP__: kergoth`: got another hung.. this time 2 python processes in loop for timeouts, handle is pipe in both cases Jan 25 19:13:22 RP__: kergoth`: http://paste.pocoo.org/show/540657/ Jan 25 20:57:16 good day. Jan 25 20:57:24 I just have a quick question. Jan 25 20:57:40 What is the minimum storae space requirement for some yocto/poky build? Jan 25 20:59:14 okay, I see that a ubiquity image takes about 6to 18mb Jan 25 22:18:14 gnarly, I can only reproduce my parser explosion on one of two machines running the same OS Jan 25 22:18:22 best try a clean build dir, etc Jan 25 22:18:28 interesting Jan 25 22:21:11 whaddya know, clean build dir and parses just fine Jan 25 22:21:32 I can't decide if that's better or worse... Jan 25 22:34:58 ping hastead Jan 25 22:35:06 hastead: ping Jan 25 22:35:22 halstead: ping, sorry just can't type atm Jan 25 22:37:26 jzhang: use tab completion ;-) I just messaged you by typing jz Jan 25 22:38:29 incandescant: nice tip, just tried with your name, thx:) Jan 25 22:38:49 np Jan 25 22:58:09 hm, how do you guys deal with the delta between bitbake names and package names? Jan 25 22:58:40 e.g. some of them match up, so I can add them to both PACKAGE_INSTALL and RDEPENDS at the same time Jan 25 22:59:11 but for others if i want just e.g. libogg0 in my image, I have to manually add libogg to RDEPENDS Jan 25 22:59:38 don't PACKAGE_INSTALL and RDEPENDS both use package names? Jan 25 23:00:10 imho there's no good way to do it other than figure out the semantics of debian package renaming Jan 25 23:00:22 I trip over it all the time as a Fedora user Jan 25 23:01:13 walters: you should just be adding libogg and the system will figure out it gets named libogg0 Jan 25 23:01:20 even after working at Red Hat for 7 years I still say "build depends", not "build requires" Jan 25 23:01:23 yeah, you shouldn't be adding post-rename stuff, there's no need Jan 25 23:02:54 so here's what i have now http://git.gnome.org/browse/ostree/tree/gnomeos/yocto/classes/gnomeos-contents.bbclass Jan 25 23:02:59 walters: The actual translation of PACKAGE_INSTALL happens in image.bbclass in the python () Jan 25 23:03:45 note i am doing something a little bit custom here because i just want to generate a tarball that i can stick in my ostree repository, rather than an ext3 image Jan 25 23:04:20 walters: I'd borrow the anonymous python to translate PACKAGE_INSTALL from image.bbclass Jan 25 23:04:47 then you will only need untranslated names Jan 25 23:05:15 oh, i see Jan 25 23:05:27 walters: You know you can do IMAGE_FSTYPES = "tar.bz2" to get a tarball? Jan 25 23:06:16 ah, well i needed to override more than that =) Jan 25 23:06:32 walters: I can see that looking at the recipe :) Jan 25 23:07:17 um...hmm...maybe i could rely on dev-pkgs being in IMAGE_FEATURES for the devel one instead of doing this via inheritance Jan 25 23:08:53 walters: I'd have thought you could get something like that to work Jan 25 23:16:19 that bit of code in image.bbclass was what i hadn't looked at closely, and answers the "how is it expected to work" question i had Jan 25 23:18:27 walters: :) Jan 25 23:19:56 walters: at least the complexity is all abstracted so you should be able to reuse it :) Jan 25 23:44:07 what's the difference between native.bbclass and nativesdk.bbclass? Jan 25 23:46:40 hmm...i'm guessing the latter is something for generating a component of a compilation environment for the target, designed to be deployed from the host? Jan 25 23:50:26 walters: The idea of nativesdk is that you can say build a compiler that would run on a 64 bit system on a 32 bit build machine Jan 25 23:51:14 walters: So effectively build an sdk that can run on a specific target platform (SDKMACHINE) Jan 25 23:51:18 hm, but the target here is not necessarily the same as MACHINE? Jan 25 23:51:27 oh i see, there's also SDKMACHINE Jan 25 23:51:40 walters: well, you'd be building an SDK that would run on SDKMACHINE and target MACHINE **** BEGIN LOGGING AT Wed Jan 25 23:56:08 2012 **** ENDING LOGGING AT Thu Jan 26 02:59:57 2012