**** BEGIN LOGGING AT Mon Oct 02 02:59:57 2006 Oct 02 04:06:26 03lenehan 07org.oe.dev * ra0a75207... 10/ (1 packages/grep/grep-native_2.5.1.bb): grep 2.5.1: Add a -native version of grep. Oct 02 04:26:53 * v8jlene is away Oct 02 05:52:13 good morning all Oct 02 06:35:51 * v8jlene is back Oct 02 06:41:12 03lenehan 07org.oe.dev * r1e38cb0e... 10/ (1 packages/perl/perl_5.8.7.bb): Oct 02 06:41:12 perl 5.8.7: Use grep -I to ignore binary files instead of looking at Oct 02 06:41:12 the output message from grep. The output message could change Oct 02 06:41:12 depending on the LOCALE which would break the current test. Make it Oct 02 06:41:12 DEPEND on grep-native to ensure we get the gnu version of grep which Oct 02 06:41:12 is needed for -I to work. As per bug #1439. Oct 02 07:09:33 03koen 07org.oe.packaged-staging * rbcb47891... 10/ (5 files in 3 dirs): stage-manager: add in to contrib/scripts Oct 02 07:11:02 morning Oct 02 07:13:43 hey mickeyl Oct 02 07:14:11 * koen wants it to be thursday Oct 02 07:15:48 me too Oct 02 07:16:02 i have a shitload of things to do before thursday.. Oct 02 07:16:08 * mickeyl drowning in work Oct 02 07:16:42 * koen throws mickeyl a life-jacket Oct 02 07:17:55 and i definitly need to prepare my laptop to be able to work during travelling Oct 02 07:17:57 morning all Oct 02 07:18:00 morning RP Oct 02 07:18:10 hey RP Oct 02 07:20:52 koen: FWIW, the build I left running from packaged staging got to generating the glib locales which failed for the expected reasons. I guess I'd better merge glibc with dev :) Oct 02 07:21:03 RP: :) Oct 02 07:25:53 03koen 07org.oe.dev * rc2533135... 10/ (4 files in 2 dirs): angstrom: start removing overlap between task-base and task-angstrom Oct 02 07:28:26 RP: the next commit is for you :) Oct 02 07:28:31 03koen 07org.oe.packaged-staging * r8975c1a4... 10/ (1 contrib/scripts/ipkg-build mtn:execute true): contrib/scripts/ipkg-build: add a modified ipkg-build that ignores usr/lib/ipkg/lists and ipkg.pyc Oct 02 07:29:17 koen: yay :) Oct 02 07:31:57 koen: Can you remind me why glibc-intermediate is ignored for packaged staging? Oct 02 07:32:17 because it needs file present in staging for 'make install' in do_stage() Oct 02 07:32:28 s/file/stuff/ Oct 02 07:32:40 So we should be able to work around that now? Oct 02 07:32:45 yes :) Oct 02 07:33:22 morning Oct 02 07:33:27 hey hrw|work Oct 02 07:33:46 hi hrw|work Oct 02 07:33:55 * koen starts a p-s build from scratch Oct 02 07:34:02 koen: and the same applied to linux-libc-headers? Oct 02 07:34:13 iirc, yes Oct 02 07:34:15 I also have some thigns to do before oedem Oct 02 07:34:24 1. finish session1 presentation Oct 02 07:34:36 0. buy bag for clothes and stuff Oct 02 07:35:01 morning all Oct 02 07:35:05 hey ade|desk Oct 02 07:35:09 hi ade|desk Oct 02 07:35:25 hi ade|desk Oct 02 07:36:30 hm Oct 02 07:36:33 *mumble* Oct 02 07:36:49 note to self: do not use bitbake threading with p-s Oct 02 07:37:47 morning Oct 02 07:39:23 morning Oct 02 07:39:33 koen: As long as theres only populate-staging running, it should be ok - we need to add something to bitbake to support that though :) Oct 02 07:39:36 hi XorA Oct 02 07:42:41 RP: I missed if you ever answered, but did you get mplayer + rgb on /dev/fb2 to work? Oct 02 07:43:39 XorA: I never tried Oct 02 07:44:04 XorA: rgb would be fb1, YUV would be fb2 Oct 02 07:44:17 RP: ok, it doesnt seem to like any fb other than fb0 here, have to dig about and see why Oct 02 07:44:59 XorA: The other framebuffers are only created when something opens them and are destroyed along with all settings when closed Oct 02 07:45:10 They default to a size of 0x0 Oct 02 07:45:43 RP: I know, but sending -fbmode blah -vm should set the mode from player Oct 02 07:46:26 RP: but the fbdev/fbdev2 code in mplayer looks a bit crazy Oct 02 07:46:51 XorA: I've just not had a chance to look into ot Oct 02 07:48:16 03rpurdie 07org.oe.packaged-staging * r397db3df... 10/ (8 files in 3 dirs): glibc: Sync with .dev Oct 02 07:52:16 ah, hmmm Oct 02 07:52:23 * koen install gcc-3.3 and libsdl12-dev Oct 02 08:03:05 03ade 07org.oe.dev * rac174cd1... 10/ (1 packages/rdesktop/rdesktop_cvs.bb): rdesktop: cvs change default pref to -1 Oct 02 08:24:45 http://qtdeveloper.net/archives/2006/10/01/opie-fud/ Oct 02 08:25:38 *sigh* Oct 02 08:26:33 I'm pretty good at ignoring things nowadays... Oct 02 08:27:12 heh Oct 02 08:27:44 I was thinking about removing opie-notes from opie and oe Oct 02 08:27:58 qt4 version is easier to write Oct 02 08:28:12 qt4 is pretty awesome Oct 02 08:28:15 mickeyl: i particularly like how ljp advocates opie not breaking compatibility, yet thinks that can somehow magically be done without regularly open source releases on their end Oct 02 08:28:21 * kergoth coughs Oct 02 08:28:31 QSql is nice Oct 02 08:28:46 hey kergoth Oct 02 08:28:48 ~mornings Oct 02 08:28:49 hey Oct 02 08:28:53 kergoth: indeed :/ Oct 02 08:28:57 Mornings MUST be destroyed! (see also http://www.destroymornings.com/) Oct 02 08:29:27 * koen offers kergoth some TT/e kool-aid Oct 02 08:29:34 hehe Oct 02 08:30:25 i'm still surprised at just how fast ljp's priorities and tt's priorities became one and the same Oct 02 08:30:34 morning kergoth Oct 02 08:30:43 hey RP Oct 02 08:30:44 kergoth: no kidding, this is the thing that annoys me the most. Oct 02 08:30:50 hey Oct 02 08:31:08 hey zecke Oct 02 08:31:09 kergoth: they almost sucked all community thinking out of him and transformed it into corporate thinking. although he claims otherwise, of course :/ Oct 02 08:31:11 morning zecke Oct 02 08:31:21 zecke: didn't you look into openssl a while ago? Oct 02 08:31:41 koen: 'look' == trying to use a non-vulnerable version? Oct 02 08:31:48 zecke: yes Oct 02 08:31:59 koen: yes, but you did the same then :) Oct 02 08:32:05 and I failed :) Oct 02 08:32:36 koen: ah well, I will go to the gym soon and then might look into cleaning it up and pushing some bits Oct 02 08:33:21 RP: http://rafb.net/paste/results/YqG5Dc77.html Oct 02 08:34:06 RP: any thoughts on marking unrelocatable stuff like quilt 'dirty' in OE? Oct 02 08:34:22 s/quilt/quilt-native/ Oct 02 08:35:15 koen: make quilt relocatable Oct 02 08:37:18 mickeyl: hmm, at least he admits Qtopia is not free Oct 02 08:40:39 zecke: when I have a chance, I'll dig out my old mail about why Opie died and post it as comment Oct 02 08:42:23 mickeyl: We lost marketing and failed to differentiate from Qtopia (simply phrased) Oct 02 08:42:32 *nod* Oct 02 08:42:44 mickeyl: and they are right. Why should I use Opie1.1 when I get Qtopia4 Oct 02 08:44:34 * koen should look into adding cleanlooks to OE Oct 02 08:45:06 qt3 looks like gtk 1.0 on my devices Oct 02 08:45:17 I know it can do better :) Oct 02 08:45:42 http://blogs.sun.com/andrei/entry/opteron_easter_egg ;D Oct 02 08:46:10 koen: well, it tries to detect the WM and MB is not known so it defaults to motif look :) Oct 02 08:47:09 zecke: I use e17 Oct 02 08:48:40 koen: Looks good. We should probably add something to the metadata which says something can't be relocated. That will probably apply to all native packages to some extent though (certainly moving them from one system to another is likely to fail due to system lib differences) Oct 02 08:49:49 RP: I'd like to have something like www.angstrom-distribution.org/releases/2007.1/sdk/feed/ubuntu-dapper/ Oct 02 08:50:24 RP: a 'developer' only has to build stuff that isn;t relocatable like quilt to start using OE Oct 02 08:50:28 koen: I'd like to see this too... Oct 02 08:51:05 but with gcc-cross + binutils-cross + glibc the annoyance is a lot less Oct 02 08:51:30 koen: We're getting there - one step at a time Oct 02 08:51:55 zecke: ok, found the mail and added a comment :) Oct 02 08:56:10 koen: glibc is the big benefit with packages staging, and with prepackaged libs, app development should be easier to as you'll not have to build the entire dependency chain Oct 02 08:56:40 RP: is the depchains stuff already in .dev? Oct 02 08:57:09 koen: Some of it certainly is, I'm not sure how much Oct 02 08:58:07 the stuff that adds RRECOMMENDS for -dev packages? Oct 02 09:00:04 ~seen Crofton Oct 02 09:00:10 mickeyl: nice Oct 02 09:00:16 crofton is currently on #casualti (1d 10h 27m 47s) #ol (1d 10h 27m 47s) #oe (1d 10h 27m 47s) #elinux (1d 10h 27m 47s). Has said a total of 13 messages. Is idling for 12h 24m 3s, last said: 'good movie'. Oct 02 09:00:40 hrw|work: I'm too busy to make it a larger response now, but that should get the major point across Oct 02 09:04:29 mickeyl: yep. I do not plan to work too much on opie. have to test/integrate Marcos work from SoC and then will probably more or less dump the ship Oct 02 09:06:52 i have mostly good feelings about that. Opie was a fantastic learning experience for me Oct 02 09:11:08 ack Oct 02 09:16:42 RP: my build is now at compiling glibc Oct 02 09:17:09 koen: excellent. Should I commit those changes? Oct 02 09:17:52 slightly ahead of you :) Oct 02 09:17:56 03rpurdie 07org.oe.packaged-staging * r986d9bc3... 10/ (1 classes/packaged-staging.bbclass): classes/packaged-staging.bbclass: use stage-manager helper script to manage moving around staging/ and cross/ Oct 02 09:17:57 * koen kicks cia Oct 02 09:19:18 poor cia Oct 02 09:19:32 koen: Complete with my own ID - interesting :) Oct 02 09:20:13 RP: 'mtn commit --author rpurdie@rpsys.net' Oct 02 09:20:53 koen: I guessed. I presume that still shows up as signed by you though? :) Oct 02 09:20:58 yes Oct 02 09:22:09 http://rafb.net/paste/results/fJiygc40.html Oct 02 09:24:38 NOTE: preparing tree for binary locale generation Oct 02 09:25:49 * koen wonders about how to parallellize locale generation Oct 02 09:32:03 for locale in localelist;do generate_locale &;done Oct 02 09:32:10 ;D Oct 02 09:34:45 hi all Oct 02 09:34:52 * koen gets some cpuj time on the uni altix cluster Oct 02 09:35:20 :-) Oct 02 09:36:45 hi florian_kc Oct 02 09:42:50 florian_kc: any news on a gpe-calendar release? Oct 02 09:43:28 koen: no :-( did neal mention if he finished his paper? Oct 02 09:43:40 I haven't spoken to neal in weeks Oct 02 09:44:48 :-( Oct 02 10:02:57 gah, gnutls has moved to autoconf 2.60 Oct 02 10:04:33 XorA: add 2.60 to OE ? Oct 02 10:04:43 * koen prepares for the cries of agony Oct 02 10:06:25 koen: I started, but wolfson not be pleased if I spend all day porting our patches Oct 02 10:10:24 right :) Oct 02 10:10:43 ~lart 2.6.9-nbp Oct 02 10:10:44 * ibot stabs 2.6.9-nbp Oct 02 10:10:51 ibot: botsnack Oct 02 10:10:52 koen: aw, gee Oct 02 10:15:22 koen: I will look at it in my own time Oct 02 10:16:32 03rpurdie 07org.oe.packaged-staging * r7882ba5a... 10/ (1 classes/packaged-staging.bbclass): packaged-staging.bbclass: Remove unneeded special cases Oct 02 10:20:47 03rpurdie 07org.oe.packaged-staging * rae09a62d... 10/ (1 classes/packaged-staging.bbclass): packaged-staging.bbclass: Remove merge artifact, reindent functions for clarity Oct 02 10:24:37 03koen 07org.oe.packaged-staging * r3e228c55... 10/ (1 classes/packaged-staging.bbclass): packaged-staging.bbclass: update BUGS and TODO Oct 02 10:27:09 it's still generating locales.... Oct 02 10:27:24 NOTE: package glibc-2.4: completed Oct 02 10:27:35 * koen puts some more '+' in the 2800+ cpus Oct 02 10:28:58 * XorA hugs his dual 64bit Xeon Oct 02 10:29:18 XorA: didn't you have a quad? Oct 02 10:29:39 koen: lrg over estimated :-) Oct 02 10:29:44 03rpurdie 07org.oe.packaged-staging * r8e6d405d... 10/ (1 classes/packaged-staging.bbclass): packaged-staging.bbclass: Remove some debug, fix LIST_CMD to run on STAGING_DIR Oct 02 10:32:33 ~dict clobber Oct 02 10:32:35 Dictionary 'clobber' (1 of 7): vt. To overwrite, usually unintentionally: "I walked off the end of the array and clobbered the stack." Compare {mung}, {scribble}, {trash}, and {smash the stack}.. Oct 02 10:34:50 03koen 07org.oe.packaged-staging * r0c381b1c... 10/ (1 classes/packaged-staging.bbclass): packaged-staging.bbclass: introduce PSTAGE_MODE Oct 02 10:38:39 NOTE: package glibc-2.4-r10: task do_populate_staging: started Oct 02 10:40:33 RP: with depchains working PSTAGE_MODE = 'repopulate' would just install the generated packages and let the package manager pull in the rest, right? Oct 02 10:41:24 koen: If the staging packages can trigger the dep chain code Oct 02 10:41:37 koen: NOTE: package nano-1.3.9: completed Oct 02 10:42:31 someone want to update nano to 1.9.99pre2? Oct 02 10:43:29 RP: I knew I was forgetting something :( Oct 02 10:43:52 koen: We can reuse the code though Oct 02 10:44:23 gosh I'm still pissed :} Oct 02 10:44:41 pissed as in drunk or as in angry? Oct 02 10:45:01 sadly drunk Oct 02 10:50:15 ~seen Crofton Oct 02 10:50:35 crofton is currently on #casualti (1d 12h 18m 6s) #ol (1d 12h 18m 6s) #oe (1d 12h 18m 6s) #elinux (1d 12h 18m 6s). Has said a total of 13 messages. Is idling for 14h 14m 22s, last said: 'good movie'. Oct 02 11:00:45 RP: are the pkgmaps getting generated with your build? Oct 02 11:01:49 koen: Just for glibc and gcc Oct 02 11:02:10 with bitbake trunk? Oct 02 11:02:17 koen: yes Oct 02 11:02:22 hmmm Oct 02 11:04:01 koen: I'm not sure the spawn files are though Oct 02 11:04:29 right Oct 02 11:04:39 that's what I feared Oct 02 11:05:23 and that's what I'm seeing here Oct 02 11:05:49 The other pkgmaps data is getting generated though Oct 02 11:06:17 I suspect the PACKAGE_FUNC stuff isn't working in p-s.bbclass Oct 02 11:06:51 perhaps it needs to inherit package,bbclass? Oct 02 11:07:33 I guess so Oct 02 11:09:21 koen: check package.bbclass for PACKAGE_FUNC = vs. += vs. ?= Oct 02 11:10:06 PACKAGEFUNCS = " Oct 02 11:10:39 It should inherit package.bbclass, then it makes sure it can append to that variable Oct 02 11:10:53 RP: but instead of appending you are overwriting :) Oct 02 11:10:56 'inherit package' doesn't fix it Oct 02 11:10:57 ~lart kergoth Oct 02 11:10:57 * ibot blames kergoth for all the evil in the world Oct 02 11:11:30 koen: hmm :-/ Oct 02 11:11:53 * koen has a fix Oct 02 11:11:57 koen: make it ?= again Oct 02 11:12:04 http://rafb.net/paste/results/C9Lmvh50.html Oct 02 11:12:30 then you have these methods n time invoked Oct 02 11:12:45 koen: e.g. if you have insanity, package_ipk and package_tar there Oct 02 11:12:57 you call these methods a lot of times Oct 02 11:13:16 and you have insanity's package func called before the package installs :) Oct 02 11:14:33 I was thinking the class handling was clever :). We need ?= in there as zecke says Oct 02 11:14:53 RP: and it was there before kergoth messed with it :) Oct 02 11:14:57 http://rafb.net/paste/results/pbo9ae55.html Oct 02 11:15:08 ~lart samba shares, monotone, sqlite for not cooperating Oct 02 11:15:09 * ibot executes killall -KILL samba shares, monotone, sqlite for not cooperating Oct 02 11:15:15 RP: but koen had one problem in the packaged staging branch Oct 02 11:15:38 koen: what does your package.bbclass look like in the branch? Oct 02 11:16:14 zecke: I merged in the changes from .dev to the best of my abilities and have evidently broken it Oct 02 11:16:19 zecke: http://www.openembedded.org/filebrowser/org.openembedded.packaged-staging/classes Oct 02 11:16:32 03koen 07org.oe.packaged-staging * rd669a2fc... 10/ (1 classes/package.bbclass classes/packaged-staging.bbclass): Oct 02 11:16:32 classes/packaged-staging.bbclass: inherit package Oct 02 11:16:32 classes/package.bbclass: use ?= for PACKAGEFUNCS Oct 02 11:16:51 zecke: What was the problem? Oct 02 11:16:52 koen: you should have plucked that ;) Oct 02 11:16:59 RP: I don't remember :) Oct 02 11:18:16 at least this highlights exactly what we need to change to get things working in .dev :) Oct 02 11:18:36 it was ?= in .dev a while ago Oct 02 11:18:50 03koen 07org.oe.dev * r421c01eb... 10/ (1 classes/package.bbclass): Oct 02 11:18:50 classes/package.bbclass: use ?= for PACKAGEFUNCS *again* Oct 02 11:18:50 * people should really watch out with introducing regressions Oct 02 11:18:50 but it seems kergoth merged in poky's classes into OE Oct 02 11:19:15 koen: yes :-/ Oct 02 11:19:16 koen: right, my fault. I bitched him and he said he need to wake up first. I didn't bicth him again Oct 02 11:19:49 * koen sends invoice to mallum Oct 02 11:21:52 "Your best defense is to not use SQLite for files on a network filesystem." :( Oct 02 11:22:06 hey likewise Oct 02 11:22:13 hi likewise Oct 02 11:22:21 hi fooks Oct 02 11:22:55 * koen bitbake glibc -c rebuild Oct 02 11:23:18 RP: so p-s effectively didn't run today Oct 02 11:23:52 koen: It did at least package everything Oct 02 11:23:58 :) Oct 02 11:33:13 RP: I feel forced to rewrite tinderbox3 :( Oct 02 11:36:50 zecke: :-( Oct 02 11:37:18 zecke: I suspected that might be unavoidable :-( Oct 02 11:38:29 I suspect packaged staging and the packaged staging fetcher is going to have some interesting implications for bitbake trunk's taskdata/runqueue as well :-/ Oct 02 11:38:38 gah Oct 02 11:38:43 * NAiL hates diet-x11 Oct 02 11:39:07 RP: has o-hand.com done anything in regard to creating a tinderbox? It would be nice to see your requirements Oct 02 11:39:27 http://bugs.openembedded.org/show_bug.cgi?id=1447 <-- diet-x11 fails to build on hosts without X Oct 02 11:39:46 zecke: We've discussed the fact we need one and disucssed the requirements (i.e. why we need it and what it would be needed to do) Oct 02 11:40:22 zecke: mallum is still of the opinion that there is software out there we can use though. Personally, I think we'll probably have to create something Oct 02 11:40:33 or seriously modify something Oct 02 11:41:18 another point for OEDEM Oct 02 11:41:33 Yes, this one would be a good one to talk about in person Oct 02 11:42:33 a real neat feature would be to have some form of 'hidden' report Oct 02 11:42:35 s Oct 02 11:42:40 NAiL: maybe we need kind of diet-x11/do_stage_headers() before do_configure? Oct 02 11:42:57 that way OE-core can monito breakages in super-secret IHV projects Oct 02 11:43:08 zecke: you might want to poke CELF about that as well Oct 02 11:43:45 speaking about celf.. Oct 02 11:44:00 hrw|work: the missing headers are already in staging Oct 02 11:44:01 koen: they have firewall blocked so no monotone Oct 02 11:44:12 hrw|work: that's a nonsense excuse Oct 02 11:44:12 diet-x11 just looks in the wrong place Oct 02 11:45:17 hrw|work: since mtn can work over file, ssh and http as well Oct 02 11:45:46 koen: I will test that ways and will suggest better way for them Oct 02 11:46:17 hrw|work: tell them to open 1 (one) port in the firewall Oct 02 11:46:22 it's not that hard Oct 02 11:46:47 koen: It depends on the company... Oct 02 11:46:53 iptable -A INPUT -p tcp --dport XXXX -j ACCEPT :-) Oct 02 11:47:21 if they are too incompetent to do that we could even give celf an account on oe.org so they can portforward over ssh Oct 02 11:47:56 koen: Its not incompetance thats the problem, its IT people' Oct 02 11:47:59 s policies Oct 02 11:48:03 or install stunnel/tor/whatever Oct 02 11:49:00 RP: right, so they want an OE test harness, but don't want it at the same time Oct 02 11:49:53 as you might have noticed, I'm too pragmatic to respect such policies Oct 02 11:50:49 just like those bogus disclaimers added to emails Oct 02 11:50:57 koen: Some of the people will want that, others will want network security. I'm not saying its right, but having been there, I know what trying to get policies changed can be like Oct 02 11:52:08 I've worked for a company where employees discovered 'poledit.exe' Oct 02 11:52:46 my desk was between the helpdesk and the techs Oct 02 11:52:47 They can open the port from specific source IP addresses. Oct 02 11:52:50 from=for Oct 02 11:53:12 you should have heard the curses flying by Oct 02 11:53:26 koen: Thats been around for years. It brings back good memories of the school network :) Oct 02 11:54:25 what is poledit? I'm not familiar with MS Windows world Oct 02 11:54:41 poledit allows you to lock down windows Oct 02 11:55:00 or open it up again if not locked down properly :) Oct 02 11:55:01 hide drives, make the registry RO, etc Oct 02 11:55:46 ah Oct 02 11:55:47 you can't secure around removing the tokenring plug and having a disk with poledit Oct 02 11:57:18 first thing to set in global policy is do not allow local policy :-) Oct 02 11:57:30 koen: These machines had passworded BIOS and would only boot from the network via a BOOTROM on the network card, just to make life interesting Oct 02 11:57:52 tmbinc_: you rock!!! Oct 02 11:58:01 RP: our school had something like that as well Oct 02 11:58:04 RP: just what a zaurus with bootp is designed to hack Oct 02 11:58:23 RP: but bios passwords are easily cracked via userspace :) Oct 02 11:58:26 03nail 07org.oe.dev * r86d76364... 10/ (3 files in 3 dirs): busybox: Fix slugos initscript to prevent dhcp server starvation. Closes oe bug #1417 Oct 02 11:58:49 koen: also all Phoenix and Award BIOS have "default" password until a couple of years ago Oct 02 11:59:10 XorA: yep, they had when I was in highschool :) Oct 02 11:59:40 They'd had the sense to change that on these... Oct 02 11:59:51 they where pretty surprised when systems suddenly booted into linux Oct 02 12:00:16 They also had the A-Level CompSci students helping with security... Oct 02 12:00:19 bypassing their overpriced 'security cards' they installed Oct 02 12:01:12 when I had to do CS courses in my second year I used e16 as a WM over remote X :) Oct 02 12:01:38 harder beasts are machines which lack any storage and have usb ports glued to not allow any external storage connected Oct 02 12:01:41 much better as the default RH6 wm Oct 02 12:03:06 * koen waits for glibc to finish generating locales Oct 02 12:03:12 hrw|work: back then, I didn't have USB disks or a Z to play with the network with. Just lots of floppy disks... Oct 02 12:12:04 * zecke pets his dreambox :) Oct 02 12:12:13 zecke: new toy? Oct 02 12:13:16 dreambox is nice toy (if you have DVB) Oct 02 12:14:40 and parse .de Oct 02 12:15:34 ups, nevermind, in mixed dbox2 and dreambox Oct 02 12:26:13 rmk just accepted the cxx00 qvga patch :) Oct 02 12:27:49 great Oct 02 12:29:07 I've compiled glibc 2.4 with nptl support in OE but I've noticed pthread_kill() is missing from its header. I did a search and pthread_kill() is part of the POSIX standard so it should be there. DirectFB uses pthread_kill() so it won't compile.. has anyone else run into this issue? Oct 02 12:29:32 s/header/header files Oct 02 12:35:07 I have this: Oct 02 12:35:09 ork/armv5te-angstrom-linux-gnueabi/glibc-2.4-r10/glibc-2.4/conform/data/signal.h-data:157:function int pthread_kill (pthread_t, int) Oct 02 12:35:10 work/armv5te-angstrom-linux-gnueabi/glibc-2.4-r10/glibc-2.4/nptl/pthreadP.h:433:extern int __pthread_kill (pthread_t threadid, int signo); Oct 02 12:35:10 work/armv5te-angstrom-linux-gnueabi/glibc-2.4-r10/glibc-2.4/nptl/sysdeps/pthread/bits/sigthread.h:36:extern int pthread_kill (pthread_t __threadid, int __signo) __THROW; Oct 02 12:37:33 koen, when I did a grep of staging/include no hits came up for pthread_kill() Oct 02 12:38:15 RP: we could that the 'stage-manager' thing a bit futher by renaming pstage to 'stage-left' :) Oct 02 12:38:59 koen: :) Oct 02 12:39:18 *still generating locales* Oct 02 12:40:28 Crofton: current linux-omap1 git builds without problems Oct 02 12:40:33 morning Oct 02 12:40:39 good :) Oct 02 12:40:45 koen, I grep'd the work directory of glibc as well and got the same thing as you, but why is there nothing in the staging area? Wouldn't it need to be there for an app to link against? Oct 02 12:40:50 I can build bs image fine atm Oct 02 12:40:57 hey chouimat Oct 02 12:41:00 I'll try flashing later today Oct 02 12:41:16 Gerrath: I'm compiling glibc atm, so I don't have anything in staging Oct 02 12:41:23 hi Crofton chouimat Oct 02 12:41:40 Crofton: still no 2.6.18-omap1 ;( Oct 02 12:41:59 Crofton: do you remember how long after 2.6.17 was 2.6.17-omap1 released? Oct 02 12:42:24 no Oct 02 12:42:26 koen, ok can you ping me when it has compiled and let me know what your results are for grep'ing pthread_kill in the staging area? Oct 02 12:42:40 Gerrath: sue Oct 02 12:42:42 sure* Oct 02 12:42:43 koen, thanks. Oct 02 12:43:38 Crofton: 2-4 weeks it looks Oct 02 12:43:40 brb Oct 02 12:59:15 03koen 07org.oe.packaged-staging * r067e01b9... 10/ (1 classes/packaged-staging.bbclass): packaged-staging.bbclass: don't abort when there are no .la files in staging Oct 02 12:59:45 RP: another bug fixed :) Oct 02 12:59:58 koen: nice :) Oct 02 13:00:14 re Oct 02 13:00:39 Crofton: if you will get board booting then can you add fixed-rev git version? Oct 02 13:00:53 Gerrath: indeed no pthread_kill in staging Oct 02 13:01:14 koen: One thing to note - currently stage-manager copies the files out of staging to package and when it installs the package, it overwrites them. It might be cleaner to move the files? Oct 02 13:01:38 koen, I grep'd the glibc binary lib files in the staging area, pthread_kill had a hit so it is in the lib.. I then just added an external reference to pthread_kill in the staging area glibc header. Directfb now compiles with out any issue, so the problem seems to be a missing or incomplete glibc header. Oct 02 13:02:01 RP: yes, that would be cleaner Oct 02 13:02:19 koen: It just needs s/cp/mv/ and testing :) Oct 02 13:03:34 koen, I'll submit a bug. Oct 02 13:08:22 RP: /media/hda4/OE/build/tmp-new/angstrom/cross/lib/gcc/arm-angstrom-linux-gnueabi/4.1.1/../../../../arm-angstrom-linux-gnueabi/bin/ld: skipping incompatible /lib/libc.so.6 when searching for /lib/libc.so.6 Oct 02 13:08:48 RP: it looks like we need to sed libc.so :( Oct 02 13:11:54 hi CoreDump|home Oct 02 13:14:33 koen, glibc bug report http://bugs.openembedded.org/show_bug.cgi?id=1448 Oct 02 13:19:03 did anyone upgrade to edgy yet? Oct 02 13:19:36 zecke: nope, seeing that they still manage to break dapper on a weekly base :( Oct 02 13:23:48 hi, folks. ;) Oct 02 13:24:16 hey leoncamel Oct 02 13:25:17 koen, :). I am enjoying my holidays here, in China. So, how is OE going on ? Oct 02 13:26:24 hi leoncamel Oct 02 13:26:33 we are close to having packaged-staging working Oct 02 13:26:45 koen, you commit some patch on class/*.bbclass ? Oct 02 13:27:01 hrw|work, hey. :) Oct 02 13:27:34 koen, so, what is the problem ? what can I do for you ? Oct 02 13:29:51 koen, I mean, I can do some basic test case for you .. :) Oct 02 13:35:43 03koen 07org.oe.packaged-staging * r0ca9b992... 10/ (1 classes/packaged-staging.bbclass): Oct 02 13:35:43 packaged-staging.bbclass: fixes Oct 02 13:35:43 * blacklist dbg, gconf and charmap packages Oct 02 13:35:43 * fix up libc.so linker script with sed Oct 02 13:35:43 leoncamel: not at the moment Oct 02 13:36:07 koen, :). you are always welcome.. Oct 02 13:36:22 RP: libogg compiles now :) Oct 02 13:36:35 RP: and s/cp/mv/ seems to be working as well Oct 02 13:38:25 koen: ok, that's good. It was just a cp at first as I didn't want to destroy things too much whilst making the script work :) Oct 02 13:38:54 koen: Why did we never see the libc problem before? Oct 02 13:39:26 RP: because do_stage did a *extra* 'make install' Oct 02 13:40:17 koen: Should it do two make installs? Oct 02 13:40:32 IMO not Oct 02 13:40:45 but the whole glibc stuff in OE is pretty arcane and obfuscated Oct 02 13:41:15 glibc is an area I stay away from... Oct 02 13:41:31 http://www.extremeironing.com/ <--- sound like a fun thing to do :) Oct 02 13:46:02 who is maintaining bitbake now ? Oct 02 13:48:24 leoncamel: Some combination of zecke and myself I guess Oct 02 13:55:59 03koen 07org.oe.packaged-staging * r2031ff65... 10/ (1 contrib/scripts/stage-manager): contrib/scripts/stage-manager: use 'mv' instead of 'cp' Oct 02 13:56:45 * koen tries a 'bitbake world' with packaged-staging Oct 02 13:58:01 could you write parts of bb in c/c++ and use swig to call them from python? Oct 02 13:58:06 koen, how many disk it would use ? Oct 02 13:58:27 leoncamel: not much, there are very few recipes present Oct 02 13:58:45 hrw|work: what's the status of rm_work? Oct 02 13:59:58 mickeyl: ping Oct 02 14:00:29 RP: apart from the threading issues packaged-staging.bbclass is looking pretty nice for .dev Oct 02 14:01:37 koen: basically it works but need one change in bitbake addtask to be really working ok Oct 02 14:01:58 koen: now you can get hit by do_deploy() after do_rm_work() Oct 02 14:02:32 hrw|work: I was thinking of making angstrom inherit it Oct 02 14:02:55 to avoid the 'OMG DISKSPACE LOL' FUD Oct 02 14:03:29 koen: enable it first and do builds - if any fail then you will know why it is not safe often Oct 02 14:03:45 to gain some diskspace Oct 02 14:03:51 hrw|work: right Oct 02 14:03:57 i write a little bash script for my team , Oct 02 14:04:14 that create an OE env for one or more project Oct 02 14:04:21 Genesis: rm_work is simple shell script which remove all WORKDIR after building Oct 02 14:04:32 mine is different Oct 02 14:04:42 that create a personnal tmp dir Oct 02 14:04:56 koen: some recipes would need patching to work with rm_work Oct 02 14:04:56 i paste it you'll see if you're interessting Oct 02 14:05:01 Genesis: pastebin it Oct 02 14:05:06 hi hvontres Oct 02 14:05:10 RP: can we make packaged-stage output something like "installing foo to ${STAGING_BASEDIR}", like package.bbclass does? Oct 02 14:05:13 i'll commit it when it 'll be finish Oct 02 14:05:58 hi hrw|work Oct 02 14:06:11 koen: I don't see why not - just add an oenote ? Oct 02 14:06:32 RP: doesn't oenote only work in python methods? Oct 02 14:06:35 http://code.bulix.org/45eiv2-18965 Oct 02 14:06:43 the idea is to have a official tree Oct 02 14:07:01 and work with a team Oct 02 14:07:24 i would like to rewrite it in a bbclass , but there is not a lot of documentation Oct 02 14:07:33 koen: I think it has shell script hooks Oct 02 14:07:50 koen: Or perhaps I'm confused Oct 02 14:07:53 RP: I remember something the shell version not working Oct 02 14:08:03 i.e. only outputting to the logs, not the sceen Oct 02 14:08:05 screen* Oct 02 14:08:23 Ah, I can understand that with the pipe redirection bitbake uses Oct 02 14:09:17 Genesis: yep.. it is much larger thing Oct 02 14:10:10 it's useful for me :) Oct 02 14:10:21 and for my teamate i hope Oct 02 14:12:08 BTW: anyone has used django? Oct 02 14:20:11 so far only pkgconfig (non-native) b0rked Oct 02 14:22:24 koen: How did that break? Oct 02 14:23:48 | testgthread.o: In function `test_private':testgthread.c:(.text+0x3dc): undefined reference to `pthread_join'| ./.libs/libgthread.al(gthread.lo): In function `g_private_set_posix_impl':gthread.c:(.text+0x1c4): undefined reference to `pthread_setspecific' Oct 02 14:24:36 ~lart konqueror for hanging Oct 02 14:24:36 * ibot raises middle finger to konqueror for hanging Oct 02 14:24:39 OE's copy of pkgconfig is ancient, mind you Oct 02 14:24:51 koen: http://www.djangoproject.com/documentation/syndication/ Oct 02 14:25:00 I think I will use django for OEs tinderbox Oct 02 14:25:28 cute Oct 02 14:30:17 my website now looks nice on phones or other limited devices Oct 02 14:31:04 zecke: it even recommends using postgres :) Oct 02 14:31:25 mickeyl: http://imthi.com/wp-pda/ Oct 02 14:36:46 koen: I think it looks really good Oct 02 14:37:54 django was hot a year ago Oct 02 14:38:00 it seems to have matured since then Oct 02 14:39:32 RP: what's the next step for p-s? Oct 02 14:40:12 RP: integrating it with depchains? making it packageformat agnostic? Oct 02 14:40:39 koen: build world? Oct 02 14:41:05 hrw|work: only pkgconfig and qemu failed in the packaged-staging branch with 'bitbake world -k' Oct 02 14:41:38 koen: good question - adding in the depchains support would certain be good to get the other mode of staging you want working Oct 02 14:42:06 RP: the -dev packages already should have the appropriate RRECOMMENDS Oct 02 14:43:03 zecke: pong Oct 02 14:43:08 koen: but the staging packages are separate from the normal generated packages? Oct 02 14:43:09 hrw|work: that's pretty cool Oct 02 14:43:21 mickeyl: yep Oct 02 14:45:07 RP: ah, right you meant the native and cross ones Oct 02 14:45:26 mickeyl: have you looked into django yet? Oct 02 14:46:01 zecke: no, that's new to me Oct 02 14:46:50 koen: I've just missed part of the way this all works :) Oct 02 14:47:19 koen: If we can ensure the -dev package brings in all the depends, we're probably ok Oct 02 14:47:34 RP: that part should already be in OE Oct 02 14:47:45 RP: that's the commit with all the regressions..... Oct 02 14:47:45 koen: I suspect we need an ALLOW_EMPTY_${PN}-dev = "1" Oct 02 14:52:28 mtn 0.30 is reallly fast Oct 02 14:57:50 furlongm: hey, you had some questions? Oct 02 14:58:24 zecke: yep, but still cleaning up after akademy, so it'll be a few days before i get back to normal stuff Oct 02 14:59:01 furlongm: at least I remember the friday :) Oct 02 14:59:29 be back later Oct 02 15:04:19 17:22 hrw@work:hrw$ apt-cache search wake on lan Oct 02 15:04:20 emacs-goodies-el - Miscellaneous add-ons for Emacs Oct 02 15:04:28 emacs roxx Oct 02 15:04:32 njs, faking configure to allow monotone compilation against boost 1.32.0 didn't work Oct 02 15:05:49 Crofton: that's why venge.net/monotone was a lot of static binaries :) Oct 02 15:06:59 hmmm Oct 02 15:07:10 I may have to pick one up :) Oct 02 15:16:59 koen: Even though i moved over to Angstrom from Familiar diet-x11 and libx11 failes. Any idea why? http://pastebin.ca/188837 Oct 02 15:17:24 goxboxlive: did you clean tmp/ ? Oct 02 15:17:32 no Oct 02 15:17:43 I'll try that. thx Oct 02 15:18:07 NAiL: did you find a fix for the diet-x11 header problem already? Oct 02 15:22:38 hey pH5 Oct 02 15:22:54 pH5: what are your thought on making llh 2.6.18 the default? Oct 02 15:22:58 hey koen Oct 02 15:23:22 I had some problems building util-linux which I couldn't resolve yesterday. Oct 02 15:23:28 same here Oct 02 15:23:30 something fdisk / llseek related Oct 02 15:25:38 I'd like to at least build one full angstrom bootstrap image before making 2.6.18 default for everyone Oct 02 15:27:35 atm it looks like installing the full asm/unistd.h and removing the __KERNEL__ #ifdef would bring us to the state we had with '2.6.15.99' Oct 02 15:29:30 pH5: that'd be neat Oct 02 15:29:41 pH5: could you also check in the task-base stuff? Oct 02 15:30:46 cu Oct 02 15:30:51 hrw|gone: cu Oct 02 15:33:35 hey.. Oct 02 15:33:40 anyone know what's up with xcalibrateext_git.bb? Oct 02 15:33:50 sources can't be found Oct 02 15:34:01 how come it's now _git rather tan _csv? Oct 02 15:44:18 koen: task-base, that is the ipaq-pxa270 kernel module stuff? Oct 02 15:51:16 http://rafb.net/paste/results/iUWsfO35.html ideas? Oct 02 15:57:02 zecke: If its tbox3 compatible, can we deal with information about the tasks concurrently? (i.e. when multithreading) Oct 02 15:57:26 RP: who needs multithreading anyway? Oct 02 15:57:26 zecke: Also, will it be able to help isolate failures by MACHINE, DISTRO, PN, task ? Oct 02 15:57:36 zecke: me ;-) Oct 02 15:57:52 pH5: yes, the module stuff Oct 02 15:58:13 RP: oh well the idea is. That each task gets reported individually. So MACHINE, DISTRO, PN, PV, PR, TASK Oct 02 15:58:32 RP: gets reported as the HTTP header (but this works today as well) Oct 02 15:58:54 zecke: run tests in qemu? Oct 02 15:58:57 RP: my tinderbox will allow to 'compose' one long log file from all tasks Oct 02 15:59:30 koen: well. You can do two things. We allow some people to upload images/ipk's Oct 02 15:59:59 koen: other boxes can use them and test and post results refering to the original upload Oct 02 16:00:17 zecke: right, but only images built by a tbox run, right? Oct 02 16:00:42 zecke: or signed using some key so scriptkiddies won't upload forkbombs, etc Oct 02 16:00:43 koen: e.g. RP builds poky, I download and run it Oct 02 16:01:55 zecke: RP's tinderclient builds poky and uploads it and it get run? Oct 02 16:02:25 koen: I could have a n770 sitting here and could flash+run the image Oct 02 16:02:29 zecke: you'd want to have a (co)relation between images and builds Oct 02 16:02:41 koen: and have a predefined testing procedure and report it Oct 02 16:02:59 03pH5 07org.oe.dev * r9482aba3... 10/ (4 files in 3 dirs): linux-libc-headers-2.6.18: don't sanitize the secret kernel syscall macros away, util-linux wants them Oct 02 16:03:14 koen: see above. I download the image from one RUN. And do the testing. When reporting I refer to the original build Oct 02 16:03:25 koen: I think that is good enough Oct 02 16:03:53 right Oct 02 16:04:53 ideally it knows about scm's as well Oct 02 16:04:54 zecke: If those things are part of the header, that should mean multithreading should work, unless tinderbox makes assumptions about task orders? Oct 02 16:05:03 or at least about 'revisions' Oct 02 16:05:53 Does anyone have any thoughts on the profile graph I posted? Oct 02 16:06:06 3.5% seemingly in os.path.join? :-/ Oct 02 16:06:07 RP: current tbox3 just apppends :) Oct 02 16:06:20 RP: posted to mailing list? Oct 02 16:06:31 zecke: yes. bitbake and oe Oct 02 16:06:44 zecke: in reply to Pauls mail (bitbake past, present and future) Oct 02 16:06:48 zecke: http://rafb.net/paste/results/PthFQf27.html Oct 02 16:06:56 koen: SCM's. Which SCM's OE?, BitBake? the ones we build? Oct 02 16:07:13 RP write bits of bb in c/c++ and use swig to extract python wrappers Oct 02 16:07:18 or http://lists.linuxtogo.org/pipermail/openembedded-devel/2006-October/000483.html Oct 02 16:07:29 zecke: the ones that contain OE trees Oct 02 16:07:46 zecke: right now that's svn and mtn Oct 02 16:08:13 so you'd send 'svn status' and 'mtn automake get_base_revision_id' Oct 02 16:09:39 RP: hmm. feeder is quite low Oct 02 16:11:18 RP: oh well. I think the proper way would be to reduce the number of data calls Oct 02 16:11:44 zecke: right, the problem is in the dataclass and as we can speed it up, we need to reduce calls or cache things Oct 02 16:11:52 bye Oct 02 16:11:57 s/can/can't/ ! Oct 02 16:11:57 RP: e.g. mithro said that our string concatinations are really slow. If we do them less, or use stringbuffer we get around them Oct 02 16:12:34 zecke: which string concatinations? you mean things like "foo" + "bar" ? Oct 02 16:13:08 RP: yes. and even getVar() + "new val" Oct 02 16:13:24 RP: there might be a way to avoid this in the parser Oct 02 16:13:33 zecke: What worries me - 3.5% in os.path.join... Oct 02 16:13:48 RP: that was from files_path? Oct 02 16:14:00 zecke: quite possibly, I don't know Oct 02 16:14:13 filespath is certainly featuring on that list though Oct 02 16:14:54 RP: you use hotshot for profiling right? Oct 02 16:15:07 RP: you can easily sort by callers? e.g. show me everyone which called os.path.join Oct 02 16:15:17 zecke: maybe not. I just ran import profile Oct 02 16:16:05 These numbers were from when I fixed up taskdata Oct 02 16:16:49 RP: http://rafb.net/paste/results/IOFeZO96.html Oct 02 16:17:41 I think these calls can be cached easily Oct 02 16:18:45 RP: and the same in BBHandler.py Oct 02 16:20:32 I think by avoiding this we get 12.000 less calls of os.path.join Oct 02 16:20:48 twelve thousand? Oct 02 16:20:55 zecke: That would be a very good start :) Oct 02 16:21:10 CosmicPenguin: simple math/guess Oct 02 16:21:14 holy nuts Oct 02 16:21:17 CosmicPenguin: That run shows over 200,000 calls to os.path.join :-/ Oct 02 16:21:26 CosmicPenguin: 4 thousand .bb files Oct 02 16:21:39 CosmicPenguin: most of them parsing base.bbclass so we are at eight thousand Oct 02 16:21:58 CosmicPenguin: all of them include .inc files so we are at 12 thousand Oct 02 16:22:20 zecke: Note that bb.data.init() was a big factor in the graph as well Oct 02 16:22:45 really? Oct 02 16:23:18 zecke: Sorry, I'm getting confused Oct 02 16:23:34 welcome :) Oct 02 16:23:45 Melancholic + Nuts == average OE dev Oct 02 16:24:05 zecke: This sort of experience with python will also come in handy on OLPC.. :) Oct 02 16:24:14 They need huge optimization help Oct 02 16:24:16 zecke: lib/bb/__init__.py, 2100 times but its an expensive call. I wonder if we can just inherit that from OE somehow? Oct 02 16:24:40 RP: oh well, I try to remember mickeyl Oct 02 16:24:44 c Oct 02 16:24:51 RP: import bb is executed exactly once Oct 02 16:25:01 RP: from bb import data is executed every time Oct 02 16:26:29 CosmicPenguin: I have built sugar, and I failed to understand the user concept :) Oct 02 16:26:33 heh Oct 02 16:26:36 CosmicPenguin: at least I found the browser and F5 Oct 02 16:26:38 At least you got it to run Oct 02 16:26:52 CosmicPenguin: I have used jhbuild and it needed only one handwork Oct 02 16:27:08 zecke: I don't understand all of the services it needs Oct 02 16:27:25 I'm sure there needs to be hacking to start avahi and dbus and whatnot before sugar, but I don't understand any of that Oct 02 16:27:32 CosmicPenguin: are your sugar .bbs online? Oct 02 16:27:37 when you are ready, I'll send you the shi^H^H^H .bbs that you need Oct 02 16:28:03 I'll try to get them online sometime soon Oct 02 16:35:27 03pH5 07org.oe.dev * rf69b3e75... 10/ (1 conf/machine/ipaq-pxa270.conf): ipaq-pxa270.conf: remove movules from PXA270_MODULES that are already accounted for in task-base Oct 02 16:35:38 03pH5 07org.oe.dev * r0888a421... 10/ (1 packages/tasks/task-base.bb): task-base: add pcmcia and pxa27x_udc kernel modules to pcmcia/usbgadget-rrecommends Oct 02 16:37:13 uh, 'movules'. I hope I put those typos only into the commit messages. Oct 02 17:08:22 hi andersee Oct 02 17:08:43 woglinde: morning Oct 02 17:09:00 * andersee decides to try and get something useful done today for a change Oct 02 17:09:07 hm Oct 02 17:09:29 * zecke concludes andersee is not a manager yet Oct 02 17:09:34 hehe Oct 02 17:09:43 andersee manages his family Oct 02 17:14:41 andersee: uclibc release? Oct 02 17:15:48 sigh - apparently 2.6.18-rc6 is newer then 2.6.18 in ipkg's world.. :( Oct 02 17:16:22 CosmicPenguin: right, that's why we have a versioning policy Oct 02 17:16:25 nod Oct 02 17:16:46 but that doesn't work when I stupidly roll my own Oct 02 17:16:48 2.6.18rc -> linux_2.6.17+2.6.818rc.bb Oct 02 17:16:52 :) Oct 02 17:16:53 * CosmicPenguin goes away Oct 02 17:21:55 I'm wondering of someone can help me understand a couple of things about using the oe filesystem in read-only mode.. Oct 02 17:22:46 I have the root entry set to 'defaults' in fstab - as per the default fstab Oct 02 17:23:00 and I pass ro to the kernel from my bootloader Oct 02 17:23:20 this seems to work ok... the root filesystem is mounted as read only Oct 02 17:23:37 then the init scripts run, and somethings get remounted as rw Oct 02 17:23:48 specifically /var - since it's a tmpfs Oct 02 17:23:59 oe filesystem? Oct 02 17:24:26 koen: I mean the base filesystem that gets generated from a base checkout... Oct 02 17:24:29 generic x86 Oct 02 17:24:47 using prety much the stock initscripts Oct 02 17:24:59 koen: keep working on it Oct 02 17:24:59 ( I have disabled a couple that I don't need) Oct 02 17:25:20 hello ppl Oct 02 17:25:23 koen: I hope to get a needed uClibc patch sorted out today and tested Oct 02 17:25:32 there is a bootmisc initscript, which is supposed to ensure that /tmp is a symlink to the tmpfs at /var Oct 02 17:25:52 but my /tmp seems to be a directory, not a symlink Oct 02 17:26:54 I'm just a little puzzeled as to why that is Oct 02 17:28:53 is what I'm doing (forcing a ro rootfs by passing ro as a kernel param) either a normal, or a sane way to achieve a read-only root filesystem? Oct 02 17:29:40 as long as the typically volatile filesystems are put into ram, you should be fine Oct 02 17:30:05 hi kergoth Oct 02 17:30:19 tkp: I think checkroot.sh remounts your rootfs rw if you don't explicitly put ro into the fstab, too. Oct 02 17:30:55 when that bootmisc initscript is run, it complains that / is readonly Oct 02 17:31:16 which is why it's failing to ensure that /tmp is indeed a symlink to /var Oct 02 17:32:24 pH5: I can see that it might do that... but it doesn't Oct 02 17:32:47 my problem is the opposite... at the point when bootmisc runs, the root is readonly Oct 02 17:33:51 I wonder why that /tmp symlink has to be generated by an init script instead of during the rootfs generation. Oct 02 17:34:07 pH5: I wondered the same myself Oct 02 17:39:00 anyone care for some simple live bug reports to fix on the spot? Oct 02 17:40:54 he mickeyl Oct 02 17:41:15 likewise: maybe Oct 02 17:41:15 I have 30 minutes left till CSI starts Oct 02 17:41:16 heya Oct 02 17:41:26 koen hehe Oct 02 17:41:27 koen: will you bring your MOTO to oedem? Oct 02 17:41:29 so, what's the purpose of volatile.cache Oct 02 17:41:30 here only 15 Oct 02 17:41:46 since it resides in /etc (which can not be written to in a ro rootfs) Oct 02 17:41:56 I think that's where my problem may lie Oct 02 17:42:32 I tried symlinking that file to somwhere in /var Oct 02 17:42:46 and looking at it, it's full of error messages Oct 02 17:42:47 mickeyl: yes, it's my primary phone Oct 02 17:42:47 mickeyl: I'll also bring the nbp if I can revive it Oct 02 17:42:48 koen: #1: lzma DEPENDS on zlib (because it uses the zlib.h header during building) Oct 02 17:42:55 koen: excellent, then i can leave mine at home. i want marcin to have a look. Oct 02 17:43:32 complaining 'Failed to set owner ....' '"Failed to set mode...', I think, one for each of the thinks in volatiles Oct 02 17:43:44 likewise: lzma-native? Oct 02 17:43:54 koen: both -native, yes. Oct 02 17:44:28 koen: #2 (related) zlib-native 1.2.3 fails to stage zlib.h, 1.2.2 works fine. (1.2.3 was converted to autotools) Oct 02 17:44:57 koen: EOF. Oct 02 17:45:14 03koen 07org.oe.dev * rf0096d0a... 10/ (1 packages/lzma/lzma-native_4.17.bb): lzma: depends on zlib-native, spotted by Leon Woestenberg Oct 02 17:45:51 likewise: zlib cross works? Oct 02 17:45:54 #/me goes incognito Oct 02 17:46:10 koen: untried for... Oct 02 17:48:02 03koen 07org.oe.dev * r0b5e6bdf... 10/ (1 packages/zlib/zlib-native_1.2.3.bb): zlib-native: stage zlib.h, spotted by Leon Woestenberg Oct 02 17:48:22 likewise: happy now? Oct 02 17:48:23 :) Oct 02 17:49:13 koen: didn't need any cre^H^H^Hblame me. Oct 02 17:49:17 -me Oct 02 17:49:36 koen: zlib cross seems to stage OK. Oct 02 17:49:46 how can bootmisc ever expect to run on a readonly filesystem? Oct 02 17:50:14 since the drive is mounted as readonly in mountall Oct 02 17:50:21 (which runs before) Oct 02 17:52:05 03Robert 07org.oe.dev * rb4ec3864... 10/ (6 files in 4 dirs): add scsi idle: turn off SCSI disks when idle Oct 02 17:52:45 likewise: some extra google juice for your efforts in the embedded world can't hurt :) Oct 02 17:53:33 koen: enough juice already, and the 1% increase in spam will hardly be noticable anyway :-) Oct 02 17:56:17 koen: thanks for the quick fix. You just closed a bug report and prevented me from filing a second one :-) Oct 02 17:56:27 :) Oct 02 18:03:54 likewise: isn't it likewise, MSc nowadays? Oct 02 18:04:25 koen: yes Oct 02 18:05:45 koen: finally, after all those years (computers can really be distracting, you know? ;-)) Oct 02 18:06:02 * koen pretends to don't know Oct 02 18:07:35 so, anyone got any ideas about this bootmisc script? Oct 02 18:08:00 tkp: it's buggy Oct 02 18:08:23 it's buggy? so this is a known problem... Oct 02 18:08:27 that it's basically impossible to use in a ro rootfs Oct 02 18:08:36 I didn't know of the poblem till you mentioned it tonight Oct 02 18:09:09 problem* Oct 02 18:11:01 wow django is nice Oct 02 18:12:21 zecke: like it? Oct 02 18:12:29 I've been thinking of trying it out Oct 02 18:13:00 ggilbert: I'm not far but currently it is promising. It allows rapid development like I know from python Oct 02 18:13:17 ggilbert: just use sqlite3 and you can just start developing Oct 02 18:14:53 ggilbert: I have just started and only used the model api so far Oct 02 18:15:17 ggilbert: I think it will be more complicated when thinking about performance, and good queries Oct 02 18:16:26 ggilbert: http://rafb.net/paste/results/iDLEUq58.html. Create the model and enter python shell and you can try your code to populate the db Oct 02 18:30:12 for those who love Eclipse(.org): http://www.mail-archive.com/monotone-devel@nongnu.org/msg05360.html Oct 02 18:31:36 kone: Even though i cleaned up the tmp folder, diet-x11 and libx11 failes. Here is the output: http://pastebin.ca/189016 What do i have to do to solve this? Or do you think there is a bug in the bb files? Oct 02 18:31:54 koen the line above this Oct 02 18:39:34 zecke: do you have any idea? Oct 02 18:42:25 goxboxlive: not now :) Oct 02 18:42:37 zecke: ok Oct 02 18:44:51 There have to be an bug in the file or so, because booth for diet-x11 and libx11 it complains about : makekeys.c:34:19: error: X11/X.h: Ingen slik fil eller filkatalog and that means, no such file or folder. And that goes also for 'Xos.h' and 'keysymdef.h' Oct 02 18:54:06 tkp: I think someone optimised the /var handling for speed and perhaps didn't consider someone wanting to use the filesystem readonly Oct 02 18:54:32 RP: I see... Oct 02 18:54:40 RP: S98configure is a killer for RO filesystems as well Oct 02 18:54:50 tkp: I don't know if it worked before that Oct 02 18:55:18 koen: Right, so some work is needed to make a working ro system :-/ Oct 02 18:55:18 koen: S98 fails to run too indeed Oct 02 18:55:46 but I'm much more bothered about getting /var set up correctly... with /tmp being a symlink Oct 02 18:55:57 RP: I think we need to get creative with qemu to solve that Oct 02 18:56:46 shouldn't mountall force things to be mounted as read/write... and then let one of the other scripts remount Oct 02 18:56:53 koen: Indeed. A load of postinst could run on offline and don't last time I looked though :-/ Oct 02 18:57:22 although stuff like blueprobe need to get adjusted Oct 02 18:57:45 as far as I can tell, currently mountall mounts things as specified in fstab... Oct 02 18:57:49 which in my case is ro Oct 02 18:58:09 hense, anything after it can not make any alterations to the filesystem Oct 02 19:02:20 hi Oct 02 19:02:27 hrw: wb Oct 02 19:04:01 anyone else fancy looking into this problem? I'm sure someone else could sort it 100 times faster than me! Oct 02 19:05:23 tkp: is it possible to run it once with RW? Oct 02 19:05:37 You have two options - you either make a way for the volatile directories to be written to or you spend a goodly amount of time re-writing the initscripts, which probably wouldn't work anyway because lots of things expect read-write to work Oct 02 19:05:47 RP: could be pinch sbrsh to do the heavy lifting? Oct 02 19:05:56 Its not super difficult - I've been using read-only directories for eons Oct 02 19:06:01 koen: not for a final dist Oct 02 19:06:31 tkp: use tmpfs for /var/ maybe? Oct 02 19:06:39 hrw: it is already... Oct 02 19:06:58 tkp: run it in rw once, save that and distribute it as RO Oct 02 19:08:15 but the point is, it's initially mounted as read only... and since /tmp is initially a directory, and it's bootmisc that turns that into a symlink - which it can't do on a read only filesystem Oct 02 19:08:15 but if you use openssh/dropbear then remember that it generate keys on first initstart Oct 02 19:08:38 hrw: this is true Oct 02 19:08:49 seesh - seriously Oct 02 19:08:51 man mount Oct 02 19:08:53 read about bind Oct 02 19:08:55 seesh Oct 02 19:09:00 hrw: we mentioned S98configure earlier :) Oct 02 19:09:16 koen: S98 is S98. dropbear generating keys is other Oct 02 19:09:42 koen: I remember patch to dropbear initscript to have keys generated and whole discussion about that Oct 02 19:09:59 I don't think running once in read-write is really a viable solution Oct 02 19:10:02 koen: it was one of my pre-r/w patches Oct 02 19:10:18 right, pregenerated keys are bad Oct 02 19:10:44 if you pregenerate they you can stop using encyption altogether Oct 02 19:10:54 or design a DRM system :) Oct 02 19:11:11 although, I can't really think of another way without fixing these initscripts Oct 02 19:14:54 heh... why ssh hrw@{one,two,three,...,any} ends on more powerfull machines then the one I'm using Oct 02 19:15:46 koen: I have found the bug in the diet-x11 and the libx11. But i dont know how to handle it. Oct 02 19:21:47 git can fetch over http or not? Oct 02 19:22:05 if the server supports it, yes Oct 02 19:22:26 thx Oct 02 19:27:19 Crofton: how goes booting recent kernel? Oct 02 19:27:32 meetings .... Oct 02 19:27:41 I need to run an errand and come back and flash Oct 02 19:27:50 thanks for nagging :) Oct 02 19:27:57 Crofton: ;) Oct 02 19:28:02 Crofton: your welcome Oct 02 19:46:10 * likewise wants mount -o union Oct 02 19:46:36 there are union tools and such in OE... Oct 02 19:48:45 JustinfP: yes, unionfs etc. Oct 02 20:01:58 does unionfs still oops like crazy? Oct 02 20:03:12 koen: CSI done? Oct 02 20:03:48 koen: dunno, did not hear much complaints about the more recent unionfs releases Oct 02 20:04:07 s/much/many Oct 02 20:05:52 kergoth: my priorities have not changed. to you it may seem like I am only speaking from TT's view, but I sit on a fence and also communicate community concerns to management. Oct 02 20:06:20 likewise: csi and stargate atlantis Oct 02 20:06:37 ljp: hey. :) Oct 02 20:07:19 hrw: are zaurus users that stupid and whiny, or is it just that oesf attracts morons? Oct 02 20:08:42 kergoth: also, there _will_ be a qtopia 4 gpl releases, and regular snapshot releases. I have had many discussions about this internally. Oct 02 20:10:25 ggilbert: I think I like django. The DB API is quite convient Oct 02 20:14:56 OLPC? Oct 02 20:14:58 hey zecke Oct 02 20:15:12 zecke: ping mickeyl or me if you need a mod_python enabled dir on oe.org Oct 02 20:15:30 zecke: and ping kergoth for a tindebox.oe.org CNAME :) Oct 02 20:15:46 koen: currently it is bonsai.oe.org Oct 02 20:16:00 btw you are correct, python string concat's are slow because strings are immutable Oct 02 20:16:25 zecke: bonsai, tinderbox, lxr, whatever Oct 02 20:16:39 mithro: I would like to do tinderboxing/QA for OLPC but Jim has not responded yet Oct 02 20:16:43 i really wish somebody created a "mutable string" which was in the standard library Oct 02 20:16:52 whats OLPC? Oct 02 20:17:01 mithro: $100 laptop thingy Oct 02 20:17:13 ahh Oct 02 20:17:33 os.join.path is string concatination btw :P Oct 02 20:18:01 mithro: Is there an easy way to avoid it? Oct 02 20:18:23 depends what you mean by avoid :p Oct 02 20:18:58 In the context of bitbake, how can we make it faster? :) Oct 02 20:19:12 well I have no idea why/when bitbake is using it Oct 02 20:20:04 my guess is that you shouldn't need more then 1 or 2 joins per bitbake file Oct 02 20:20:25 Right, I agree with that :) Oct 02 20:20:40 http://rafb.net/paste/results/jzVgHf91.html my bonsai updater for SVN Oct 02 20:21:44 actually, you shouldn't need any joins in parsing - only when building Oct 02 20:22:01 o btw, I'm unemployed now :/ Oct 02 20:22:21 mithro: worked for BenQ germany? Oct 02 20:22:36 mithro: When parsing, it does need to find files so I suspect we'll not get away without doing it Oct 02 20:22:43 mithro: :-( Oct 02 20:24:26 actually I doubt you need to do the joins Oct 02 20:24:45 finding the files is quite an easy process Oct 02 20:25:12 koen: what this time? Oct 02 20:25:55 no, the company i was working for layed of the whole section related to report generation (which really was just 3 days - inc me) Oct 02 20:26:48 any monotone guru here? Oct 02 20:26:56 i've still yet to figure out how they are going to furfill their contract obligations with other companies (as they have nobody who can keep the reporting stuff going :/) Oct 02 20:27:03 Im having issues with trying to push stuff to my monotone server keep getting denied Oct 02 20:27:08 RP: you got some profiling output? Oct 02 20:27:46 mithro: http://svn.berlios.de/wsvn/bitbake/trunk/bitbake/lib/bb/parse/parse_py/ConfHandler.py?op=file&rev=0&sc=0 in handle() is the kind of use we have of joins. I've never looked at the parser though Oct 02 20:27:54 mithro: yes, one second Oct 02 20:29:01 btw - some trivial optimisations Oct 02 20:29:28 mithro: http://www.rpsys.net/openzaurus/temp/bitbake-profile.txt Oct 02 20:29:35 "x.a.b()" is a lot slower then "b = x.a.b; b()" Oct 02 20:30:15 that loop is doing things totally the wrong way Oct 02 20:30:49 btw - why check if it's readable? Oct 02 20:30:53 if os.access(currname, os.R_OK): ? Oct 02 20:31:15 The file might not exist? Oct 02 20:31:48 but that error would be caught by opening it with 'r' Oct 02 20:32:22 is try except faster than the readable check? Oct 02 20:32:40 depends on the common case Oct 02 20:32:53 I suspect the common case is file not found Oct 02 20:33:34 then it should be an os.path.exists? Oct 02 20:34:26 one way to get around a os.path.join is to do a cwd Oct 02 20:34:44 RP: do you have spectrum based wifi card? Oct 02 20:34:49 hrw: I do Oct 02 20:34:53 but in this case I think it would be slower... might be something to test Oct 02 20:34:56 rm_you|: get it to oedem please Oct 02 20:35:03 RP: get it to oedem please Oct 02 20:35:11 it would also depend on path length Oct 02 20:35:19 RP: would be nice to make small zaurus hack party too Oct 02 20:35:26 the bigger the strings the slower the concat Oct 02 20:35:33 hrw: What's up with it? Oct 02 20:35:59 RP: I do not remember does 3.5.4.2-rc2 contain everything to get it working on poodle Oct 02 20:36:15 mithro: BBHandler in the same directory is probably the bigger problem as we have a lot more .bb's than conf files. It has all the same problems though Oct 02 20:36:54 def obtain(fn, data = bb.data.init()): Oct 02 20:37:07 mithro: As you can see from the profile - the biggest cpu user is expand in the data class. We really need to reduce its usage to get the most speed improvements I guess Oct 02 20:37:09 are you sure you want what that line does? Oct 02 20:37:41 bb.data.init() will only get called once Oct 02 20:38:25 mithro: It won't be called each time the function is called with that arugment missing? Oct 02 20:38:34 no Oct 02 20:38:37 zecke? :) Oct 02 20:38:46 it will be called when the function object is created Oct 02 20:39:40 default arguments are bound at the time the function (object) is created Oct 02 20:39:43 ie Oct 02 20:39:52 That is probably not what is intended but I'd ask zecke about that one... Oct 02 20:40:09 def a(b, c=[]): c.append(b); print c Oct 02 20:40:14 hrw: I don't know about poodle - I can probably test though Oct 02 20:40:17 RP: we shouldn't call this without an argument anyway :) Oct 02 20:40:19 it will be with me Oct 02 20:40:38 RP: it is from before I have ever touched BitBake and did not want to break 'compability' Oct 02 20:40:40 a('a') -> ['a']; a('b') -> ['a', 'b'] ! Oct 02 20:40:46 aha now libx11 and diet-x11 compiled. It needed xorg headers installed on my machine. Shouldent this in the staging folder? Oct 02 20:41:01 that is a WTF which gets a lot of people Oct 02 20:41:31 mithro: I can see that. I didn't realise python worked like that... Oct 02 20:41:53 admitingly me neither but this is really old code :) Oct 02 20:42:03 That file is full of them... Oct 02 20:42:17 my guess is that you actually never called it without data Oct 02 20:42:31 because that would make little sense Oct 02 20:42:35 I'm going to remove it and see what happens... Oct 02 20:42:36 mithro: right, it is from bitbake 1.x where x <= 1.2 Oct 02 20:43:37 btw you are better off using lists everywhere Oct 02 20:43:45 you seem to be continually splitting and joining Oct 02 20:44:20 split when you do the parsing then treat it as a list Oct 02 20:45:08 doing that will also make a lot of your string concats go away (as they become list additions) Oct 02 20:45:12 but it's a rather major change Oct 02 20:45:32 mithro: splits in which file in particular? Oct 02 20:46:02 or you mean split certain variables up? Oct 02 20:46:50 that would be one major change... Oct 02 20:48:08 mithro: From what you said before, import domain from bb.msg would be faster than referencing bb.msg.domain all the time? Oct 02 20:48:10 ie you currently have : and such Oct 02 20:48:18 RP kinda of Oct 02 20:48:33 do a "import bb.msg.domain" Oct 02 20:48:49 then in the begining of your function do "domain = bb.msg.domain" Oct 02 20:48:59 doing it in the beginining of your function is the key Oct 02 20:49:10 because then it's a direct local lookup Oct 02 20:49:19 right, I see :) Oct 02 20:49:54 currently it goes, "local to function, globals, bb, msg" Oct 02 20:50:17 if you make it a function local it's a huge imporvment - specially with stuff called in loops Oct 02 20:50:18 so its like searching PATH in a shell :) Oct 02 20:52:18 kind of Oct 02 20:52:48 mithro: Are references resolved in advance or at execution time? e.g. if you have some long reference in a loop but in an uncommon path, will it cost you speed? Oct 02 20:52:58 storing things as strings is the anti-python methodology Oct 02 20:53:05 not really Oct 02 20:53:20 it only make an effect when you call the thing 100,000 times Oct 02 20:53:39 ok Oct 02 20:54:05 mithro: The problem is that the metadata is string based :-/ Oct 02 20:55:56 you want to keep strings short and immutable Oct 02 20:59:54 Does IMAGE_TYPES = squashfs compress to a squashfs using LZMA compression by default (as it depends on it) ?? Oct 02 21:00:45 likewise: it use squashfs normal iirc Oct 02 21:01:08 wrtoe distro defined squashfs-lzma for own usage iirc Oct 02 21:01:33 likewise: did you find lzma patch for recent squashfs maybe? Oct 02 21:03:31 most of your stuff is lists of strings Oct 02 21:03:53 anyway - time to bed for me Oct 02 21:03:59 cu you tomorrow guys Oct 02 21:04:11 cu hrw|gone Oct 02 21:05:01 mithro: This is another nasty that shows up on that profile: http://www.rpsys.net/openzaurus/temp/base.bbclass Oct 02 21:06:11 which part? Oct 02 21:07:03 that again is split/joins Oct 02 21:07:14 Just the variable becomes a very long string and its run for every .bb file Oct 02 21:07:49 right, and I'd bet thats where a lot of the os.path.joins come from Oct 02 21:08:15 possible Oct 02 21:08:43 i wish somebody would pay me to optimise bitbake - then I would have no excuse to put it off :P Oct 02 21:09:42 koen|away: how did your SoC go? Oct 02 21:09:59 mithro: I added an example of what FILESPATH becomes to that link Oct 02 21:10:33 mithro: I wish someone would pay me to work on bitbake - I do far too much of it in my free time :-/ Oct 02 21:11:49 RP: we should start bitbake inc then get CELF, Nokia and AMD to throw money at us ;P Oct 02 21:11:58 hah Oct 02 21:11:59 has if Oct 02 21:12:19 mithro: You can rule Nokia out - they have scratchbox Oct 02 21:13:58 The overrides = overrides + ":" in that files looks unneeded at the very least Oct 02 21:14:34 RP: thats why we need to become jedi first so we can use our jedi mind tricks to get them to forget it! :P Oct 02 21:14:42 * mithro has been playing to much Kotor Oct 02 21:14:48 :) Oct 02 21:15:23 just finished the second one an hour or two ago Oct 02 21:15:55 RP: (after looking at the example) no wonder python is having a spaz Oct 02 21:17:18 spliting and manipulating that string must take virtuall forever Oct 02 21:17:50 mithro: yes :-( Oct 02 21:18:19 split it into a list and keep it that way - bazillion times faster Oct 02 21:19:07 but the metadata doesn't work that way :-/ Oct 02 21:20:25 why not? Oct 02 21:20:50 with a little bit of tricks you could easily make it transparent to the meta data Oct 02 21:21:26 mithro: When you run getVar, everything expects a string? or are you proposing a getVarList ? Oct 02 21:21:51 also, some variables are split by :, some by " " Oct 02 21:22:24 i am proposing that you make the internals aware of lists of strings and then convert the meta data to lists when it's parsed Oct 02 21:22:57 but how do you know that a given variable is a list or not? Oct 02 21:23:31 dunno, guess? Oct 02 21:24:00 treat everything as lists of strings? Oct 02 21:24:21 Lists of one item would be the problem (OVERRIDES = "conf") as you'd guess wrong... Oct 02 21:24:47 actually treating everything as a lists of strings may work quite well Oct 02 21:24:48 Sometimes, the variables are complete functions Oct 02 21:25:33 you could make a "list of strings" object which behaves like a list or a string depending on the calling convention Oct 02 21:26:00 That would be the only way we could convert... Oct 02 21:26:12 then use the lists of strings feature to optimize cases Oct 02 21:26:52 That could work Oct 02 21:27:09 the other option would be to create a mutable string class Oct 02 21:27:44 I quite like the sound of that option... Oct 02 21:30:28 that doesn't solve the split/join problem however Oct 02 21:30:31 which i think is a pretty common case Oct 02 21:30:55 It is Oct 02 21:33:22 which is more common, lists or real strings? Oct 02 21:33:56 in the long term could you fix the meta data so that it makes a distinction? Oct 02 21:34:08 it would really make your life so much similar Oct 02 21:34:23 s/similar/simpilar/ Oct 02 21:35:50 in the metadata, blobs are much more common than lists Oct 02 21:36:01 mithro: we wont be able to talk anyone out of not using scratchbox until we're a usable development tool. we're far too weak on that end to compete with it for many companies Oct 02 21:36:05 but the blobs are stored as strings Oct 02 21:37:58 RP: the metadata doesn't make any distinction between lists/blobs? Oct 02 21:38:28 i havn't looked at BB metadata in 6 months so I can't really remember the details Oct 02 21:38:56 mithro: everything is a string, including 200 line functions Oct 02 21:39:23 mithro: all in one big data dictonary which we have to search at each data lookup Oct 02 21:39:46 that data dicitionary needs a lot of optimistation too... Oct 02 21:40:04 mithro: The data dictonary is right at the top of that profile :-/ Oct 02 21:40:43 i know, the parser/data dictionary was what i was going to concentrate on with my SoC proposal Oct 02 21:40:56 there are quite a few things which can be done to speed it up Oct 02 21:41:04 it's pretty horrible python in many ways Oct 02 21:41:48 http://www.rpsys.net/openzaurus/temp/bitbake_data_dump.txt gives an idea of the variables in the dictonary Oct 02 21:42:23 hrw|gone, kernel doesn't seem to boot Oct 02 21:42:44 mithro: http://svn.berlios.de/wsvn/bitbake/trunk/bitbake/lib/bb/data_smart.py?op=file&rev=0&sc=0 expand() is the biggest bottleneck Oct 02 21:43:02 yeah Oct 02 21:43:16 the way you do expanding just plan sucks :P Oct 02 21:44:13 mithro: How do we do it better? :) Oct 02 21:44:30 it's partly a metadata problem Oct 02 21:45:03 The flexibility of the metadata is a big part of OE... Oct 02 21:45:16 yeah i know Oct 02 21:48:24 RP: how big is the OE metadata? IE whats a du -h of it produce? Oct 02 21:48:41 mithro: The .bb files? Oct 02 21:48:46 yeah Oct 02 21:50:14 151MB Oct 02 21:50:43 how much memory does bitbake use currently (after parsing the whole bb data)? Oct 02 21:50:55 mithro: (/RP) That's probably only the org.openembedded.dev branch Oct 02 21:51:32 mithro: Not that much as we throw away most of the data after parsing due to memory issues Oct 02 21:51:55 what is the way to parallelize bitbake tasks, I cannot find the variable in the manual? Oct 02 21:51:56 mithro: Its faster to reparse any .bb file we need again Oct 02 21:52:17 147MB of that is in actual packages btw Oct 02 21:52:22 so currently you do a first parse to just build the dependency graph? Oct 02 21:52:24 although some will be patches Oct 02 21:52:34 mithro: yes Oct 02 21:53:01 RP: what about making the bb cache list-based? then anything past the parser can count on things being lists Oct 02 21:53:03 mithro: well, we save any data, accessed by the dependency / task planning code Oct 02 21:53:46 * mithro ponders Oct 02 21:54:20 mithro: The cache is transparent to the dependency graph code - we just watch what it accesses and store that so on a subsequent run, we can return that data instead of reparsing everything Oct 02 21:54:24 i think i know a way to make sure variables are expanded only once, and then only when they are needed Oct 02 21:54:53 That would be a big help as expanding them once would be much better than expanding all the time Oct 02 21:55:40 it would be very similar to how the COW stuff works Oct 02 21:56:08 I think I might have been wondering about something similar :) Oct 02 21:56:20 although I'm not sure my python skills are up to it Oct 02 21:56:35 to bad my time managment isn't up to it :/ Oct 02 21:56:47 :-( Oct 02 21:57:16 Why isn't DEPENDS_append_machine = " libmad" written as DEPENDS_machine += "libmad" ? Oct 02 21:57:37 I'm guessing it would take about 40 man hours to complete and test Oct 02 21:58:10 likewise: Presumably there is risk of something resetting DEPENDS Oct 02 21:58:55 likewise: those are two different operations. DEPENDS_machine will overwrite the contents of DEPENDS upon expansion. _append_machine conditionally appends. _machine += appends to a conditional Oct 02 22:01:16 RP: there is a chance I'll do some work on it now that I have 1 day free a week Oct 02 22:01:58 mithro: any help would be gratefully received... Oct 02 22:02:03 kergoth: So _machine += will add a condition to the existing conditions (if any) ? Oct 02 22:02:24 I guess the other thing is that I don't actually use any OE work at the moment Oct 02 22:02:42 no, it appends to a foo_machine variable, which is the machine specific version of that variable. it will blow away foo when it expands, not appending to it. Oct 02 22:02:58 likewise: The behaviour of using overrides and += is unpredictable and I'd keep off it Oct 02 22:03:11 * mithro really wishes his memory would get here Oct 02 22:03:25 well, not exactly unpredictable... Oct 02 22:03:43 unintuitive. Oct 02 22:03:43 * hvontres|poodle wishes his had never left him...:) Oct 02 22:03:45 not unpredictable Oct 02 22:03:52 mithro: Actually using OE does give an incentive for development... Oct 02 22:03:59 kergoth: right :) Oct 02 22:04:31 kergoth: tnx Oct 02 22:04:33 yeah, my zaurus is sitting in a box Oct 02 22:04:37 RP: tnx Oct 02 22:04:43 it all comes back to the weakness of the way we do overrides. the append/prepend mess was a consequence of that Oct 02 22:04:51 my laptop is small enough that it goes everywhere with me, so no need for the zaurus Oct 02 22:04:53 * hvontres|poodle pondering what COW is...googled "python COW" and found lots of COWtapults :) Oct 02 22:05:04 COW = Copy on Write Oct 02 22:05:17 mithro: ahh thanks... Oct 02 22:07:10 Can I test bitbake's multi-threading features already? Oct 02 22:07:46 RP: actually my idea would make all the data stuff thread safe Oct 02 22:09:00 zecke: how goes the test cases for bitbake? Oct 02 22:09:11 mithro: threading and python is something I've not got into, other than knowing how to fork. How would it help? Oct 02 22:10:46 the biggest problem you need to be careful of, mutable stuff is shared between threads - so need to make sure you copy something if you want the other threads to not see the changes Oct 02 22:11:13 there is also the global intepretor lock you need to watch out for Oct 02 22:11:42 it's not like C, you can't crash or anything Oct 02 22:12:04 mithro: If I call python's fork, that behaves the same way as C though? Oct 02 22:12:15 never used it Oct 02 22:12:27 but i guess it would just be a wrapper around the C fork Oct 02 22:12:46 I decided to stick to what I knew - fork it as needed and use waitpid :) Oct 02 22:12:57 waitpid? Oct 02 22:13:34 mithro: http://svn.berlios.de/wsvn/bitbake/trunk/bitbake/lib/bb/runqueue.py?op=file&rev=0&sc=0 Oct 02 22:13:39 although, does bitbake really need to be multithreaded? a non-blocking version should be able to spawn multiple stuff Oct 02 22:13:43 mithro: waitpid is a unix thing Oct 02 22:14:05 s/unix/C/ Oct 02 22:14:58 mithro: Which module would you recommend for python threads (or "spawning multiple stuff") ? Oct 02 22:16:27 rather intresting way of doing it Oct 02 22:16:53 mithro: very C like ;-) Oct 02 22:16:55 seems rather complicated Oct 02 22:17:33 It just has to be careful about all the error cases Oct 02 22:18:36 why don't you just build a proper tree? Oct 02 22:19:06 mithro: What do you mean by proper tree? Oct 02 22:19:08 s/tree/graph Oct 02 22:20:06 I tried several things and couldn't see to manipulate them the way I needed. Pointers at good graph implementations welcome... Oct 02 22:20:11 seems like your runqueue should be the following, Oct 02 22:21:38 while true: if (len(running) < NUMBER_OF_THREADS): run(graph.nexttask()) else wait() Oct 02 22:23:31 actually you could even make graph impliment the iter stuff properly and that would be even more natural Oct 02 22:24:09 Is there an existing graph implemenation? Oct 02 22:24:39 probably, but I would most probably end up roling my own very simple one in about 1-2 hours Oct 02 22:25:06 Set would be a good start Oct 02 22:25:19 ah well. Nice to know I totally waste my spare time :) Oct 02 22:25:45 it works does it not? Oct 02 22:26:02 on the most part it does Oct 02 22:26:18 so it's a 1000 times better then my implimentation at the moment ;) Oct 02 22:26:26 ;-) Oct 02 22:26:42 python is really a KISS language Oct 02 22:26:51 as in KISS my ass Oct 02 22:27:27 IE If you having trouble reading your code in english it's most probably to complicated Oct 02 22:28:12 mithro: The problem is that code is more C like than python. Since I think in C, it probably reads better to me though :) Oct 02 22:28:30 RP: probably Oct 02 22:28:47 mithro: It also runs surprisingly quickly once I nuked a single bottleneck (in taskdata) Oct 02 22:29:07 btw how do you choose which tasks to start first? Oct 02 22:29:27 mithro: There's a weighting algorithm in there Oct 02 22:29:40 guessing bitbake doesn't really know about actual build times Oct 02 22:29:59 GCC/GLIBC still a graph bottleneck? Oct 02 22:30:06 We just build the tasks with the most dependent tasks first Oct 02 22:30:38 and glibc comes quite well up there but another thread can be fetching, unpacking, patching so its not really a problem Oct 02 22:31:12 btw do multiple threads do fetching at one time? Oct 02 22:31:25 mithro: They can Oct 02 22:31:40 or is there one fetching, one unpacker, one patcher, etc Oct 02 22:31:58 (IE Pipelineing Oct 02 22:32:00 ) Oct 02 22:32:03 would be good to consider what system resources are used most by a given task, i remember us talking about needing that back when we discussing the possibilities for threading when schurig and i were first hashing it out Oct 02 22:32:14 The whole "how many threads to run" algothirm still needs writing. The fixed number of threads there is a better than nothing implemenation :) Oct 02 22:33:13 I think the best implimentation would be a pipeline with multiple threads at some points in the line Oct 02 22:33:17 kergoth: I remember the dicussion well. The idea is we can fit an algorithm into runqueue when we're ready - it has a "get_next_task()" hook ready Oct 02 22:33:25 cool Oct 02 22:33:51 ie it doesn't make much sense to have all threads downloading (as they would compete for bandwidth) Oct 02 22:34:18 mithro: The thinking is you'd set a limit on the number of threads of a given type that would run (say 3 fetchers) Oct 02 22:34:34 yeah Oct 02 22:34:47 that would even make sense on a single CPU machine Oct 02 22:34:54 mithro: right :) Oct 02 22:36:00 (then it would be 1 thread for each type) Oct 02 22:36:23 maybe a few for the fetcher if you have a fast link and slow downloads Oct 02 22:36:46 btw do you still spawn wget for http/ftp downloads? Oct 02 22:36:55 mithro: we do Oct 02 22:37:39 while i'm dreaming about improvements.... Oct 02 22:37:56 having that in python would allow you to run as many threads as bandwidth avaliable Oct 02 22:38:34 suppose you could use twisted for that, no? heavy dep though Oct 02 22:38:46 ie while (used_bandwidth_average < actually_average): spawn more Oct 02 22:38:58 kergoth: no need, the python inbuilt library would be good enough Oct 02 22:39:26 but then again i'm biasest against twisted Oct 02 22:39:31 bitbake knows how to fetch a lot more than http/ftp. Oct 02 22:39:55 sure, svn, cvs, mecurial, etc Oct 02 22:39:58 i'd rather see something independent like libcurl get the fetch stuff we support, but.. Oct 02 22:40:22 fetchers and bandwidth limits are a low priority... Oct 02 22:40:43 eventually if you had all handlers able to report currently used bandwidth (however it's done) Oct 02 22:41:22 thats just another thing to put on the never ending todo list :P Oct 02 22:41:50 I can near enough guarantee I'll not be doing that one ;-) Oct 02 22:43:33 :P Oct 02 22:45:19 maybe bitbake could burn my excess calories for me Oct 02 22:45:24 that would be a cool feature! Oct 02 22:46:19 I think I just found a 40% speed increase in parsing :) Oct 02 22:48:03 o, how? Oct 02 22:48:21 http://www.rpsys.net/openzaurus/temp/bitbake_expand_cache.patch Oct 02 22:49:48 i would be a bit careful there Oct 02 22:50:07 wouldn't that rapidly become very very large? Oct 02 22:50:32 Testing with metadata says its 40% faster... Oct 02 22:50:39 any write nukes the cache Oct 02 22:50:59 compare the memory usage? Oct 02 22:51:16 For a 40% speed improvement, I'm not sure I care ;-) Oct 02 22:51:40 can I make python show memory usage? Oct 02 22:52:30 profile should be able to show memory usage Oct 02 22:53:58 memory usage looks the same looking at top Oct 02 22:54:04 ~50MB max Oct 02 22:54:13 a pipelined approach with only one fetching thread does not take into account that the bandwidth bottleneck may be at the server (i.e. it makes sense to have multiple download threads). Oct 02 22:54:28 likewise: That was why I said 3 Oct 02 22:54:40 RP: good :-) Oct 02 22:54:41 likewise: but its all going to be configurable Oct 02 22:54:52 someone just needs to write the code ;-) Oct 02 22:55:47 Design First, Code Later. (or was it Design in Code, and Code on Paper ?) Oct 02 22:56:48 RP: how does that mix with COW? Oct 02 22:57:07 RP: line 50 is the ctor? Oct 02 23:02:31 nite Oct 02 23:06:02 nite all Oct 02 23:18:59 'night all Oct 02 23:21:01 night **** ENDING LOGGING AT Tue Oct 03 02:59:57 2006