**** BEGIN LOGGING AT Sat Jan 23 02:59:57 2010 Jan 23 10:56:26 hrm Jan 23 10:56:53 with the latest dist-upgrade my babbage hangs somewhere in the init-bottom scripts Jan 23 11:37:11 wow, i cant even chroot into the disk anymore from initramfs Jan 23 12:04:53 you broke it Jan 23 12:06:08 yeah, definately Jan 23 12:06:16 no way to get it booting again Jan 23 13:21:16 sigh Jan 23 13:24:54 grrr and cdimage isnt reachable through rsync Jan 23 13:24:58 * ogra goes mad Jan 23 13:35:53 * persia provides ogra with a tea in which to dip himself Jan 23 13:55:52 heh Jan 23 13:56:43 i really dont get what broke it Jan 23 13:56:56 looks like upstart hangs Jan 23 14:03:01 upstart hangs? Jan 23 14:03:21 Maybe set init to strace upstart? :) Jan 23 14:04:29 well, if i boot without initramfs its hangs at Freeing init memory: 176K Jan 23 14:04:38 no output afterwards at all Jan 23 14:09:00 OK. What if you boot with init=/bin/bash without initramfs? Jan 23 14:09:25 didnt try that yet, i'm just checking what updates come on a 1day old livefs Jan 23 14:09:41 since it stopped booting after apt-get dist-upgrade Jan 23 14:10:10 but there is nothing noticeable ... Jan 23 14:10:20 i removed plymouth from disk already Jan 23 14:10:41 busybox-initramfs cant be it since i did boot without initramfs Jan 23 14:10:47 and the rest are desktop libs Jan 23 14:10:53 That shouldn't have done much: I heard it didn't work for non-intel FBs right now. Jan 23 14:11:09 well, plars said he sees a splash Jan 23 14:11:13 (on dove) Jan 23 14:11:16 I know. I don't understand why. Jan 23 14:11:36 But slangasek said he didn't (on some non-intel x86 box) Jan 23 14:11:49 And that he was trying to sort it, and suspected it was intel-only. Jan 23 14:12:08 But my knowledge of plymouth stops after configure runs :) Jan 23 14:12:27 http://paste.ubuntu.com/361263/ thats what apt-get dist-upgrade on a running livefs gets me Jan 23 14:12:53 not sure what libbsd0 does Jan 23 14:13:15 seems its in standard Jan 23 14:13:49 It's some random BSD-only stdlib stuff that doesn't appear in linux. Jan 23 14:14:10 * ogra tries what happens if he uses the redboot setup from teh livefs to boot Jan 23 14:14:32 I don't see anything there that should break it either :( Jan 23 14:16:34 Begin: Running /scripts/init-bottom ... Jan 23 14:16:34 Done. Jan 23 14:16:38 hangs ... Jan 23 14:16:54 the funbs stuff is, it doesnt actually hang hard or something Jan 23 14:17:11 i have a cursor and pressing enter moves it on the screen Jan 23 14:18:23 Cool! Jan 23 14:18:26 its just that init doesnt seem to proceed Jan 23 14:18:29 set -x ? Jan 23 14:18:35 in init ? Jan 23 14:18:42 its a binary, not a script Jan 23 14:18:48 Ah, I missed the "Done." Jan 23 14:19:05 yeah Jan 23 14:19:19 Freeing init memory: 176K doesnt show up if using an initramfs Jan 23 14:19:28 i'll try /bin/bash Jan 23 14:20:51 bash: cannot set terminal process group (-1): Inappropriate ioctl for device Jan 23 14:20:51 bash: no job control in this shell Jan 23 14:20:51 root@(none):/# Jan 23 14:21:01 works Jan 23 14:21:22 OK, so it's not the kernel. Jan 23 14:21:42 no, else i wouldnt get into busybox either Jan 23 14:21:55 i can properly break and the like Jan 23 14:22:06 its also not uboot Jan 23 14:22:14 since redboot shows the same bahavior Jan 23 14:22:18 *be Jan 23 14:22:38 bootloader shouldn't be affecting anything post-kernel load anyway. Jan 23 14:23:05 if the HW is wrongly initialized it does Jan 23 14:23:18 else we wouldnt have issues with the NIC ion uboot :) Jan 23 14:23:20 *in Jan 23 14:24:40 Well, that's HW interfaces. I like to live in a happy little world where userspace apps just talk to each other. Sometimes USB HID is permitted. Jan 23 14:25:44 But yeah, blame upstart. See if there's some debug mode: perhaps you can get it to output which job it's trying when it hangs. Jan 23 14:26:14 well, either upstart or the first script that runs or so Jan 23 14:26:21 i dont get to fsck Jan 23 14:26:37 and i dont see the mount /dev to none error message Jan 23 14:27:10 Right, but for both cases you need upstart debug messages. Jan 23 14:27:13 Or strace Jan 23 14:27:39 gah, running dhclient in init=/bin/bash mode isnt such a good idea Jan 23 14:27:50 (can one strace init on boot? That would be very helpful to determine the issue, since it's likely to be either exec(...) or some HW call. Jan 23 14:27:57 no job control ... no ctrl-c Jan 23 14:27:58 heh. No! Jan 23 14:28:06 * ogra reboots Jan 23 14:28:41 i think you can start init from bash in that setup somehow, Keybuk has magic commands for that Jan 23 14:28:45 but i dont know them Jan 23 14:29:41 oh Jan 23 14:29:55 `strace /sbin/init` ? Jan 23 14:29:55 my mtab is totally messy Jan 23 14:30:27 write(2, "init: Failed to connect to socke"..., 74init: Failed to connect to socket /com/ubuntu/upstart: Connection refused Jan 23 14:30:27 ) = 74 Jan 23 14:30:30 pfft Jan 23 14:30:54 Try asking in #upstart. Might be some leftover upstream around. Jan 23 14:31:25 i can wait for monday too :) Jan 23 14:31:40 i wouldnt see why upstart should suddenly stop working Jan 23 14:31:58 especially since it works in the live image Jan 23 14:32:41 Are there any update-initramfs calls that happen with apt-get dist-upgrade? Jan 23 14:32:57 yes, plymouth enforces them Jan 23 14:33:06 but i'm currently booting a live initramfs Jan 23 14:33:12 Then get plymouth out before you upgrade, and see if that works. Jan 23 14:33:19 already done Jan 23 14:33:21 long ago Jan 23 14:33:24 mount, chroot :) Jan 23 14:33:27 that was my first setp :) Jan 23 14:33:37 *step Jan 23 14:33:40 Ah. Jan 23 14:33:47 since indeed i did install plymouth yesterday Jan 23 14:33:50 But you've already messed up the initramfs. Jan 23 14:33:54 and then ran a dist-upgrade later Jan 23 14:34:06 Thanks for that, by the way. I'd think it'd be cool if it worked. Jan 23 14:34:09 no, i have plenty of initramfses around Jan 23 14:34:22 lots of backups Jan 23 14:34:27 Well, stuff in a working initramfs, and try to boot :) Jan 23 14:34:35 its definately nothing in the initramfs Jan 23 14:34:54 the live initramfs boots the live system just fine Jan 23 14:35:09 so why shouldnt it boot my SATA disk Jan 23 14:35:35 and in the livefs plymouth is the older version Jan 23 14:35:42 The *same* initramfs can't boot your drive? Something is seriously wonky. Jan 23 14:35:49 yes Jan 23 14:35:56 but the issue has to be on the drive Jan 23 14:36:40 * ogra removes the messy mtab Jan 23 14:36:45 and rebbots ... Jan 23 14:37:45 iirc the first thing that runs on boot is mountall Jan 23 14:38:37 nope, mtab didnt help Jan 23 14:39:14 hey ... Jan 23 14:39:18 weekend reminder ;) Jan 23 14:39:29 bah .. Jan 23 14:39:39 fix my board and i can sleep tonight :) Jan 23 14:46:19 root@(none):/# cat /etc/init/mountall.conf Jan 23 14:46:19 root@(none):/# Jan 23 14:46:22 hmm Jan 23 14:47:50 heh Jan 23 14:48:55 ogra: you shouldnt have tried to upgrade just today ;) Jan 23 14:49:06 j.k. Jan 23 14:49:09 dont know whats going on Jan 23 14:49:10 asac_, i left it running yesterday Jan 23 14:49:12 still wrestling ffox Jan 23 14:49:23 when i got up today i just wanted to take a look Jan 23 14:49:31 Done. Jan 23 14:49:31 mountall: Could not connect to Plymouth Jan 23 14:49:31 fsck from util-linux-ng 2.16 Jan 23 14:49:31 e2fsck 1.41.9 (22-Aug-2009) Jan 23 14:49:31 /dev/sda6: clean, 146974/2321984 files, 1007361/9277521 blocks Jan 23 14:49:34 * ogra dances Jan 23 14:49:37 finally ! Jan 23 14:49:55 cheers Jan 23 14:50:07 https://edge.launchpad.net/ubuntu/lucid/+queue?queue_text=chromium-browser# Jan 23 14:50:09 ;) Jan 23 14:50:10 doesnt seem to move further though Jan 23 14:50:24 archive admins are laggers :) Jan 23 14:50:29 heh Jan 23 14:50:32 12h Jan 23 14:50:33 ;) Jan 23 14:50:51 hmm, i suspect there is more trashed in /etc/init/ Jan 23 14:50:53 crap Jan 23 14:55:34 asac_: None of the archive admins have signed up for saturday or sunday as archive admin days. You'll probably have to wait until Monday. Jan 23 14:57:06 i know Jan 23 14:58:16 So, how are they laggers? Jan 23 14:58:24 You could volunteer to be an archive admin :) Jan 23 14:58:45 persia, they didnt approve it last night before finishing their day Jan 23 14:58:47 and then you could pick one of the unclaimed days ... Jan 23 14:58:51 indeed they are laggers :) Jan 23 14:59:08 * persia does a timezone check Jan 23 14:59:09 it surely was in the queue for at least 1h before the last AA left :) Jan 23 14:59:30 Friday needs a backup archive-admin, it only has one. Jan 23 14:59:49 Tuesday would also qualify for a backup, but doesn't need one. Jan 23 15:20:52 bah, i'll do a reinstall on monday ... there is more broken than i thought ... Jan 23 15:21:25 always hitting the reset button when working on bootloaders doesnt do good to a filesystem :P Jan 23 15:21:44 though i thought ext4 was more robust Jan 23 15:43:27 ogra: what error are you getting? Jan 23 15:43:36 my usb-storage issues also complained about ext4 issues Jan 23 15:44:17 i havent got any errors Jan 23 15:44:28 i have some zero byte files Jan 23 15:44:41 but its to late now, i wiped the partition Jan 23 15:45:34 i threatened the disk really hard the last two weeks though ... its no wonder that there are corrupted files Jan 23 15:45:51 about 50 resets per day or so Jan 23 15:46:06 never shot it down properly Jan 23 15:47:36 asac_, our uboot installation will cause a lot more issues than we had yet btw Jan 23 15:48:05 it just struck me that we need to teach ubiquity that /boot can *only* live on an SD card Jan 23 15:48:46 i dont think there is any code in partman that could do that yet Jan 23 15:48:56 that will be a huge task to implement Jan 23 15:49:02 unless we also want to put the images directly to the boot floppy as a backup Jan 23 15:49:31 well, still the "boot floppy" has to be the SD Jan 23 15:49:38 no Jan 23 15:49:40 well Jan 23 15:49:41 yes Jan 23 15:49:43 ? Jan 23 15:49:53 not on fs ... just behind the master boot record Jan 23 15:49:53 the babbage can only boot from Sd or flash Jan 23 15:49:56 raw data Jan 23 15:50:04 and flash is to small to hold more than a bootloader Jan 23 15:50:09 so flash-kernel will do it Jan 23 15:50:20 right Jan 23 15:50:23 but how does redboot do it? Jan 23 15:50:35 redboot doesnt cerate /boot Jan 23 15:50:42 but where does it put the stuff? Jan 23 15:50:42 raw fis partition Jan 23 15:50:45 right Jan 23 15:50:51 which we dont have Jan 23 15:50:54 so thats the equivalent we could do as a fall back Jan 23 15:50:54 with uboot Jan 23 15:51:00 hrm Jan 23 15:51:07 we already have a /boot partition Jan 23 15:51:18 why use two Jan 23 15:51:30 we need to re-use that /boot partition anyway Jan 23 15:51:42 yes. what i mean is: make the non-fs data partition bigger and put it in there as raw data if we dont find it through the boot.scr looping Jan 23 15:51:44 but we need to teach partman and ubiquity that this is the only option Jan 23 15:52:02 does redboot support usb? Jan 23 15:52:06 no Jan 23 15:52:18 thats why we took the bootfloppy approach with it Jan 23 15:52:27 anyway Jan 23 15:52:31 uboot will just make everything a lot more complicated Jan 23 15:52:33 ogra: right. i mean we can do both Jan 23 15:52:40 would be waste sure Jan 23 15:52:46 sure, but that doesnt go into flash-kernel Jan 23 15:52:47 but if ubiquity fix is too hard Jan 23 15:53:01 well, it has to go into ubiquity Jan 23 15:53:02 flash-kernel updates the data in the fis partition, right? Jan 23 15:53:10 right Jan 23 15:53:17 and not right :) Jan 23 15:53:19 thats waht flash-kernel would do for uboot ... just put it in the no-fs data Jan 23 15:53:22 depends where you look at Jan 23 15:53:30 ok Jan 23 15:53:37 in the installer it uses the postinst from the package instead Jan 23 15:53:48 that would be the complex part Jan 23 15:54:15 additionally /boot as vfat in the installed system wont work Jan 23 15:54:22 something else ... can we grep sed etc. something in hushshell? Jan 23 15:54:28 i did some tests and update-initramfs gets very unhappy Jan 23 15:54:30 e.g. basically parsing numberes out of command output? Jan 23 15:54:37 no, no grep, no sed Jan 23 15:55:00 hush only has loops and conditionals Jan 23 15:55:07 and the test binary Jan 23 15:55:08 i didnt expect them to be there ... just hoped it had some syntax for parsing stuff Jan 23 15:55:12 ok Jan 23 15:55:19 why do we go for uboot again? Jan 23 15:55:20 ;) Jan 23 15:55:28 because david wants it :P Jan 23 15:55:34 whats his motivation? Jan 23 15:55:45 he thinks we gain anything if we can put multiple kernels into /boot Jan 23 15:55:56 i thoguth we told fsl to move there and now they moved there and we have to follow Jan 23 15:56:02 which is very hard to do (not impossible though) with a fis partition Jan 23 15:56:15 yeah i see that Jan 23 15:56:23 you can do it indeed Jan 23 15:56:34 but you wont be able to easily select Jan 23 15:56:34 i mean the gain from multiple kernels Jan 23 15:56:41 fallback Jan 23 15:56:42 but we need framebuffer in uboot to make that really useful for normal users Jan 23 15:56:58 also if you write to fis and power off the board, your bootloader setup is trashed Jan 23 15:57:00 (as you properly mentioned at some point) Jan 23 15:57:04 yeah Jan 23 15:57:22 the power-off scenario is the one that bothers david most Jan 23 15:57:47 thats true. imo you would just another set of fallback fis partitions Jan 23 15:57:51 and then you have at least 2 Jan 23 15:57:52 but effectively, the more i work on our uboot i notice drawbacks Jan 23 15:57:54 which should resolve that Jan 23 15:58:08 uboot is definately slower in loading Jan 23 15:58:09 well. uboot is buggy. that lets one think its all crap ;) Jan 23 15:58:16 likely because of the FS inbetween Jan 23 15:58:17 if it works and has usb support that would be great Jan 23 15:58:28 yeah Jan 23 15:58:33 i dont say its bad Jan 23 15:58:35 ogra: uboot is probably slower because the mmc code is busted Jan 23 15:58:41 loading the raw dat wasnt fast either Jan 23 15:58:47 but loading kernel and initramfs takes abouot twice as long Jan 23 15:58:48 i managed to load a 2M kernel with mmc read Jan 23 15:58:59 and that took about 5 seconds (optimistically measured) Jan 23 15:59:02 did you compare to redboot Jan 23 15:59:10 but since we get hangs i wonder if thats really a generic mmc bug Jan 23 15:59:16 yes, it should be really fast Jan 23 15:59:19 < 1s Jan 23 15:59:20 the time we load the kernel and initramfs the user sits in front of a black screen Jan 23 15:59:33 there is something fishy with the mmc part i am 98% sure Jan 23 15:59:34 i noticed i get impatient since using uboot Jan 23 15:59:41 i doubt that Jan 23 15:59:54 you simply have an additional layer between you and the data Jan 23 15:59:54 how can you doubt that? loading 2M with rad mmc read Jan 23 16:00:01 taking 5 seconds means that the mmc driver is busted Jan 23 16:00:11 ogra: i am not talking about fatload Jan 23 16:00:17 hmm, k Jan 23 16:00:23 with the 2009.01 uboot ? Jan 23 16:00:37 do this: type mmc read 0x90008000 3000 Jan 23 16:00:38 or something Jan 23 16:00:43 dont have help mmc at hand Jan 23 16:00:44 i'm only takling about .01 Jan 23 16:00:50 ogra: i only tried it on .08 Jan 23 16:00:54 currently i cant, just running a reinstall Jan 23 16:01:01 the blk number * 512 is the size Jan 23 16:01:16 so 2048 is 1M Jan 23 16:01:20 try to load that to some random address Jan 23 16:01:25 if its fast then its fat Jan 23 16:01:26 will do Jan 23 16:01:40 we cant use fat in the installation anyway Jan 23 16:01:49 we cant? Jan 23 16:01:54 update-initramfs gets very unhappy Jan 23 16:02:06 how? Jan 23 16:02:08 we should fix that Jan 23 16:02:09 it tries to do hardlinks in /boot for the backup files Jan 23 16:02:14 isnt fat even included in default initram? Jan 23 16:02:39 i'm talking about *update*-initramfs :) Jan 23 16:02:44 not the initramfs itself Jan 23 16:03:01 the hardlinks are temporary? Jan 23 16:03:06 i dont se them in my /boot Jan 23 16:03:17 yup Jan 23 16:03:24 its during the .bak file creation Jan 23 16:03:40 but surely has a speed reason that hardlinks are used Jan 23 16:04:19 beyond that se should have vmlinuz and initrd.img symlinks that point to the current kernel Jan 23 16:04:27 which isnt possible either on vfat Jan 23 16:04:45 so during install we should reformat the SD /boot partition to ext2 Jan 23 16:05:23 which sadly means we wipe it, which in turn adds another risk like the power-off scenario david fears so much Jan 23 16:06:14 (only during install indeed, not for every kernel update or re-rolled initramfs) Jan 23 16:06:19 so dove uboot supports ext3/4 or what? Jan 23 16:06:27 no, it uses ext2 Jan 23 16:06:40 but that can be anywhere (ide, usb, MMC) Jan 23 16:07:02 but it uses fat on the live image? Jan 23 16:07:04 and it doesnt re-use the installation media Jan 23 16:07:06 or ext2 there too? Jan 23 16:07:08 i think so Jan 23 16:07:12 fat Jan 23 16:07:22 but as i said, you dont re-use the same media Jan 23 16:07:29 its differnt Jan 23 16:07:52 we have at least 1 spare place for another partition on our live image in worst case Jan 23 16:08:01 sure Jan 23 16:08:10 but they will show up on your desktop :) Jan 23 16:08:12 so we could simulate that, just that we have a special partition on the live disk because we cant do usb etc. Jan 23 16:08:22 i will ask freescale whats up with usb support Jan 23 16:08:31 maybe its really just filling in right numbers etc Jan 23 16:08:33 far out ... last time i asked Jan 23 16:08:44 no direver was written yet Jan 23 16:08:45 hmm ok Jan 23 16:09:19 but indeed we should ask Jan 23 16:09:27 i asked at UDS time last time Jan 23 16:10:47 maybe we should stop shipping images, but rather a image builder that allows to reproduce our exact "release" images and many more thing ;) Jan 23 16:11:00 heh Jan 23 16:11:09 that was the initial approach of rootstock Jan 23 16:11:21 yes. something like that Jan 23 16:11:28 just much more professional Jan 23 16:11:38 that was the initial approach of rootstock :) Jan 23 16:11:42 well ... professional in a way that i covers the whole stack Jan 23 16:11:53 rotostock was only a quick first approach Jan 23 16:11:54 from fs production to image for target device Jan 23 16:12:04 i know Jan 23 16:12:05 right Jan 23 16:12:09 that were my plans Jan 23 16:12:23 but they always fell over because of other more important specs Jan 23 16:12:38 or because the images didnt work or... or ... Jan 23 16:12:47 always something that stopped work on it Jan 23 16:12:57 right. so we should kill the images and focus on that ;) j.k. Jan 23 16:13:03 heh Jan 23 16:13:04 but a good idea imo Jan 23 16:13:13 we might be able next release, who knows Jan 23 16:13:45 hrm, whats up with that installer Jan 23 16:13:49 doesnt run Jan 23 16:13:59 sigh Jan 23 16:14:09 i thought partman was fixed Jan 23 16:14:12 now that dove is fixed, imx51 is broken? ;) Jan 23 16:14:24 its not arch specific Jan 23 16:14:30 joking Jan 23 16:14:34 the partman bug was there in A2 already Jan 23 16:15:51 not sure if you were here still ... the images work again now ... the bootchartgui hangs turned out to be triggered by outdated .pyc files of the previous python ;) Jan 23 16:16:25 the import uno hang feels like its caused because of our libgcc3_uno.so.jaunty to me Jan 23 16:16:34 which i found out is pulled in by that import Jan 23 16:17:07 uno == ooo Jan 23 16:17:34 dove really feels like a bloody minefield to me :-P Jan 23 16:17:38 yeah, i read the backlog Jan 23 16:17:43 it is Jan 23 16:18:02 the prob is that the silicon often doesnt fulfill the actually promoted features Jan 23 16:18:12 seems if everything is working right, then all is fine Jan 23 16:18:21 but if stuff that isnt expected happens, it goes mad Jan 23 16:18:22 imx51 has its own issues though Jan 23 16:18:53 like being behind on kernel or software being not ready (see uboot) Jan 23 16:18:57 too bad ... so the fact that there were still firefox packages in hardy/dapper didnt help Jan 23 16:19:03 the source is in new :( Jan 23 16:19:07 that hit us way harder in the last reelases Jan 23 16:19:48 the luck with imx51 vs dove this time is that the silicon doesnt change anymore Jan 23 16:20:00 so for you it feels like dove is a minefield :) Jan 23 16:20:15 but it was both arches behaving like that in the last releases Jan 23 16:20:42 yeah. i can bet that Jan 23 16:21:16 i think we could have done better Jan 23 16:21:22 doko for instance need both boards Jan 23 16:21:37 if thta was the case he would have noticed that dove hangs before uploading the thumb2 switch Jan 23 16:21:43 yeah Jan 23 16:21:54 otherwise ncommander or someone else should have checked that Jan 23 16:21:59 well, but thzumb2 was advertised by the vendor Jan 23 16:21:59 ;) Jan 23 16:22:13 so whom do you trust :) Jan 23 16:24:08 grumble ... Jan 23 16:24:14 install doesnt want to start Jan 23 16:31:21 ogra: so how was lost? Jan 23 16:31:21 ;) Jan 23 16:31:30 great :) Jan 23 16:31:31 is that on kabel 1? Jan 23 16:31:45 yeah, the new episodes are on K! Jan 23 16:31:48 K1 Jan 23 16:31:49 the new season starts while we are in portland :) Jan 23 16:31:53 unfortunately final season Jan 23 16:32:01 no, thats one season ahead of us Jan 23 16:32:08 we have 5 here Jan 23 16:32:20 yes, i am tracking US ;) Jan 23 16:32:20 when we're in portland 6 starts in the US Jan 23 16:32:25 ah Jan 23 16:32:26 already have seen all ;) Jan 23 16:32:31 heh Jan 23 16:32:34 unforunately Jan 23 16:32:38 but fortunately too Jan 23 16:32:45 i'm pondering a HD+ reciever to watch them in HD Jan 23 16:32:48 the voices are really better in original Jan 23 16:32:54 as usual Jan 23 16:33:00 but i watch them with susie Jan 23 16:33:05 heh Jan 23 16:33:08 he english doesnt suffice :) Jan 23 16:33:12 i can give you the full collection ;) Jan 23 16:33:26 but not dubbed :) Jan 23 16:33:30 noo Jan 23 16:33:56 subtitles are good for her ;) Jan 23 16:34:01