**** BEGIN LOGGING AT Wed Jul 28 02:59:56 2010
Jul 28 07:15:05 <lag> Morning mythripk
Jul 28 07:17:00 <hrw> morgen
Jul 28 07:19:03 <Taalas> morgen
Jul 28 07:21:07 <lag> Mornin' all
Jul 28 07:42:03 <mythripk> morning lag
Jul 28 09:32:37 <ogra> lool, btw, my ext2 image probs were caused by the FS running out of inodes
Jul 28 09:32:57 <ogra> formatting with a higher default value should solve it
Jul 28 09:34:44 <lag> ogra: Any luck?
Jul 28 09:35:03 <ogra> archive is out of sync
Jul 28 09:35:12 <ogra> i'm waiting for it to test the fix i have
Jul 28 09:35:24 <lag> How do you mean 'out of sync'?
Jul 28 09:35:28 <lag> Out of sync with what?
Jul 28 09:35:41 <ogra>   computer-janitor: Depends: python-fstab (>= 1.2) but it is not installable
Jul 28 09:35:41 <ogra>   libmailtools-perl: Depends: libtimedate-perl but it is not installable
Jul 28 09:35:48 <lool> ogra: it's quite surprizing TBH; do you resize the fs at some point?
Jul 28 09:36:01 <lool> (I saw the chat and agree it's a number of inode problem)
Jul 28 09:36:02 <ogra> lool, nope only later on first boot
Jul 28 09:36:36 <ogra> i create an empty file with dd and format it, then loop mount it and cp -ax
Jul 28 09:36:44 <ogra> thats all i do
Jul 28 09:37:56 <ogra> it might be the way dd creates the file if you use count=0 and seek=<$imagesize>
Jul 28 09:38:24 <lool> ?
Jul 28 09:38:45 <ogra> i know that creates a file with holes (which i.e. swapon complains about)
Jul 28 13:39:10 <lool> Anybody at GUADEC?
Jul 28 13:50:54 <rsavoye> lool: nope, blew it off to get work done...
Jul 28 14:18:55 <cwillu_at_work> Check my logic:  the qemu vm issues exist in the -static builds as well;  -static + chroot works better for at least partially unknown reasons
Jul 28 14:19:45 <rsalveti> cwillu_at_work: you mean, with rootstock?
Jul 28 14:19:59 <cwillu_at_work> rsalveti, yes
Jul 28 14:20:09 <rsalveti> yep, had a hug debugging day yesterday
Jul 28 14:20:15 <cwillu_at_work> oh, really?
Jul 28 14:20:24 <rsalveti> with full vm things are slower, seg faults and hangs
Jul 28 14:20:26 <cwillu_at_work> too bad I missed it, because the -static is really hurting me right now :)
Jul 28 14:20:42 <rsalveti> with user emulation sucks with programs that request info from /proc
Jul 28 14:20:49 <rsalveti> like the stupid mono package
Jul 28 14:20:50 <cwillu_at_work> I'm getting mysterious "method http died" messages, and if I shuffle things around, the mysterious deaths move to pip
Jul 28 14:21:14 <cwillu_at_work> I've been running fine for months, and then yesterday my images just started failing
Jul 28 14:21:46 <cwillu_at_work> it's almost as if some update to a package I was installing broke things
Jul 28 14:22:23 <rsalveti> cwillu_at_work: hm, what package failed?
Jul 28 14:22:42 <cwillu_at_work> rsalveti, as far as I can tell, no package failed
Jul 28 14:22:51 <rsalveti> hm
Jul 28 14:22:58 <cwillu_at_work> it's just that some process will randomly die after aptitude finishes
Jul 28 14:23:09 <rsalveti> cwillu_at_work: oh, ok
Jul 28 14:23:09 <cwillu_at_work> (installing packages in the chroot)
Jul 28 14:23:25 <rsalveti> cwillu_at_work: what distro version are you trying to bootstrap?
Jul 28 14:23:29 <cwillu_at_work> lucid
Jul 28 14:23:41 <cwillu_at_work> debootstrap finishes fine
Jul 28 14:24:01 <rsalveti> cwillu_at_work: I'm planning to change to user mode emulation when running with root, like what you're doing, and also add native arm support
Jul 28 14:24:03 <cwillu_at_work> I could push things out to first boot, but I'd really prefer not too
Jul 28 14:24:12 <rsalveti> today, I mean
Jul 28 14:24:20 <cwillu_at_work> which, rootstock?
Jul 28 14:24:22 <rsalveti> yep, that sucks
Jul 28 14:24:30 <rsalveti> cwillu_at_work: yep
Jul 28 14:24:34 <cwillu_at_work> sec
Jul 28 14:25:18 <rsalveti> full vm doesn't work, lots of bugs, and user mode emulation works fine for most of the cases
Jul 28 14:25:46 <rsalveti> then if you still can't create the rootfs, do it on arm
Jul 28 14:27:33 <cwillu_at_work> so, yesterday, did you figure anything out re: triggering it?
Jul 28 14:29:18 <persia> How much of the vm issues can be attributed to issues with the kernel targets?  Might we just be seeing something odd there, especially with -updates things could be loosely tested for VM targets.
Jul 28 14:29:32 <cwillu_at_work> chroot isn't using the arm kernel though
Jul 28 14:29:47 <rsalveti> yep, just full vm
Jul 28 14:29:55 <rsalveti> persia: with full vm I'm getting the same behavior with different kernel
Jul 28 14:29:58 <cwillu_at_work> but that's something I haven't checked:  whether I'm using -updates as my source
Jul 28 14:30:23 <rsalveti> cwillu_at_work: first, if you use maverick's qemu, you'll get the unsupported syscall for pselect again
Jul 28 14:30:27 <rsalveti> and a huge backlog
Jul 28 14:30:42 <cwillu_at_work> I don't follow
Jul 28 14:30:48 <rsalveti> then if you install anything related with mono, it'll hang
Jul 28 14:31:06 <rsalveti> cwillu_at_work: this was fixed for lucid, but we have a regression for maverick
Jul 28 14:31:22 <cwillu_at_work> I'm not targeting maverick :p
Jul 28 14:31:30 <rsalveti> apt-get uses pselect, and this syscall is implemented at lucid
Jul 28 14:31:40 <rsalveti> happens if you're using maverick as the host :-)
Jul 28 14:31:46 <cwillu_at_work> not doing that either
Jul 28 14:32:00 <rsalveti> cwillu_at_work: also, I get a seg fault while installing humanity-icon-theme
Jul 28 14:32:17 * cwillu_at_work repeats himself:
Jul 28 14:32:27 <cwillu_at_work> same package set worked fine a week ago :)
Jul 28 14:32:51 <persia> The lucid case should be fairly different from the maverick case, but the lucid Vm kernels come from the linux source package, which doesn't see much careful testing on updates except for i386 and amd64, usually.
Jul 28 14:33:07 <cwillu_at_work> persia, we're not using the vm kernels at all though
Jul 28 14:33:14 <cwillu_at_work> persia, qemu-arm-static doesn't require one
Jul 28 14:33:25 <rsalveti> cwillu_at_work: but for me rootstock is working fine for most of the basic cases
Jul 28 14:33:39 <rsalveti> cwillu_at_work: what packages are you requesting rootstock to install?
Jul 28 14:33:46 <cwillu_at_work> rsalveti, it was for me too, up until a week ago :)
Jul 28 14:33:52 <cwillu_at_work> sec
Jul 28 14:34:05 <cwillu_at_work> I'm just going to pastebin my version
Jul 28 14:34:14 <rsalveti> ok
Jul 28 14:34:18 <persia> cwillu_at_work: Sorry then: I thought the issue was comparison of -static to the VM case.  Ignore me :)
Jul 28 14:34:36 <cwillu_at_work> persia, well, it kinda is, but I'm focusing on the parts where -static fails in the same way :)
Jul 28 14:35:52 <cwillu_at_work> http://pastebin.com/1iynGS4j
Jul 28 14:35:54 <cwillu_at_work> line 620
Jul 28 14:36:02 <cwillu_at_work> you should be able to run that locally if you remove the git calls
Jul 28 14:36:34 <rsalveti> ok, will try
Jul 28 14:36:38 <cwillu_at_work> if I don't download the packages first, aptitude will die while installing firefox (i.e., second invocation)
Jul 28 14:36:38 <rsalveti> just a sec
Jul 28 14:36:49 <cwillu_at_work> as written, it makes it through to the pip calls
Jul 28 14:37:03 <cwillu_at_work> ugh, which are commented out in this version :p
Jul 28 14:37:43 <cwillu_at_work> you might need an empty modules.d directory in the working dir
Jul 28 14:38:05 <cwillu_at_work> given locally cached downloads, it takes about 20-25 minutes to run
Jul 28 14:48:52 <cwillu_at_work> rsalveti, it seems like anything which touches the network dies after that point
Jul 28 14:49:14 <cwillu_at_work> if I remove the pip calls and the aptitude update call at the end, it finishes
Jul 28 14:55:35 <cwillu_at_work> rsalveti, actually, there's an odd thing:
Jul 28 14:56:00 <cwillu_at_work> I split up the installer into multiple files as you noticed, which are each called in the same chroot, but different invocations of it
Jul 28 14:56:24 <cwillu_at_work> (that was done to try to isolate things after they went weird yesterday)
Jul 28 15:00:28 <cwillu_at_work> ....
Jul 28 15:00:47 <cwillu_at_work> hmm, it could be the other arm chroot I had open to build packages was breaking things?
Jul 28 15:03:02 <persia> That should have no effect.  I've routinely had multiples open (via schroot) without any apparent effect.  Mind you, that sample may not be large enough to prove a negative.
Jul 28 15:03:27 <cwillu_at_work> persia, arm chroot's?
Jul 28 15:04:18 <persia> arm-on-amd64 foreign schroots
Jul 28 15:04:26 <cwillu_at_work> oaky
Jul 28 15:04:26 <persia> Err, armel-on-amd64
Jul 28 15:05:42 <cwillu_at_work> yep
Jul 28 15:06:06 <cwillu_at_work> well, I'm rerunning without the other chroot's open
Jul 28 15:06:48 <persia> I'd run that test a few times, as there's supposed to be some separation.  If you can demonstrate a convincing effect, then we clearly need to do something more advanced with LXC
Jul 28 15:07:23 <rsalveti> cwillu_at_work: sorry, will look at it now, was doing other things
Jul 28 15:07:59 <cwillu_at_work> np
Jul 28 15:08:23 <cwillu_at_work> persia, I've run 40-50 builds in the last day, and hundreds over the last few months
Jul 28 15:08:50 <cwillu_at_work> I haven't established that the extra chroot was the cause though, if that's what you're asking
Jul 28 15:08:59 <cwillu_at_work> but whatever changed is consistent
Jul 28 15:09:12 <persia> cwillu_at_work: I figured, but I doubt you have data on which of them were run with a simultaneous build chroot active (although I'd be happy to know otherwise).
Jul 28 15:09:34 <cwillu_at_work> I could figure it out (both have start and end timelogs)
Jul 28 15:09:39 <persia> Are you targeting -updates?  I really think that's the most likely source of regression.
Jul 28 15:09:59 <cwillu_at_work> I checked, I'm not
Jul 28 15:10:06 <persia> -security ?
Jul 28 15:10:08 <cwillu_at_work> take a look at the pastebin I posted
Jul 28 15:10:10 <persia> How about on the host?
Jul 28 15:10:20 <cwillu_at_work> I haven't applied updates this week yet
Jul 28 15:10:23 <cwillu_at_work> at it worked on friday
Jul 28 15:10:30 <cwillu_at_work> s/at/and/
Jul 28 15:10:41 <persia> Ugh.  phase-of-the-moon problem :(
Jul 28 15:10:46 <cwillu_at_work> :)
Jul 28 15:11:05 <cwillu_at_work> MIRROR="http://repository:3142/ports.ubuntu.com/ubuntu-ports"
Jul 28 15:11:05 <cwillu_at_work> REAL_MIRROR="http://ports.ubuntu.com/ubuntu-ports"
Jul 28 15:11:05 <cwillu_at_work> COMPONENTS="main universe"
Jul 28 15:11:27 <persia> Right.  That should be the same as it was at release.
Jul 28 15:12:13 <cwillu_at_work> the reason I mention the other chroot isn't so much that I had builds running at the same time (that was one of the earlier things I checked), but rather that I never closed the chroot itself
Jul 28 15:12:34 <cwillu_at_work> that's the test I'm running right now
Jul 28 15:12:40 <persia> That really shouldn't have an effect
Jul 28 15:13:11 <cwillu_at_work> you've said this :)
Jul 28 15:13:16 <cwillu_at_work> I'll know in ten minutes
Jul 28 15:15:30 <persia> Well, a simultaneously active chroot could have an effect if the chroot boundaries are insufficient, leading to a need to do more with LXC, but an inactive chroot is about the same whether chroot() has been called on it or not.
Jul 28 15:18:47 <cwillu_at_work> it had /proc, /sys/, dev and so forth mounted inside it
Jul 28 15:19:01 <persia> But no files open, right?
Jul 28 15:19:11 <cwillu_at_work> whatever a shell would have, yes
Jul 28 15:19:17 <cwillu_at_work> here's the thing though:
Jul 28 15:19:24 <cwillu_at_work> (have you looked at the pastebin yet? :p)
Jul 28 15:19:38 <persia> I wouldn't expect a shell to have enough open to make a difference, but maybe
Jul 28 15:19:38 <cwillu_at_work> I split the script that runs in the chroot into four pieces
Jul 28 15:19:39 <persia> yes
Jul 28 15:19:40 <cwillu_at_work> to debug this
Jul 28 15:19:50 <cwillu_at_work> each script is run in a separate chroot, sequentially
Jul 28 15:20:27 <cwillu_at_work> Now, I can rerun rootstock, and it'll get up to the same point each time (i.e., first download works, next thing to touch the network dies)
Jul 28 15:20:37 <cwillu_at_work> but... the next thing to touch the network is in a different chroot
Jul 28 15:20:45 <cwillu_at_work> and it still dies
Jul 28 15:20:51 <rsalveti> hm, werid
Jul 28 15:20:55 <cwillu_at_work> ...even though re-running rootstock doesn't
Jul 28 15:21:47 <cwillu_at_work> I'm pretty sure this will reduce to a config change that I forgot I made or something silly like that, but even so, I don't think I'm doing anything that _should_ be broken :)
Jul 28 15:21:49 <persia> Very odd.
Jul 28 15:22:24 <persia> At least it's isolated enough that it can be debugged, so once it's known, it can be made to never happen.
Jul 28 15:23:12 <cwillu_at_work> I'm secretly hoping that this is the same trigger as the grief in qemu, but I'm not sure how that plays into what you said rsalveti earlier about lucid patches
Jul 28 15:24:15 <cwillu_at_work> moments away from knowing
Jul 28 15:24:33 <cwillu_at_work> nope, that wasn't it :p
Jul 28 15:24:34 <cwillu_at_work> damn
Jul 28 15:25:12 * persia is glad the LXC integration isn't actually required, as that has looked painful the last few investigations
Jul 28 15:25:22 <cwillu_at_work> LXC?
Jul 28 15:25:42 <rsalveti> cwillu_at_work: I'm running it here, will let you know if it worked or not
Jul 28 15:25:42 <persia> http://lxc.sourceforge.net/
Jul 28 15:25:49 <cwillu_at_work> k
Jul 28 15:26:07 <cwillu_at_work> rsalveti, I don't think there's too many hardcoded dependencies on my environment
Jul 28 15:26:23 <persia> Basically, one can create even more segregation than with a regular chroot, which we haven't (quite) needed for anything yet, but I keep expecting it when people start talking about issues with multiple simultaneous chroots.
Jul 28 15:26:33 <rsalveti> cwillu_at_work: I removed most of the stuff I could easily identify
Jul 28 15:26:33 <cwillu_at_work> ah, k
Jul 28 15:26:59 <rsalveti> mirror, rsync, etc
Jul 28 15:27:40 <cwillu_at_work> the rsync might be of interest
Jul 28 15:29:48 <cwillu_at_work> (of modules.d, at least)
Jul 28 15:35:16 <cwillu_at_work> pulling up a shell inside the chroot after it starts dying
Jul 28 15:35:37 <cwillu_at_work> I'd like to confirm that it's network related activity
Jul 28 15:46:15 <rsalveti> cwillu_at_work: yep, worked fine
Jul 28 15:54:02 <cwillu_at_work> okay, bash prompt up
Jul 28 15:54:41 <cwillu_at_work> yep, definitely something weird on this box
Jul 28 15:58:10 <cwillu_at_work> ifconfig eth0 shows information, ifconfig dies with ": error fetching interface information: Device not found"
Jul 28 15:58:22 <cwillu_at_work> with no indication of which device it's looking for
Jul 28 16:00:34 <cwillu_at_work> host repository
Jul 28 16:00:34 <cwillu_at_work> qemu: Unsupported syscall: 250
Jul 28 16:00:34 <cwillu_at_work> errno2result.c:111: unable to convert errno to isc_result: 38: Function not implemented
Jul 28 16:00:34 <cwillu_at_work> socket.c:3851: epoll_create failed: Function not implemented
Jul 28 16:00:34 <cwillu_at_work> /usr/bin/host: isc_socketmgr_create: unexpected error
Jul 28 16:03:09 <persia> That's exceedingly annoying.  Is that unique to one install, or replicable?
Jul 28 16:05:08 <cwillu_at_work> not sure yet;  copying the files over to my desktop to try it there
Jul 28 16:10:44 <cwillu_at_work> strace doesn't work :D
Jul 28 16:11:23 <persia> strace-in-chroot or strace-on-qemu?
Jul 28 16:11:31 <cwillu_at_work> chroot
Jul 28 16:11:42 <lool> qemu can't emulate ptrace right now
Jul 28 16:11:45 <lool> and it would probably be hard
Jul 28 16:11:56 <cwillu_at_work> which is the package for binfmt?
Jul 28 16:11:58 <persia> Well, the issue is in qemu, if you're getting "Function not implemented".  Try stracing that.
Jul 28 16:12:06 <cwillu_at_work> binfmt-support?
Jul 28 16:12:16 <persia> that's the base, but it's pluggable.
Jul 28 16:12:45 <persia> You want to edit the binfmt entry for armel binaries to call strace, or run host strace attaching to a PID.
Jul 28 16:17:52 <cwillu_at_work> attached
Jul 28 16:19:57 <cwillu_at_work> will be a moment, hit ctrl-c in the prompt once too many times
Jul 28 17:05:43 * cwillu_at_work cries
Jul 28 17:29:48 <cwillu_at_work> persia, did you want an strace of qemu when a "host google.ca" fails?
Jul 28 17:32:22 <cwillu_at_work> persia, rsalveti, http://pastebin.com/8agEskqT
Jul 28 17:32:51 <cwillu_at_work> http://pastebin.com/6EfBPFLu is the shell output
Jul 28 17:50:54 <cwillu_at_work> there's another thing I didn't notice before:
Jul 28 17:51:13 <cwillu_at_work> my build environment is an unpacked copy of the output of my rootstock
Jul 28 17:53:00 <cwillu_at_work> which I haven't regenerated in a few weeks
Jul 28 17:53:13 <cwillu_at_work> same problems in it though
Jul 28 18:01:25 <cwillu_at_work> er, not quite
Jul 28 18:01:34 <cwillu_at_work> host dies in the same way, pip and apt seem to work fine :/
Jul 28 18:16:09 <cwillu_at_work> and, success
Jul 28 18:16:21 <cwillu_at_work> bouncing everything through a local proxy on 127.0.0.1 works
**** ENDING LOGGING AT Thu Jul 29 02:59:57 2010