**** BEGIN LOGGING AT Wed Sep 16 19:21:26 2020 Sep 16 19:21:33 JPEW: I suspect so too which is why I've not done it Sep 16 19:21:50 khem: should gcc-runtime ptesting chew disk io heavily? Sep 16 19:24:29 khem: we need to find where the machine contamination is coming from and why the selftests don't spot it Sep 16 19:25:06 RP: I think logically the patch is right, since for nativesdk we should have MACHINE var dep but I think it will be good to find the reason why, but I guess its flowing via OVERRIDES Sep 16 19:26:27 khem: why should we have a vardep? Sep 16 19:26:54 Does anyone know how ionice interacts with the extX disk journal? Sep 16 19:27:45 RP:its in overrides for nativesdk Sep 16 19:27:57 and I guess overrides value changes when MACHINE is changed Sep 16 19:28:43 so perhaps MACHINE=a bitbake nativesdk- MACHINE=b bitbake nativesdk- might reproduce it Sep 16 19:30:51 hmm, https://unix.stackexchange.com/questions/491805/ext4-jbd2-and-i-o-priority seems to echo my worries Sep 16 19:33:01 has anyone ever seen this out of bitbake: Sep 16 19:33:06 NOTE: Executing Tasks Sep 16 19:33:06 [Errno 11] write could not complete without blocking Sep 16 19:36:50 zeddii: interesting. no. Sep 16 19:37:37 aaaand now it cleared. was only giving me that for about 10 minutes. Sep 16 19:37:45 JPEW: I'm guessing that the writes to the disk all end up in the journal thread for the disk and lose their io priority and gain the io priority of the jbd process :/ Sep 16 19:38:13 the only hit I got on the error message was from some logging discussions. Sep 16 19:38:17 ah, its back. whee. Sep 16 19:38:44 zeddii: what is it building? Sep 16 19:39:02 sounds like the pipes to the workers/cooker are being overloaded with log data Sep 16 19:39:06 core-image-minimal Sep 16 19:39:30 hmm, we do that often enoug Sep 16 19:39:50 it's on a builder that I've been using all day. trying to get a core image that can boot and runtime test Xen. Sep 16 19:39:55 https://pastebin.com/ZP8ranmX Sep 16 19:39:58 and now, that. Sep 16 19:40:00 very odd. Sep 16 19:40:32 zeddii: its odd its not even up to the building part Sep 16 19:40:41 lots of disk, fully idle machine. Sep 16 19:40:49 I know! and it just popped in out of nowhere. Sep 16 19:40:56 but that's how my day has been. Sep 16 19:41:48 hah. failed once more. ran it again. and it worked. something is definitely contending on this box. I'll see what I can figure out. Sep 16 19:42:26 it is a multiconfig build, but otherwise, pretty simple. Sep 16 19:53:39 halstead: Could we try switching the /home partition to writeback journal only? Sep 16 19:55:45 RP yes. I do that for distros that have it as an install option. I'll set it up for all the workers. Sep 16 19:57:26 halstead: Ah! this may explain why we're seeing the problem particularly on the fedora workers Sep 16 19:58:10 halstead: if we could switch those over that'd be great, it might well avoid the problems we're seeing Sep 16 19:58:40 I think the SUSE installer has the option. I'll get the other ones. Sep 16 20:01:14 halstead: thanks, this could explain a few things Sep 16 20:17:25 sgw: swapped out openssh for dropbear in my qemumips sato-sdk image and serial login takes 20s instead of 45s. Why is serial login depending on ssh daemon? Sep 16 20:35:27 strace is your friend? Sep 16 20:36:20 some kind of network lookup it would seem. Sep 16 20:36:52 RP is it a dependency on "login", moved from one version to another? Sep 16 20:38:33 RP: I kicked off a new full build on the primary server and waited until after it had finished sqlite3-native, and then built that target on the other machine. It pulled everything from the sstate-cache, so using bitbake-hashserv is working as expected. Sep 16 20:39:13 I dunno if this is related, but back in the day, I changed the hostname (well, really net-cat'd the same image to another turd) and changed /etc/hostname but didn't change /etc/hosts for 127.x entries Sep 16 20:39:59 noted that serial logins took longer - never went past tracking it down to /etc/hosts entries for 127.x Sep 16 20:40:24 sounds rather similar. Sep 16 20:40:56 or, completely unrelated! Sep 16 20:41:58 paulg: I'm just trying to think of how you strace the first boot serial login :) Sep 16 20:42:15 fullstop: that is good news :) Sep 16 20:42:52 paulg: that does sound kind of similar... **** BEGIN LOGGING AT Wed Sep 16 21:07:13 2020 Sep 16 21:47:21 paulg: hostname and hosts appear to match and have the right mappings but it could well be something silly like that Sep 16 21:50:24 RP, yeah - it was a long time ago and I wish I could help with more context, but brain-flush has claimed any further details. Sep 16 21:50:56 like a lot of things in life, I just recall getting savaged by it. Sep 16 21:51:19 RP: can it be because the rand entropy is not gathered quick enough? I saw some delays in starts because of this Sep 16 21:52:14 do you have haveged package installed? Sep 16 21:56:14 zandrey: virtio rng passthrough is there and enabled Sep 16 21:56:39 question is how/why would a serial getty depend on the ssh daemon Sep 16 21:56:53 RP: okay. Sep 16 21:57:34 i just saw that getty gets delayed in the startup by ssh server, and that one get delayed because the entropy was not enough Sep 16 21:58:00 and i do recall i had similar results with openssh - around 1 min to console Sep 16 21:59:38 this was however almost 2 years ago, and i do not recall much details now on how i did traced it down and brought haveged to address the entropy, effectively speeding-up the openssh to start Sep 16 22:00:22 i guess if the randr is ok - there is not much for me to suggest further Sep 16 22:04:45 zandrey: its a good suggestion, thanks. I'm just not sure it is that Sep 16 22:05:30 kanavin_home, I've fixed the split DNS config and added a host override for smtp1. Although we should still remove all references to smtp1 **** BEGIN LOGGING AT Wed Sep 16 22:19:01 2020 Sep 16 22:23:17 * RP is getting totally lost. I know I need to fix something, I just can't remember what :/ Sep 16 22:27:17 halstead: thanks, I will send a patch to autobuilder that replaces the reference Sep 16 22:27:43 Thanks for getting this fixed kanavin_home. Sep 16 22:30:54 sakoman: I've put a fix into helper for the python3 result tool error, its as yet untested though Sep 16 22:32:15 RP: just a matter of time till a worker with an old distro is used! **** BEGIN LOGGING AT Wed Sep 16 22:37:23 2020 **** BEGIN LOGGING AT Wed Sep 16 22:40:18 2020 Sep 16 22:52:53 sakoman: thanks, sounds good **** BEGIN LOGGING AT Thu Sep 17 00:06:32 2020 **** BEGIN LOGGING AT Thu Sep 17 01:53:56 2020 **** BEGIN LOGGING AT Thu Sep 17 02:09:47 2020 **** BEGIN LOGGING AT Thu Sep 17 02:19:27 2020 **** BEGIN LOGGING AT Thu Sep 17 02:32:58 2020 **** ENDING LOGGING AT Thu Sep 17 02:59:57 2020