**** BEGIN LOGGING AT Mon Oct 26 02:59:59 2015 Oct 26 08:09:49 morning all Oct 26 08:43:08 good morning Oct 26 08:43:23 morning Oct 26 08:43:35 good morning Oct 26 09:35:38 mornin Oct 26 09:56:13 hmm, I'm being plagued by a problem in do_rootfs where exactly one and the same package produces an error "opkg_install_pkg: Package xyz md5sum mismatch. Either the opkg or the package index are corrupt. Try 'opkg update'." Oct 26 09:56:17 it's a multimachine build Oct 26 09:56:22 and this package is an allarch one Oct 26 09:56:33 when I -c cleanall stuff around and retry a couple of times it then works Oct 26 09:56:53 so far I failed to find the source of the problem, the recipe itself looks not different from many others that I have, yet exactly this one is failing Oct 26 09:57:01 and hints on how to debug this? Oct 26 09:58:11 this keeps on coming up Oct 26 09:58:28 I don't think anyone's come up with a reproducer for it though Oct 26 09:58:38 the fact that it's an allarch recipe is suspicious though Oct 26 09:58:38 aha, so its somewhat a known issue? Oct 26 09:58:52 I get it pretty consistently Oct 26 09:58:53 well, it's been reported by a few different people Oct 26 09:59:08 so if you have ideas on what to look for I could try to debug Oct 26 09:59:50 you'd first want to establish if the recipe is rebuilding when you change machines, if it is then use bitbake-diffsigs to compare between the two executions Oct 26 10:00:17 well, in my case the recipe had no changes Oct 26 10:00:27 and I have ab out 5 recipes that look exactly the same and they work just fine Oct 26 10:00:33 its a recipe of some installation manuals Oct 26 10:01:03 so it keeps failing from time to time while the recipe itself did not undergo any changes Oct 26 10:01:30 which for me is weird in itself, I mean.. why would it complain about something that has not changed? Oct 26 10:02:12 let me google up on the diffsigs thing Oct 26 10:04:47 bluelightning: well, now I am in a state where doing my build would only run do_rootfs, everything else is through; if I cleanall the package in question it will probably work again Oct 26 10:05:13 what task name would I use for the diffsig thing? Oct 26 10:05:40 Jin^eLD: you're sure that if you change machine you don't get any setscene tasks run? pipe through cat to make it more obvious Oct 26 10:06:06 well I have a log of the whole build Oct 26 10:06:09 so I can look stuff up Oct 26 10:06:12 its a nightly-autobuilder Oct 26 10:06:54 it builds every night, same distro, machine A, B Oct 26 10:06:58 machine A failed tonight Oct 26 10:06:59 ok Oct 26 10:10:15 Running setscene task 1658 of 1758 ( Oct 26 10:10:21 so it does run setscene for the failed package Oct 26 10:10:31 and writes a new ipk Oct 26 10:10:38 although there were no changes Oct 26 10:10:57 the same happens with a number of other packages that also did not have any changes Oct 26 10:12:18 what does setscene do btw? Oct 26 10:12:30 it's restoring the output from sstate for the task Oct 26 10:12:46 ah found it in the bitbake manual, will read up on it Oct 26 10:12:59 but.. what now? we know its running setscene, I understand it was not supposed to do that? Oct 26 10:13:20 or actually it was ok, from what I understand in the manual Oct 26 10:13:24 it handles prebuilt artifacts Oct 26 10:13:37 in this context it just means what's currently in the sysroot doesn't match with what the metadata says it should be, so it knows it needs to do something - if it successfully runs the setscene task that means it can restore it from sstate; if it hadn't been able to do that it would have rebuilt it instead Oct 26 10:13:43 does this allarch recipe have any dependencies? Oct 26 10:15:12 it only has an RDEPENDS on something in the rootfs but no DEPENDS Oct 26 10:15:16 https://git.digitalstrom.org/dss-oe/dss-oe/blob/master/yocto/dS/meta-digitalstrom-devel/recipes-digitalstrom/dss11-help/dss11-installation-manual-tr-tr_0.9.bb Oct 26 10:15:19 that's the reciep btw Oct 26 10:15:24 really a simple one Oct 26 10:15:47 and I have about 5 which are more or less the same for the other languages, but only this one fails consistently Oct 26 10:16:32 now I notice I should fix the description :) but that's not the source of the error anyway Oct 26 10:17:23 btw I can see that it does write ipk's for all the other manual recipes as well, which also have not changed Oct 26 10:19:48 here's also a log with the grep for this package to see what it is going through https://pastebin.mozilla.org/8850409 Oct 26 10:20:33 btw do we have a ticket for it? it may make sense to attach all the stuff there too Oct 26 10:35:10 bluelightning: could not find a ticket, I'll submit one and attach my logs Oct 26 10:36:22 when you say ticket, where do you mean? Oct 26 10:36:45 I mean an issue on bugzilla Oct 26 10:36:48 sorry :) Oct 26 10:36:53 whose though? Oct 26 10:37:08 used to call it tickets (working with redmine a lot) Oct 26 10:37:35 bluelightning: what do you mean "whose"? which component? Oct 26 10:38:25 i was referring to bugzilla.yoctoproject.org Oct 26 10:38:31 right, that was my question Oct 26 10:38:37 bitbake? Oct 26 10:38:46 well, we see that packages that had no changes get rebuilt, right? Oct 26 10:38:56 which component is responsible for that? imho bitbake? Oct 26 10:39:05 help me out here :) Oct 26 10:39:09 it's a little more complicated than that Oct 26 10:39:17 file it against OE-Core Oct 26 10:39:23 hm ok Oct 26 10:39:34 but the question will be have you run bitbake-diffsigs against the siginfo file for the tasks that get re-executed Oct 26 10:40:15 I was asking what parameters do I give the diffsig thingie? it wants recipe (clear) and "taskname" - what do I use there? Oct 26 10:40:28 aha.. "against the sigfinfo file.." Oct 26 10:40:29 ok, so let's wind back a bit Oct 26 10:40:36 ok.. Oct 26 10:40:53 you want to find out which task is the first task that's re-running... that is where there is some difference that we want to compare Oct 26 10:41:10 ok... Oct 26 10:42:17 and I do that how exactly? Oct 26 10:42:19 one way to do that - go into your build directory, bitbake -c cleansstate the recipe, build it, change to the other machine, cleansstate it again, then build it again piping through cat and noting the first task that re-runs Oct 26 10:42:34 ok let me try that Oct 26 10:42:43 actually hrm, that's not going to work Oct 26 10:42:49 mhm Oct 26 10:43:06 do both cleansstates first, so that means two machine changes in the process Oct 26 10:43:52 so cleansstate the recipe, change machine, cleansstate again, then build it, then change machine, build it again and note the first task that executes Oct 26 10:47:16 https://pastebin.mozilla.org/8850411 Oct 26 10:47:28 so thats the log of the last step, "change machine, build it again" Oct 26 10:48:07 do_populate_sysroot_setscene seems to be the first one? Oct 26 10:49:28 do_package_write_ipk is the first real task Oct 26 10:49:50 oh ok... but... why is it writing an ipk for an allarch package that was already built? Oct 26 10:50:11 I don't know, that's what we're attempting to find out Oct 26 10:50:16 indeed :) Oct 26 10:50:36 FWIW I do wonder if it really makes sense for a package like this to have a dependency on lighttpd Oct 26 10:51:06 bluelightning: its an RDEPENDS... and since it supplies an html page the idea was to make sure the webserver that's configured to serve it is also around... Oct 26 10:51:21 but I could also drop it in this case because the webserver is anyway included elsewhere too Oct 26 10:51:24 don't you have other means to ensure that's in the image? Oct 26 10:51:26 right Oct 26 10:51:54 but theoretically speaking there shouldn't be anything wrong with having the RDEPENDS there? Oct 26 10:52:12 thinking about it that may be causing this; it may be triggering the allarch recipe to actually be architecture-specific Oct 26 10:52:22 worst case its redundant but should not hurt, no? Oct 26 10:52:30 aha hmm Oct 26 10:52:41 take it out and see if it changes the behaviour Oct 26 10:52:44 I suspect it will Oct 26 10:52:58 ok so cleansstates again, take it out, rebuild? Oct 26 10:55:09 ok lets see.. Oct 26 10:55:59 thats the new log https://pastebin.mozilla.org/8850412 Oct 26 10:56:38 indeed a lot shorter Oct 26 10:57:01 the write_ipk is there but the qa is gone? Oct 26 10:57:19 I think that's probably ok... since the sysroot will actually be different for the different machine (since the sysroot is per-machine), it's correct that it'd need to populate it for the second machine, but it's doing it entirely from sstate Oct 26 10:57:37 no real tasks executed Oct 26 10:58:12 so, I think we've fixed the underlying cause, but there are a couple of issues here Oct 26 10:58:45 1) you were sometimes getting an error from opkg triggered by this, that should never have happened Oct 26 10:59:15 2) it seems like there ought to be some kind of warning in this case; we've talked about it before though and I'm not sure there's clear agreement on how to handle that Oct 26 10:59:48 is it actually OK that RDEPENDS triggers such a rebuild or is it a bug? Oct 26 11:00:22 especially where neither the recipe itsself nor the one listed in RDEPENDS have changed, stayed the same for both machines Oct 26 11:03:36 I'm not sure but I feel like either it shouldn't or there should be a warning that it has Oct 26 11:04:11 you say that lighttpd hasn't changed, I think you'll find it has Oct 26 11:04:35 what is the architecture of the two machines? Oct 26 11:05:17 arm5 and imx6 Oct 26 11:05:29 right, so not the same Oct 26 11:05:35 yep, not the same Oct 26 11:05:40 so lighttpd has been built differently for each one Oct 26 11:05:44 that's the trigger Oct 26 11:05:44 yes Oct 26 11:05:55 but why is it a trigger for RDEPENDS? Oct 26 11:06:08 isn't that only interesting for a runtime installation? Oct 26 11:07:37 not sure, but I believe the behaviour is deliberate, whether it's correct or desirable in all situations is a different question Oct 26 11:08:51 so what do you suggest now? I mean, sure, I can work around by removing the RDEPENDS and that's what I will do in the short term, and I guess I would not mind the rebuild if not the opkg error Oct 26 11:09:15 but hearing that I'm not the first one who had these problems it'd be cool to find a more permanent solution Oct 26 11:09:33 well the opkg issue definitely needs fixing Oct 26 11:09:43 a solid reproducer would help a lot there Oct 26 11:10:25 I think we just got rid of the solid reproducer :) Oct 26 11:10:31 for the other one, we need to have a wider discussion, figure out the desired behaviour, and document it Oct 26 11:10:35 but I can activate it again of course Oct 26 11:10:45 naturally, you'd need to ;) Oct 26 11:11:17 so should I file a bug or just drop a mail with our findings to the ml? Oct 26 11:11:44 we need a bug for the opkg issue Oct 26 11:12:08 ok so I'll try to summarize our findings there Oct 26 11:12:24 well, let's not conflate the two things, I don't want them getting tangled up Oct 26 11:12:31 even if they are related Oct 26 11:12:51 file a bug for the opkg issue and start a thread on the mailing list for the other one, how does that sound? Oct 26 11:13:34 brb Oct 26 11:13:57 ok, deal Oct 26 11:14:03 will do that right after lunch Oct 26 11:14:05 brb Oct 26 11:39:51 nerdboy: hello? Oct 26 11:42:06 bluelightning: btw now that I removed the RDEPENDS on that package I simply started to get the opkg error eslewhere at a different place, I guess on something that RDEPENDS on python.. which is not in the image by default so removing the RDEPENDS there would not be that nice... Oct 26 12:06:11 Jin^eLD: I guess we need to properly solve the opkg issue then Oct 26 12:07:11 ok let me first enter the issues/post to the ml as we discussed and maybe if you have a little time we could have another look since it seems to be nicely reproducible on my setup? Oct 26 12:07:37 unless you prefer to have another look before me submitting an issue :> Oct 26 12:09:09 Jin^eLD: I guess we can do, what's the new situation? Oct 26 12:11:31 bluelightning: well, not really "new", its again an allarch recipe that RDEPENDS on some stuff Oct 26 12:11:49 and produces the same md5sum mismatch error message with opkg upon do_rootfs Oct 26 12:12:09 so the problem only shifted to another recipe after I removed the RDEPENDS in the previous one Oct 26 12:16:07 well the only real workaround available is to make the recipe not allarch Oct 26 12:16:13 at the moment anyway Oct 26 12:16:55 switching from allarch to machine specific and back also poses issues at times Oct 26 12:17:23 I'd rather look at the opkg problem while I can reproduce it, but I guess I need a bit of guidance since I am not that well familiar with the iternals Oct 26 12:18:15 I'm guessing it's not machine-specific, but arch-specific in this case Oct 26 12:18:23 but yes Oct 26 12:20:47 I'm poking around in the workdir but not yet sure what to look for Oct 26 12:21:19 probably I should compare the packages.gz md5sums with the actual package md5sum Oct 26 12:21:33 usually this error happens (at least on live systems) when the package-index is out of date Oct 26 12:22:04 maybe for some reason the package indexing code thinks the package was not updated when it actually was Oct 26 12:22:14 and thus it doesn't update the md5sum Oct 26 12:22:19 (a guess ...) Oct 26 12:22:27 not a bad one actually, would make sense Oct 26 12:22:54 could you point me to the code that does the indexing when the rootfs is done? Oct 26 12:23:58 well it's before building the rootfs, not afterwards, but hang on Oct 26 12:24:39 yes I did not express myself well enough, I meant "when the rootfs is being generated" Oct 26 12:25:09 packages.gz indeed has a different md5sum, not the one of the package, so index not up to date Oct 26 12:25:32 Jin^eLD: see OpkgIndexer in meta/lib/oeqa/utils/qemurunner.py Oct 26 12:27:39 OpkgIndexer? grep does not return anything on that in meta/lib/oeqa/utils/ Oct 26 12:28:05 and I do not find anything opkg related in qemurunner.py ? Oct 26 12:28:08 am I missing something? Oct 26 12:28:31 ah sorry I pasted completely the wrong path Oct 26 12:28:37 meta/lib/oe/package_manager.py: Oct 26 12:28:42 :) Oct 26 12:32:02 I do not see who calls that in do_rootfs Oct 26 12:32:34 generate_index_files() is only called from package-index.bb Oct 26 12:32:43 I do not yet see how that plays together with do_rootfs Oct 26 12:34:06 I do not see it calling package-index either Oct 26 12:34:16 but I guess I do not fully understand the whole process yet Oct 26 12:34:58 do_rootfs() is a python function that calls create_manifest(d), create_rootfs(d) and create_image(d) Oct 26 12:35:21 but none of those seems to be calling generate_index_files()? Oct 26 12:35:21 Jin^eLD: there are calls to write_index() in meta/lib/oe/rootfs.py, that's on the package manager class, which calls through to the indexer Oct 26 12:35:43 ok found it Oct 26 12:36:58 I thought it would go via generate_index_files but write_index is called directly Oct 26 12:50:07 bluelightning: to be honest I'm a bit lost there Oct 26 12:50:33 does it create an index command for each file that needs to be indexed? Oct 26 12:51:23 hm no, it indexes the whole directory Oct 26 12:51:26 right Oct 26 12:51:34 if there's a problem it's likely in opkg-make-index Oct 26 12:52:15 it creates a list for each arch with a directory per arch, so its indexing directories, not single packages Oct 26 12:52:28 now what I could check if it tries to index the allarch at all Oct 26 12:52:30 or if it skips it Oct 26 12:53:14 bb.note expects strings and print is not visible, is there a utility function to print python objects? Oct 26 12:53:21 opkg-make-index comes from the opkg-utils recipe (its -native variant, in this instance) Oct 26 12:53:46 just do: "%s" % obj Oct 26 12:53:47 I first want to check if it attempts to update the index for the allarch directory at all Oct 26 12:53:56 it tole me its not a string Oct 26 12:54:00 *told Oct 26 12:54:03 str() it then ;) Oct 26 12:54:05 I wanted to print the list Oct 26 12:54:09 ah, did not htink of it :) Oct 26 12:54:26 that may not help though Oct 26 12:54:49 you might need to actually poke into the attributes of the object and print those Oct 26 12:57:24 allarch is there though Oct 26 12:57:25 '/oe/dss-oe/poky-devel-build/build/sysroots/x86_64-linux/usr/bin/opkg-make-index -r /oe/dss-oe/poky-devel-build/build/deploy/ipk/all/Packages -p /oe/dss-oe/poky-devel-build/build/deploy/ipk/all/Packages -m /oe/dss-oe/poky-devel-build/build/deploy/ipk/all' Oct 26 12:57:57 right, it's more likely that opkg-make-index itself is skipping the package Oct 26 12:58:38 ok I'll focus on that one Oct 26 13:09:59 hmm somehow the printouts made in opkg-make-index do not make it to my console Oct 26 13:16:14 worked around it, still debugging Oct 26 13:18:16 and well, actually I can call it directly :) Oct 26 13:34:24 bluelightning: ok I think I have something Oct 26 13:34:25 if filename in pkgsStamps and int(fnameStat.st_mtime) == pkgsStamps[filename]: Oct 26 13:34:48 this is true for my package, so it does not get re-calculated Oct 26 13:34:52 and keeps the old md5 Oct 26 13:35:05 hmm, so why doesn't the mtime change? Oct 26 13:35:13 if I skip this if then the packages index is correct, but then I force everything to get recalculated Oct 26 13:35:28 or maybe the mtime actually gets updated in the dict before that code runs? Oct 26 13:35:37 good question Oct 26 13:35:56 hmmm Oct 26 13:35:59 but now I ran it outside OE Oct 26 13:36:10 because I otherwise had troubles seeing the error messages Oct 26 13:36:27 however it kept the wrong index until I hacked out the if Oct 26 13:37:17 ok let me re-reproduce it and then add some more debugigng to the pkgsStamps part Oct 26 13:52:26 bluelightning: damn, now I "lost" it, it did build :( Oct 26 14:26:36 bluelightning: I "lost" the reproducibility for now, I guess I will continued debugging when it fails next time, but at least I now know where to continue Oct 26 14:26:40 thanks a lot for the help Oct 26 14:27:15 I'll drop a mail regarding RDEPENDS that triggers a rebuild Oct 26 14:27:20 not sure about the issue in bugzilla though Oct 26 14:30:39 if you could file the bug anyway, cc alejandro castillo and note where you got to with the debugging Oct 26 14:31:41 ok Oct 26 14:31:43 vs oe-core? Oct 26 14:31:50 or what part is the opkg-utils? Oct 26 14:32:05 git://git.yoctoproject.org/opkg-utils Oct 26 14:33:15 yes, devtolls/toolchain Oct 26 14:33:16 got it Oct 26 14:52:32 bluelightning: #8578 Oct 26 17:18:04 e Oct 26 17:52:12 buying a vowel? Oct 26 19:05:22 hi, anyone know how to find the machine name on the target? (runtime). I have a few applications that need to behave a bit different, depending on the machine they are running on, and wanted to simply add MACHINE To the default env, by modifying /etc/profile Oct 26 19:26:48 mvader: you can create a bbappend to the "base-files" recipe and in your bbappend use the ${MACHINE} variable in a case statement (or something similar). Oct 26 19:27:00 hi, I'm running into trouble with a patch that is supposed to create a file with executable permissions, that will be run as part of the boot process (zynq boot.bin generation, on top of mainline u-boot) Oct 26 19:27:13 http://pastebin.com/qrngjKcU Oct 26 19:27:35 I double checked the patch (create mode is 755) Oct 26 19:27:48 am I doing something fundamentally wrong here? Oct 26 19:28:39 stephano: thanks, I'll give that a try Oct 26 19:29:42 fischerm, you have a do*_append to run ./tools/zynq-boot-bin.py Oct 26 19:30:06 you could "python ./tools/zynq-boot-bin.py" Oct 26 19:30:15 Crofton: so the thing is, if the patches are in tree Oct 26 19:30:20 Crofton: it works as expected Oct 26 19:30:45 Crofton: if they're part of the SRC_URI ones they end up non executable Oct 26 19:31:35 oh crap, I forget zediii only hangs out in #yocto Oct 26 19:31:40 * Crofton curses Oct 26 22:03:56 F me Oct 26 22:04:07 llvm changed from .gz to .xz Oct 26 22:04:17 making SRC-URI in include file hard Oct 26 22:08:52 :) Oct 27 00:20:58 cbrake: question on your survey... Oct 27 00:22:40 "How do you develop custom applications? [ ] application-SDK" <= is this a vendor/upstream thing or locally built OE sdk build? **** ENDING LOGGING AT Tue Oct 27 02:59:58 2015