**** BEGIN LOGGING AT Wed Jan 15 02:59:57 2020 Jan 15 19:46:16 Suppose I've got a recipe that has a native and target flavor, pulling from a repo with AUTOREV. Right now there is no guarantee that during parsing the two recipes will end up with the same SRCREV. This is because the srcrev cache is keyed based on repo + pn. One workaround would be to change the srcrev cache to be keyed just on repo (perhaps Jan 15 19:46:17 controlled by a variable, BB_SRCREV_CACHE_GRANULARITY). Another option is to do like poky/meta/recipes-devtools/gcc/gcc-source.inc and have a separate recipe perform the fetch, which the actual recipes just DEPEND on. What do you all think? Jan 15 20:12:27 RP: any thoughts? Jan 15 20:32:10 laplante: the trouble is you could intentionally set different revisions depending on PN Jan 15 20:33:09 laplante: you probably need to set the revisions you want ahead of time, you're into the realms of very specific usage scenarios :( Jan 15 20:34:08 RP: but in the case of different revisions based on PN, the srcrev cache won't be consulted. For example: SRCREV ?= "${AUTOREV}" Jan 15 20:34:16 SRCREV_class-native = "githash" Jan 15 20:34:41 And yes I realize this is niche :). Jan 15 20:36:00 laplante: you can't know they don't change the branch or something :/ Jan 15 20:36:16 laplante: I think you swap one set of problems for another Jan 15 20:36:40 laplante: perhaps it should cache by url? Jan 15 20:38:28 RP: at least for git, the branch is also part of the key Jan 15 20:39:07 laplante: what exactly is it keying off? Jan 15 20:39:23 RP: give me a sec, will pop open the cache to dump an example Jan 15 20:41:29 RP: so if you look in fetch2/__init__.py, generate_revision_key is what is responsible for the key. It suffixes the result of calling "self._revision_key" with the "-${PN}" Jan 15 20:41:54 RP: for git, _revision_key is defined as: "git:" + ud.host + ud.path.replace('/', '.') + ud.unresolvedrev[name] Jan 15 20:42:19 RP: so basically, the host, path, and branch or tag name (I guess) Jan 15 20:43:15 not brilliant :/ Jan 15 20:43:33 RP: lol which part? Jan 15 20:43:37 I hate poking at this code as we'll end up breaking other things :( Jan 15 20:44:04 I'm just thinking of all the other interesting parameters people keep adding to the git fetcher Jan 15 20:45:53 laplante: the fetcher is called fetch2 as fetch became such a nightmare. We still have pieces of it that leaked through, its better but not brilliant Jan 15 20:46:43 we probably could get away with keying on the other data Jan 15 20:47:01 I guess to solve your problem we have to try it Jan 15 20:47:20 I know who will get the bugs to chase down :( Jan 15 20:47:46 RP: So I have actually already done it locally and it does work :). Jan 15 20:48:15 I already chased down one interesting bug in persist_data.py, where a constraint violation was violated on the key. Jan 15 20:49:12 laplante: I spend most of my time chasing down weird bugs that only appear on the testing infrsatructure intermittently :( Jan 15 20:49:29 RP: I've noticed, that's why I hesitated to bother you :/ Jan 15 20:50:15 laplante: I agree with your analysis and that its probably something to fix. I just worry about the impact of changes like this Jan 15 20:50:19 RP: Re persist_data.py: There's a race in SQLTable::__setitem__, since the SELECT is executed under a read transaction, and is not upgraded to a write transaction until the INSERT or UPDATE actually runs. This was never a problem before removing -${PN} from the key, since conflicting keys were actually impossible. Jan 15 20:51:00 laplante: Well spotted, we should fix that too Jan 15 20:51:09 RP: Yep, will submit a patch Jan 15 20:51:19 laplante: I know JPEW found a load of other problems in persist_data, its horribly broken Jan 15 20:52:06 https://bugzilla.yoctoproject.org/show_bug.cgi?id=13181 Jan 15 20:52:32 RP: Perhaps we could gate this change on a new var as I suggested, e.g. BB_SRCREV_CACHE_GRANULARITY? Jan 15 20:52:37 JPEW: :O that's a doozy Jan 15 20:53:02 laplante, Yep. That was a *fun* debugging session ;) Jan 15 20:53:12 laplante: Lets not do that, just creates more of a mess Jan 15 20:53:51 RP: OK. I'm pretty confident (famous last words) that this fix is correct for the git fetcher. Not sure about the others that implement unresolved refs. Jan 15 20:53:58 e.g. SVN Jan 15 20:54:18 laplante: (the reason is a config option then doubles the number of tests needed) Jan 15 20:54:48 Gotcha Jan 15 20:54:52 strategy for svn is becoming "point and laugh" :/ Jan 15 20:55:25 To be fair, that's my strategy for usage outside BitBake/Yocto as well Jan 15 20:57:33 RP: BTW I doubt you've had time to go through the recent email I sent (regarding building tagged releases) but I hope to post a prototype tinfoil-based script soon that is capable of generating the SRCREV_pn- overrides. Jan 15 20:58:07 RP: My original approach of shoehorning the necessary support into fetch2 has been abandoned (and it sounds like I'm better off not playing with it too much anyway) Jan 15 20:58:52 laplante: I think I would much prefer to see things using its APIs, even event driven as handlers or through tinfoil Jan 15 20:59:04 it simply won't scale to everyone;s special way of handing sources Jan 15 20:59:35 laplante: I did read through it breifly and your journey through different appraoches Jan 15 21:00:12 RP: Hope it was an entertaining read - I don't think I've ever rewritten a piece of code as many times as I did there :) Jan 15 21:01:09 laplante: I feel your pain, I'm open to ways of fetcher help but I think we need new approaches Jan 15 21:01:19 if we could rip autorev out the fetcher that would rock Jan 15 21:01:42 RP: where would it move to? Jan 15 21:02:47 laplante: eventhandler? Jan 15 21:02:58 laplante: not sure, thinking out loud Jan 15 21:03:13 RP: hmm interesting. Like a RecipePreFinalise or something? Jan 15 21:03:37 laplante: or similar. Could have its own event if needed Jan 15 21:05:21 RP: I suppose it would be neat to add a stage after parsing that is actually "resolve upstream references". Jan 15 21:05:27 RP: Since "parsing" right now basically covers both Jan 15 21:06:10 Would be convienent to decouple them Jan 15 21:07:39 laplante: Its not quite as simple as "after parsing" since some of the code you're parsing depends on the revision/version Jan 15 21:07:52 which is what makes autorev such fun Jan 15 21:07:54 Ah true. Jan 15 21:09:29 laplante: not trying to put you off, just being clear about the challenge Jan 15 21:09:40 RP: no I get it :) Jan 15 21:10:44 RP: (the point you're making, not the full consequences/minutiae of the challenge) Jan 15 21:11:54 laplante: I've never liked how that code works... Jan 15 21:12:17 or uninative installation process which I'm currently reminded of :/ Jan 15 21:14:48 RP: what specifically about uninative do you not like? Jan 15 21:15:55 laplante: its hacked onto the front of the system without a dedicated event or a dedicated point in the task graph Jan 15 21:16:19 plan was to prove it and add something proper in due course Jan 15 21:16:27 worked too well Jan 15 21:17:02 laplante: to see what I mean, watch the NATIVELSBTRING in a first build in a clean directory vs the second Jan 15 21:21:55 RP: Gotcha. Jan 15 21:22:44 RP: The one thing I've noticed about it is that since it operates outside the task graph, it can't be setscened, so never makes it to a sstate mirror :/ Jan 15 21:23:25 laplante: setscene makes no sense there, its already a binary Jan 15 21:23:39 sstate mirror is a different story though Jan 15 21:24:20 RP: Ah yes, messed up my terminology. Jan 15 21:24:21 FYI I backported the recent sstate-cache changes to zeus and they are fixing problems we had Jan 15 21:24:46 fray: the length ones? Jan 15 21:24:53 * RP made a lot of changes recently Jan 15 21:25:08 yes, that series of 4.. Jan 15 21:26:16 there were 2 others from master I pulled in as well.. (I brought in anything that modified sstate.bbclass) Jan 15 21:48:08 RP: do you know why fetch2 allows more than one name for a SRC_URI entry? e.g. SRC_URI = "git://whatever;name=name1,name2" ? Jan 15 21:49:02 (^ last question, I promise) Jan 15 21:51:54 laplante: no, that seems strange Jan 15 21:52:19 fray: so you're saying we should backport them officially? Jan 15 21:52:45 for me I needed them.. but changing the sstate paths means more of a decision from someone other then me Jan 15 21:52:48 RP: just as you wrote that I think I found the "answer". I think it's becuase multiple branches can be specified, and you need one name per branch. Which raises why would you need to clone more than one branch? Jan 15 21:52:56 (but yes, I've been using them now for about 18 hours and they're working great) Jan 15 21:53:07 laplante: the kernel used to do that Jan 15 21:53:08 this is on top of zeus w/ out own internal layers.. Jan 15 21:53:47 (I can get you the list of what I merged if you care.. but easy enough to figure out) Jan 15 21:53:55 laplante: it needed a bare repo where we ensured X branches were present with revisions ... Jan 15 21:54:22 fray: I can guess, really just a question of whether we want to do that in stabl Jan 15 21:54:48 at a minimum all of the ones other then the change of the directory structure I think we do.. Jan 15 21:55:03 I personally think the directory structure change is a good idea.. but I'm not sure it is for general zeus maitneance Jan 15 21:57:40 fray: well, the filename limiting is way more invasive Jan 15 21:58:13 but without it, things are broken Jan 15 21:59:13 fray: and have been since forever Jan 15 22:00:23 but it was bad enough I couldn't build.. **** ENDING LOGGING AT Thu Jan 16 02:59:59 2020