**** BEGIN LOGGING AT Sat Mar 28 02:59:59 2020 Mar 28 08:37:04 Hi, so since about a week or two, I was (seemingly) able to salvage sstate cache functionality for this client project... but some additional state is now biting me and I'm not sure what to look for Mar 28 08:38:03 Basically, in order to test, instead of checking out many GB of source code for every edit/compile/test cycle... I was removing the build directory between builds Mar 28 08:39:23 sorry looking for what that is in BB variables but cant find Mar 28 08:40:18 basically it's a directory that has bitbake.lock, downloads, work, work-shared, and sstate-cache (downloads and sstate-cache default to there if you don't override those to somewhere persistent) Mar 28 08:41:02 So anyway, in between every build I would remove that whole directory... and the results were good, I am only rebuilding recipes when their recipes or source code (or dependencies) change Mar 28 08:41:05 BUT Mar 28 08:41:26 If I remove the whole source directory and checkout somewhere else, I am unconditionally rebuilding glibc on the target Mar 28 08:42:04 (i.e. not the initial stages of glibc, I believe the cross toolchain never gets rebuilt at least) Mar 28 08:42:53 I'm using yocto 2.1.2, and I'm wondering if there is anything in particular I should be looking at before I deep dive again Mar 28 08:48:03 Oh and another significant observation, is that the sstate cache grows considerably more with each build Mar 28 08:48:42 So really, there is some misinterpretation of checksums going on, two separate checkouts of the entire source tree has some low level recipes interpreting different checksums Mar 28 09:18:54 khem: I do not like the idea of dropping the autoreconf pieces from recipes Mar 28 09:19:30 khem: I'll not merge anything for now Mar 28 09:59:15 So as I debug this I find myself deep in lib/bb/siggen.py... Mar 28 09:59:58 I've been seeing this output to `bitbake-diffsigs -t linux-libc-headers fetch`: Checksum for file index changed from 1abd78ec0458ebb5b4998fe9b9c7fb25 to ca09e584b49931b209cf068ae964c12c Mar 28 10:00:26 Naturally I'm thinking there is some kind of file index which bitbake is keeping track of and checksumming Mar 28 10:00:43 But no, to my surprise... it is in fact a file called "index" ! Mar 28 10:01:20 This leads me to suspect, but not be sure of... due to lack of full paths in the output... that we have a .git/index file mucking up the process Mar 28 10:02:44 gtristan45: that sounds possible Mar 28 10:03:18 gtristan45: I can say that bitbake doesn't use a file called "index" Mar 28 10:09:17 yeah the code seems to show this, at least with this 2.1.2 siggen.py is doing os.basename() on the files it reports Mar 28 10:09:49 gtristan45: it has to as the files get placed into the sstate cache and can move around Mar 28 10:10:24 Ummm, but wouldnt they be relative to the directory they might get staged to before a build ? Mar 28 10:10:41 Like I'd expect ".git/index" there, not an absolute path Mar 28 10:10:52 Anyway, not here to bicker about that detail ;-) Mar 28 10:11:00 gtristan45: that code only sees absolute paths so it does the best it can with them Mar 28 10:11:14 * gtristan45 is more wondering why they did not heed your advice in linux-libc-headers.inc Mar 28 10:11:32 People love to ignore that Mar 28 10:11:34 It appears that the build dir for linux-libc-headers has the custom kernel in there Mar 28 10:11:39 Jaysus Mar 28 10:12:20 It does say quite clearly not to do that. Not sure what else we can do Mar 28 10:12:29 I guess I've been building stuff too long, it just doesn't make sense to trigger a full rebuild just because of some custom kernel module header Mar 28 10:12:48 It was easy for them to hack it Mar 28 10:13:52 Still it appears I'm seeing two issues, one being that they are using the wrong thing (custom kernel to build glibc)... the other being that I suspect they are using the wrong checksumming algorithm Mar 28 10:14:20 I.e. it would appear the recipes expect this to be a tarball extraction directory or such, while in fact it's a git repository Mar 28 10:14:42 Hmmm, perhaps that is not baked into yocto (sorry I don't know the insides of yocto enough to know) Mar 28 10:15:37 What happens when (A) You have a bunch of code checked out adjacent to your metalayers as vendor directories... (B) Those checkouts are in fact git repositories Mar 28 10:15:53 On the one hand, you want to be able to trigger rebuilds with unstaged changes Mar 28 10:16:28 And on the other hand, you don't want to take VCS data into account when calculating checksums (since they are unrelated to the payload you want to build with) Mar 28 10:19:02 gtristan45: it will depend on what you configure the system to do Mar 28 10:23:55 Ok I've worked out how it's configured Mar 28 10:23:59 I think ;-) Mar 28 10:25:00 So I've got poky's base linux-libc-headers which want's a specific kernel tarball, this is overlayed with a .bbappend file which ends up setting SRC_URI / S variables to a custom kernel Mar 28 10:26:03 Now I might have completely the wrong idea, but I'm suspecting that the original poky recipe expects a tarball extraction directory and calculates the checksums according to that expectation Mar 28 10:26:52 And that the .bbappend is giving it instead a directory where a git repository is, without telling the build system to expect a git checkout Mar 28 10:28:37 If this is the way things work, I would think I could at least modify the .bbappend to set some variables which would inform the build system to treat the directory like a git checkout (and perhaps then it would know to ignore the .git/ directory) Mar 28 10:29:23 * gtristan45 takes about 5 steps back and tries to reason about whether this matches the behavior he's been seeing Mar 28 10:34:56 https://www.yoctoproject.org/docs/2.1.2/ref-manual/ref-manual.html#checksums gives a good overview about variables and scripts, but I wonder: At which junction is it decided how the payload gets checksummed ? Are there different algorithms for checksumming input source payloads ? Mar 28 10:35:22 * gtristan45 thinks that in some cases, the sha256sum of a downloaded tarball + the checksums of any patches would be sufficient Mar 28 10:35:34 for the inputs of do_fetch() Mar 28 11:07:56 gtristan45: the system only knows what is in SRC_URI Mar 28 11:08:25 RP, I was just looking through https://www.yoctoproject.org/docs/2.1.2/ref-manual/ref-manual.html#var-SRC_URI coincidentally Mar 28 11:08:34 For a tarball, it knows it can checksum the tarball, for a file:// url it will checksum the whole directory Mar 28 11:09:16 Right this is very valuable info thanks, so is there a way to coerce a file:// URL to behave like a git URL ? Mar 28 11:10:27 An awful hack I was thinking of trying, would be do_fetch_append() { rm -rf .git/ } (Assuming that checksumming of the staged files is done after the fetch task) Mar 28 11:11:36 If I were to change this to a git:// URL, then I suppose the build system would want an explicit git commit sha and clone it, which would not match the workflow Mar 28 11:22:46 Hmmmm: https://www.yoctoproject.org/docs/2.1.2/bitbake-user-manual/bitbake-user-manual.html#git-fetcher Mar 28 11:23:35 This SO post https://stackoverflow.com/questions/49535881/how-to-setup-an-yocto-recipe-to-build-from-a-local-git-source ... seems to suggest I can use a git:// URL and configure to "git fetcher" to use a different method of obtaining the code Mar 28 11:23:55 SRC_URI = "file:///home/user/git/myTest/;protocol=file" something like that Mar 28 11:28:05 Ah maybe I misunderstood, I think 'protocol' here only tells the git fetcher to use a different URI when invoking git, which it will do unconditionally Mar 28 12:39:30 New news from stackoverflow: Why some kernel config didn't show in .config Mar 28 18:28:16 Hi all. I'm somehow lost...I've played with yocto and poky. Then I found oe-core. This let me to the question, if there is yocto without poky? Mar 28 18:39:56 Yocto is the Project Mar 28 18:40:16 Poky is a reference distro used by The Yocto Project Mar 28 18:40:56 Poky consists of 4 repos: Bitbake, oe-core, yocto-docs and meta-yocto Mar 28 18:41:40 The Yocto Project uses Poky for Build validation and for QA Mar 28 18:42:23 the basic setup is Oe-core with Bitbake Mar 28 18:42:57 @armpit: thank you. so if i want to make up my own distro, do i need poky? Mar 28 18:43:21 nope. you just need a layer that defines your own distro Mar 28 18:44:00 but meta-poky will be still there? or do i just bitbake-layers remove meta-poky? Mar 28 18:44:06 poky will work with other distros. it is all governed by the DISTRO= variable in local.conf Mar 28 18:44:27 sorry, gotta go :/ Mar 28 18:44:29 meta-poky will be ingnored Mar 28 18:44:40 will be back later. thanks for now :) Mar 28 18:44:50 it better ignore it or we have a bug : ) Mar 28 18:45:08 enjoy Mar 28 18:45:08 thanks for your advice! :) Mar 28 18:45:13 np Mar 28 18:45:50 define your distro via my-layer/conf/distro/{distro_name}.conf Mar 28 18:46:07 than use {distro_name} in local.conf **** ENDING LOGGING AT Sun Mar 29 03:01:48 2020