**** BEGIN LOGGING AT Thu Dec 27 02:59:59 2018 Dec 27 20:56:30 oh man, having 'path' set in vim based on bbpath is so convenient, even if it does slow down the load of the first file. wonder how much work it'd be to make it async Dec 27 20:58:08 RP: I wonder if there'd be any value in adding a debug message when a varaible is set to the value it already had. i.e. to find redundant lines in distro and machine .confs, for example Dec 27 21:00:24 I'll have to add this 'path' stuff to vim-bitbake, but make it optional and disabled by default Dec 27 21:12:06 oh, i could improve this by using vim python support when it's enabled, locate bitbake, import bb and tinfoil.. Dec 27 21:13:13 Hmm, do we have a way to do a sort of non-blocking bitbake command? that is, don't wait for a timeout to acquire the lock or talk to the server, do the thing or exit with an error immediately? would b ehelpful when called from an editor for example.. Dec 27 21:13:38 wonder if that could be done programmatically.. Dec 27 22:57:16 kergoth: we don't but it wouldn't be hard... Dec 27 22:57:41 kergoth: I changed the contention code recently Dec 27 22:58:43 kergoth: I've been thinking about parsing performance recently. Can't decide if we can improve it or not... Dec 27 22:58:45 it'd be nice to add more otpions for things like shell completion, editor completion, editor bbpath availability, etc, ideally without massive slowdowns, particularly for memres Dec 27 22:59:04 this is where memres was going... Dec 27 22:59:21 * RP ends up bogged down in keeping the day to day stuff running :( Dec 27 22:59:46 * kergoth nods Dec 27 23:02:39 I wonder how much time we're losing in the actual linewise processing and regular expressions in the parsing. would it be worth reworking the actual parser at that level, or is it not statistically significant Dec 27 23:03:08 it does seem like we should be able to do more to improve performance, seems like we've already hit the low hanging fruit and now it might involve some higher level rework/rethought architecturally... hopefully not, but.. Dec 27 23:03:09 hmm Dec 27 23:09:51 kergoth: Our biggest two performance bottlenecks are the anon python processing and the variable dependency/hash calculations Dec 27 23:10:02 kergoth: file format isn't the problem :/ Dec 27 23:10:07 Ah, interesting Dec 27 23:10:22 kergoth: Can you remember why we use Feeder() in the parser? I seem to remember a multiprocessing bug? Dec 27 23:10:48 yeah, i think it was due to multiprocessing bugs, plural. got that's such a pain in the ass for our purposes Dec 27 23:11:11 kergoth: with a 3.4 min version are those fixed now? Dec 27 23:11:40 kergoth: I did notice that feeder locking was noticeable on the profile :/ Dec 27 23:12:15 honestly given we're dealing with i/o, I'm not sure why we didn't just use threads, cooperative multitasking, or async i/o or fiddling with select. probably didn't want the tasks fiddling with global state, but i think we've mostly addressed that sort of issue at this point due to the memres work Dec 27 23:12:24 didn't *need* processes for this sort of task, really Dec 27 23:12:32 other than avoiding the global issues Dec 27 23:12:51 s/tasks/event handlers and anonfuncs/ Dec 27 23:13:42 kergoth: It would be interesting to compare performance. I might try hacking Feeder() out, see if anything breaks Dec 27 23:14:12 kergoth: I think the key to performance might be minimising the work we do with anon python Dec 27 23:14:36 kergoth: or see if we can do some of it earlier, once in config parsing rather than in every recipe Dec 27 23:14:50 or defer it and do it in the data store at expansion? Dec 27 23:14:52 yeah, i'm sure there are a lot of cases where t hings could be done in particular event handlers, or via inline python function calls Dec 27 23:15:06 anonfuncs are the easy answer at times Dec 27 23:15:14 right, easy but slow :/ Dec 27 23:15:38 kergoth: you should try "rm tmp/cache; bitbake -p -P" and have a look at the profile-parsing.log :) Dec 27 23:16:01 kergoth: I'm sure there are things I'm not seeing Dec 27 23:16:02 sure, i'll take a look, i'ts been a while, and sounds like an interesting task to work on while i'm doing boring bsp work :) Dec 27 23:16:12 (ugh, bsp layers) Dec 27 23:20:23 kergoth: nobody has me play with hardware these days which is kind of sad too... Dec 27 23:32:08 same here until recently. now i'm rusty Dec 27 23:37:48 kergoth: I would be as well :/ Dec 27 23:38:06 kergoth: quick test suggests we don't need feeder now, at least its not obviously breaking Dec 27 23:38:31 doesn't help the lock contention at all though :( Dec 27 23:47:43 RP got to sleep Dec 28 00:01:32 * RP notes its the sentinel handling that is causing the lock waits Dec 28 00:01:56 armpit: I will now :) Dec 28 00:05:44 kergoth: 166s of 194s parsing time is spent in ast.py:finalize() Dec 28 00:06:27 108s in siggen:_build_data() **** BEGIN LOGGING AT Fri Dec 28 02:19:09 2018 **** BEGIN LOGGING AT Fri Dec 28 02:30:05 2018 **** ENDING LOGGING AT Fri Dec 28 02:59:58 2018