**** BEGIN LOGGING AT Thu Oct 13 02:59:57 2011 Oct 13 22:09:40 http://wee.e-lnx.org/tmp/thememanager-prealpha/ if someone wants to test the app, but be warned at the moment the theme applying takes about 5-10 minutes :) this is something that I will work on to take that under 1 minute before alpha release on Monday / Tuesday Oct 13 22:50:27 ok so its the node.js that causes the 5-10 minutes time, executing the script from terminal takes 30seconds :| Oct 13 22:50:56 is there a wrong and correct way to execute system files from node.js that I am not aware of :) Oct 13 22:51:17 sconix, halfhalo would know Oct 13 22:51:27 yes he would Oct 13 22:51:44 whats the script look like now? Oct 13 22:52:15 so I use exec call from node.js to execute "sh script.sh" and it takes ages to run Oct 13 22:52:33 so I guess I am using wrong way to execute it Oct 13 22:54:24 no, that should work fine. Oct 13 22:54:54 hmm then what the hell, even on emulator when the command is called by node.js service it takes 5 minutes to complete Oct 13 22:55:00 when I run it over novaterm it takes 15seconds Oct 13 22:55:03 whats cpu usage like? Oct 13 22:55:43 lunasysmgr seems to be taking a lot of cpu Oct 13 22:55:50 like 60% Oct 13 22:56:10 when the script is run Oct 13 22:56:10 is the script sending alot of data back to you? Oct 13 22:56:33 it should not, its directing all output into log file, tried even sending in dev/null isntead and no difference Oct 13 22:56:47 i.e. the logging the script does, it does quite a lot though Oct 13 22:57:02 like 1000 lines of log per run Oct 13 22:57:46 but if I change the log file to be /dev/null it did not change a thing, maybe took 30 seconds away Oct 13 22:58:24 see if ls-monitor is sending messages regarding the logging Oct 13 22:59:06 lots of filenotifyd calls Oct 13 23:00:08 so thats most likely the problem but what causes that Oct 13 23:00:35 ah no that comes from the log file which I palced back Oct 13 23:00:41 let me test with the /dev/null thing Oct 13 23:02:45 ok the /dev/null silences the ls-monitor Oct 13 23:02:57 but realy it does not help much regarding the execution time Oct 13 23:03:35 are you logging the execs error, stdout, and stderr to see if theres any issues there? Oct 13 23:04:07 haven't looked, will set it to log them so I can see, but I am pretty sure I looked them at some point and they were empty Oct 13 23:09:24 does node.js have any other calls than exec? Oct 13 23:10:39 ah shit it does have output in stdout I forgot to silence one thing, lets see if it makes difference Oct 13 23:13:18 ok now now output but still takes 2 minutes to complete through node.js call and 20seconds from command line Oct 13 23:13:34 another thing to check is to see if the script is still slow if you run it in the apps chroot if it has one, but not in node. Oct 13 23:14:30 not sure I got it, the script is in the node.js services dir at the moment, the node.js service is rooted service so where I should put the script? Oct 13 23:14:43 rooted=not jailed Oct 13 23:17:08 hmm. i really don't have any ideas why it would be slow then. since the second you call the sh script.sh, node is just sitting there waiting for that process to finish. Oct 13 23:17:26 yeah thats why this is really puzzling me as well Oct 13 23:17:47 I was blaming the script for a long time until I finally realized to run it from terminal Oct 13 23:18:01 could not believe that it could be the node call Oct 13 23:18:05 so something odd is going on Oct 13 23:18:16 could try to run if via the node command like, see if its slow outside the service architecture Oct 13 23:18:56 will have to try different things, but there was no other calls to execute system commands in node.js than the exec call right? Oct 13 23:19:36 just spawn, which exec wraps Oct 13 23:34:52 tried even that just to be sure, no effect :( Oct 13 23:54:12 ok in the device its bit smaller different 5mins vs 7mins so the problem at least partly is in the script itself need to optimize it Oct 13 23:57:53 well whats interesting is that the difference in emulator and most likely in device comes from the fact that service is called from enyo app... Oct 13 23:58:21 if I call the same command from command line with luna-send the service executes the script in same time as jsut executed from command line Oct 13 23:58:25 interesting... Oct 14 00:14:26 sconix: is the source for all this (already) available somewhere? Oct 14 00:14:50 * dtzWill just found http://git.webos-internals.org/applications/theme-manager/ nvm :) Oct 14 00:15:25 just pushed the latest changes Oct 14 00:16:14 that takes ~5 minutes to run?? Oct 14 00:16:25 update.sh i'm assuming is what is being discussed? Oct 14 00:16:29 that is rather odd :( Oct 14 00:16:37 yes it runs the aupt scripts Oct 14 00:17:54 seems that the 2 minutes comes from being called from an app, if called with luna-send it takes less :) Oct 14 00:17:57 odd Oct 14 00:18:08 but it seems that the greps are the main cause for the time Oct 14 00:18:13 will have to optimize the scripts Oct 14 00:19:05 now I understand why the md5sum file was generated on package side Oct 14 00:19:08 well the first suggestion i was going to make seems to be addressed--making use of optware's version of various tools where possible, might be faster than the busybox swiss-army equivalents Oct 14 00:19:12 I can implement the same for theme manager so its not a problem Oct 14 00:19:34 meh just wanted to see code so i could bounce through it and help if anything stood out. many eyes are good for a project or something :) Oct 14 00:19:37 err on server side for patches not package side :| Oct 14 00:19:47 yeah sure Oct 14 00:20:00 not because i think i'm necessarily gonna fix anything O:) Oct 14 00:20:18 but I now know how to "fix" the problem and where it is, just got side tracked the oddity that in emulator the difference is so big when running from node and when from console Oct 14 00:21:27 am i the only one that wonders what would happen if shell scripts (and their invocation of common system tools) could be automagically compiled into optimized versions that do the same thing? Oct 14 00:22:10 save various exec overheads, use of pipes, etc Oct 14 00:22:10 it would make quite big difference Oct 14 00:22:23 as well as being optimized for their respective inputs--specialized for the flags given, etc, etc Oct 14 00:22:56 sounds like a fun project XD, and yeah offhand i'd expect it'd make a big difference Oct 14 00:23:52 important that you can do what things like other scripting languages do--keep the script source as the entry point and just automagically build specialized versions as needed (preserve ability to hack on script trivially, there's a reason these aren't all written in C hehe) Oct 14 00:26:17 * halfhalo pines for multithreaded js and hw accelerated js and dom and css stuff.... Oct 14 00:31:28 one wuick optimization made the applying time go from 7mins to 3mins :) Oct 14 00:31:45 now I can go to bed and on Monday work off two minutes more and then release alpha version Oct 14 00:45:08 sconix: \o/ gj Oct 14 00:45:31 added caching of those md5sum files and now it takes 1minute :) Oct 14 00:45:37 haha wonderful Oct 14 00:46:05 will do for alpha, maybe optimize more later on, sadly have to leave this for monday, headin to a cabin for the weekend Oct 14 00:46:30 so good night and be back on monday Oct 14 00:47:51 sconix: have fun Oct 14 00:48:10 sconix: don't forget to push :3 Oct 14 00:49:03 I did not push this change yet since I need to check that all is correct and don't ahve time now, so the slower version will have to do until monday, its still good for applying only single components from themes ;) Oct 14 00:49:25 rwhitby: will do, its always relaxing to get time off from computers :) Oct 14 00:49:28 oh for sure haha was just curious what you did, np. Oct 14 00:51:20 well just made TM to make own "database" of md5sum for only picture/css/sound files and then added extra "caching" in the aupt script to put used md5sums into tmp file since those md5sums are fetched 3 times during the process so getting them 2 times from very small file reduced the time even more Oct 14 00:51:37 oh, wonderful Oct 14 00:52:13 so now what took 150secs per 60 files takes 25secs Oct 14 00:52:24 quite big difference Oct 14 00:53:11 still the initial grep takes 20secs then the "cached" ones only couple secs Oct 14 00:53:51 could optimize the initial fetching as well to get that down to 5 secs as well but that would need parsing json files from a shell script so maybe not worth the trouble Oct 14 00:54:33 although I could do that on server side so maybe I get this to go under 30secs for applying large theme :) well lets see on monday Oct 14 00:54:59 server=service side Oct 14 00:55:04 definitely need some sleep now :) -> Oct 14 00:57:18 well good thing is that this optimizing is also working for the aupt scripts so that makes installing patches faster and therefore less time for something to go wrong while patching Oct 14 00:57:43 although only patches that have additional files are mine :) Oct 14 00:57:52 oh well does not hurt anyway Oct 14 00:58:22 indeed, sounds like good progress across the board :) **** ENDING LOGGING AT Fri Oct 14 02:59:57 2011