**** BEGIN LOGGING AT Sun Sep 12 02:59:57 2010 Sep 12 04:22:01 * ppmt is away: Away Sep 12 04:26:53 cpg: it is not a good idea to use nand for swap Sep 12 04:29:22 hmm Sep 12 04:29:31 lifetime issues? Sep 12 04:29:36 or ? Sep 12 04:36:50 tinker-f595: is that an issue with life of the nand or performance, or ? Sep 12 04:36:56 seems to give errors Sep 12 04:37:24 lifetime for a start Sep 12 04:37:40 better off using a cheap sd card Sep 12 04:37:59 or a usb hard drive Sep 12 04:40:15 k Sep 12 04:40:24 say i do not care about lifetime Sep 12 04:40:35 tinker-f595: do you know if it's possible at all? Sep 12 04:41:30 do not know if it is psosible Sep 12 04:41:42 but it will drastically shorten the life time Sep 12 04:42:00 if it is writing a lot to swap **** BEGIN LOGGING AT Sun Sep 12 04:52:32 2010 Sep 12 04:58:01 Call me naive, but why use swap? Just plan your application usage, and let OOM handle the rest. ;) Sep 12 05:04:44 that is naive. most oom handling is, let just say, less than graceful Sep 12 06:16:00 cpg: depends on the kernel version Sep 12 06:16:19 and how many processes are you trying to run at once? Sep 12 06:16:43 and how much memory do they use? Sep 12 06:16:48 we're trying to reduce them ... Sep 12 06:16:56 we put swap on a disk Sep 12 06:17:18 however, as soon as swapping starts .. performance goes way down Sep 12 06:17:34 what do you have that starts swapping? Sep 12 06:17:38 it ought to not have such a degradation Sep 12 06:17:48 running amahi www.amahi.org Sep 12 06:18:07 running on a sheevaplug runs ok Sep 12 06:18:35 trying to reduce the processes to be able to run on a dockstar or pogo Sep 12 06:19:32 hmm you cna do all of that with some shell scripts and if you want to give it a nice ui a web page Sep 12 06:19:53 sure :D Sep 12 06:20:03 or is that what you are doing? Sep 12 06:20:17 quite a bit more Sep 12 06:20:27 but yeah, you can look at it that way Sep 12 06:20:58 amahi has the largest set of webapps out there, with one click install Sep 12 06:21:34 Friends of mine do all that sort of stuff with shell scripts Sep 12 06:22:25 we have our own private distributed system we experiment with Sep 12 06:22:40 spread across 4 states Sep 12 06:22:48 nice Sep 12 06:23:00 amahi has thousands of users Sep 12 06:23:09 but it started as x86 Sep 12 06:23:12 with lots of resources Sep 12 06:23:26 just recently started on the plug Sep 12 06:23:32 trying to cut the fat out Sep 12 06:23:34 I keep telling the person that writes the bulk of the scripts she could make money form them. Her shell scripts are very elegant Sep 12 06:23:38 but still Sep 12 06:23:43 swapping should not be so bad Sep 12 06:24:02 should not swap at all for what you are doing Sep 12 06:24:16 the web interface is in RoR Sep 12 06:24:36 until we switch to dnsmasq, we're stuck with bind and named Sep 12 06:24:43 sry dhcpd Sep 12 06:24:53 not exactly supermodel thin Sep 12 06:24:58 hmmm might be the RoR Sep 12 06:25:08 yes, and we're trying to reduce that Sep 12 06:25:11 depending on how the code is written Sep 12 06:25:25 but in the mean time, going to swap ought to not be so degrading Sep 12 06:25:54 well bind and named should not use much Sep 12 06:26:03 resources Sep 12 06:26:08 so we were trying to get some swap on something faster than the HD, just to see the perf difference Sep 12 06:26:20 just for testing/evaluation Sep 12 06:26:41 that is a long term project Sep 12 06:27:06 but for now, need to evaluate with a faster swap, to see if that *is* in fact the factor Sep 12 06:28:37 well have you analyzed memory usage and cpu usage? Sep 12 06:29:11 yes, to a fair extent Sep 12 06:29:50 we cut out some fat already from the RoR Sep 12 06:30:09 it goes about 15 to 30 MB into swap Sep 12 06:30:13 what web server is it sitting on top of? Sep 12 06:30:19 apache Sep 12 06:30:51 and you are not loading any unnecessary apache modules? Sep 12 06:31:02 and you have tuned apache? Sep 12 06:31:11 no Sep 12 06:31:19 we just started tuning Sep 12 06:31:22 ah Sep 12 06:31:45 you may find that you can tweak apache to use less memory Sep 12 06:31:54 however, going into swap ought to not bring the machine so far into it's knees Sep 12 06:31:55 might take a while but it may be possible Sep 12 06:32:09 yes, i know. taht is a long term project Sep 12 06:32:38 well is it the going into swap or is it also spiking cpu usage? Sep 12 06:33:20 and by bringing it to its knees what do you mean? Sep 12 06:33:37 how do you characterize that? Sep 12 06:34:13 cpu goes up, for sure, but on a sheeva, it works certainly much faster when the app installation process kicks in Sep 12 06:34:26 which is when the spike comes in Sep 12 06:34:43 something that takes a few seconds takes several minutes Sep 12 06:35:44 out of curiosity while this is happening are you ssh'd into the sheevaplug and running any analysis tools? Sep 12 06:35:55 no Sep 12 06:35:58 occasionally top Sep 12 06:36:35 might be worth while and also analysis the ethernet port to see if it is getting saturated Sep 12 06:36:56 well need to go to sleep Sep 12 06:37:01 kk Sep 12 06:37:03 it is late here Sep 12 06:37:12 good luck Sep 12 06:37:16 thanks Sep 12 06:37:18 fun fun Sep 12 06:37:50 might be interesting for oyu to write a paper on tuning the plug Sep 12 06:37:53 cia Sep 12 06:37:55 ciao Sep 12 06:47:32 i think it would be Sep 12 06:47:51 in our case it would not be too hard. have a lot of fat to begin with Sep 12 07:29:07 cpg: If your web framework is swapping, you've already lost. Your stack should never swap. Hence, no need for swap. ;) Swapping kills performance. Sep 12 07:29:33 trying to shrink it Sep 12 07:29:50 swap is good so your host doesn't shit itself with OOM before Sep 12 07:30:07 cpg: Once swap starts to be used, everything after the initial swap will start to be swapped out, not just the original process. Sep 12 07:30:19 Not if your swappiness is set low. Sep 12 07:30:41 also, the original process is not swapped Sep 12 07:33:14 cpg: But, if you are running apache, then multiple processes handle the same "session". Sep 12 07:33:47 yup ... just reducing that as we speak! Sep 12 07:35:10 bkero: It just delays the inevitable... Plus, OOM killer can be tuned. Sep 12 07:36:22 (along with other memory nobs) Sep 12 07:39:43 NightMonkey: Why not just tune swappiness instead? Sep 12 07:39:44 cpg: You might also consider lighttpd or ngnix rather than Apache. You often spend less time pairing Apache back than beefing those two web servers up. Sep 12 07:40:16 agreed, but we do a lot of stuff that is apache specific Sep 12 07:40:42 amahi has like 300 webapps in the pipeline. would need to test a lot Sep 12 07:41:02 trying to evaluate how hard it would be to pick up the low hanging fruit Sep 12 07:46:14 NightMonkey: OOM handling is still a feature that needs major improvement in linux. Killing sshd can make things really bad. Sep 12 07:47:16 bkero: Well, it all depends on your use case. Sep 12 07:47:52 NightMonkey: I handle large clusters of database, web, and caching servers. Sep 12 07:47:53 bkero: oom_kill_allocating_task can be used to make the OOM kill the biggest memory hog, which isn't usually ssh. Sep 12 07:48:00 bkero: Good for you. Sep 12 07:48:32 NightMonkey: often that's not ideal though as other things should often be killed before that Sep 12 07:49:44 And having swap allows us to get our monitoring daemons to correct problems with hooks instead of having to deal with OOMed services Sep 12 07:49:56 bkero: We are talking about plugs here, right? Sep 12 07:50:14 NightMonkey: aye Sep 12 07:53:33 Lots of plugs ;) **** BEGIN LOGGING AT Sun Sep 12 11:57:28 2010 Sep 12 19:36:10 hello, i have a plug with sata and i can see the sata device, however, the kernel cannot mount the root partition. it seems like it cannot see /dev/sda1 ... Sep 12 19:36:42 even though uboot was able to see the device and read the kernel from it Sep 12 19:37:46 i also tried /dev/hda1 as boot Sep 12 19:37:51 sorry, root Sep 12 19:44:04 getting this http://www.fpaste.org/T34C/ Sep 12 20:03:24 cpg: what is your bootcmd/bootargs? Sep 12 20:03:59 ide reset; ext2load ide 0:1 0x06400000 /boot/uImage Sep 12 20:04:12 bootm 0x06400000 Sep 12 20:04:35 with ... setenv bootargs_root 'root=/dev/sda1 rootdelay=15 rotfstype=ext3' Sep 12 20:15:11 Tootoot222: to be precise http://www.fpaste.org/2O6q/ Sep 12 20:22:24 Tootoot222: looks like CONFIG_SATA_MV=m in the kernel ... Sep 12 20:22:28 http://sheeva.with-linux.com/sheeva/2.6.35.4/sheeva-2.6.35.4.config Sep 12 22:11:14 Tootoot222: having CONFIG_SATA_MV=y in the kernel did not help Sep 12 22:11:27 not sure what steps we can take here ... Sep 12 22:15:49 cpg: try set root to some other device (usb, sd, etc) Sep 12 22:15:55 and mount the esata Sep 12 22:16:01 from the running kernel/os Sep 12 22:16:15 just to see if it exists, you mean? Sep 12 22:17:24 yeah, see if the kernel can even do it Sep 12 22:17:34 and see, like you said, if it requires some module Sep 12 22:22:01 nope. can't see it Sep 12 22:22:11 even with sata_mv loaded Sep 12 22:22:20 i'll be darned Sep 12 22:22:27 uboot sees it Sep 12 22:25:05 cpg: is it one of those sheevaplug-with-esata or a guruplug? Sep 12 22:25:09 (or some other plug) Sep 12 22:25:42 some other one Sep 12 22:26:05 but i am told it's like a sheeva, with sata built in Sep 12 22:33:51 wait, sata? not esata? Sep 12 22:33:54 they have those? **** ENDING LOGGING AT Mon Sep 13 02:59:57 2010