**** BEGIN LOGGING AT Mon Oct 02 03:00:01 2017
Oct 02 10:15:01 <DocScrutinizer05> ohnoes, load 35 again
Oct 02 10:16:10 <DocScrutinizer05> pabs3: too many trac.fcgi again, load shoots the roof, no free RAM
Oct 02 10:16:49 <DocScrutinizer05> well, the latter is not *really* the problem, it seems it's mostly cache/buffers
Oct 02 10:18:29 <DocScrutinizer05> pabs3: http://susepaste.org/7667687
Oct 02 10:26:54 <DocScrutinizer05> and since neither http://docs.openmoko.org/trac/ nor e.g. http://wiki.openmoko.org/wiki/Distributions#Images drops requests with a "temporarily overheated", the situation explodes in an avalanche as soon as there's one more request per time window than the server can serve
Oct 02 10:28:07 <DocScrutinizer05> wow, after 2 minutes, http://wiki.openmoko.org/wiki/Distributions#Images browser claims "55% progress on loading"
Oct 02 10:29:34 <DocScrutinizer05> http://susepaste.org/76704260
Oct 02 10:32:10 <DocScrutinizer05> also *cough* @ /usr/bin/git --git-dir=/space/git/repositories/git/kernel.git rev-list --header --full-history --max-count=101 --skip=100 refs/tags/v2.6.23-rc1 -- fs/ext3
Oct 02 10:36:44 <DocScrutinizer05> or stuff like usr/bin/git --git-dir=/space/git/repositories/git/kernel.git rev-list --header --author=Alistair John Strachan --regexp-ignore-case --fixed-strings --max-count=101 --skip=0 557e993224b12f6f9d
Oct 02 10:39:15 <DocScrutinizer05> I don't think *anybody* is actively using (as in: "writing") git or particularly trac anymore. Either we shut that stuff down completely or just convert it to read-only static content
Oct 02 10:40:11 <DocScrutinizer05> I'd not be surprised if trac gets massively abused by spambots
Oct 02 10:40:21 <DocScrutinizer05> creating link farms
Oct 02 10:49:28 <DocScrutinizer05> maybe a robots.txt on trac may already help
Oct 02 10:51:26 <DocScrutinizer05> seem obvious to me that something tries to do a wget mirror of the docs.openmoko.org and git website, with infinite depth so e.g. git gets searched for bizarre queries like "--author=Alistair John Strachan"
Oct 02 10:52:52 <DocScrutinizer05> pabs3: your MaxClients helped in that it now seems steady @ load:30 instead of former load:140
Oct 02 10:53:45 <DocScrutinizer05> still we need a load <8
Oct 02 10:53:55 <DocScrutinizer05> aiui
Oct 02 11:22:51 <DocScrutinizer05> being a *total* noob on webserver config, I'd prolly consider looking into  MaxSpareServers, MaxRequestsPerChild, ThreadsPerChild, ServerLimit, and MaxSpareThreads in addition to MaxChilds
Oct 02 11:23:50 <DocScrutinizer05> and check robots.txt
Oct 02 11:59:55 <DocScrutinizer05> pabs3: please check http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html
Oct 02 12:22:17 <DocScrutinizer05> particularly http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxprocesses
Oct 02 12:22:41 <DocScrutinizer05> assuming the server uses mod_fcgid
Oct 02 20:54:54 <DocScrutinizer05> pabs3: did you change any settings?
Oct 02 20:55:35 <DocScrutinizer05> right now only 4 *trac.fcgi* processes and basically load:0
Oct 02 20:56:25 <DocScrutinizer05> 5
Oct 02 20:57:52 <DocScrutinizer05> unrelated SecInfo: https://security.googleblog.com/2017/10/behind-masq-yet-more-dns-and-dhcp.html
Oct 03 00:10:42 <pabs3> DocScrutinizer05: no, Harald may have
Oct 03 00:11:03 <DocScrutinizer05> anyway looked good
Oct 03 00:11:12 <DocScrutinizer05> still looks good
Oct 03 00:11:36 <DocScrutinizer05> I wonder if the spiders stopped or some config got changed
Oct 03 00:12:21 <DocScrutinizer05> don't you think we could completely shut down admin-trac.om.org?
Oct 03 00:13:21 <pabs3> no idea, I can look later (half awake at this point)
Oct 03 00:14:51 <DocScrutinizer05> np, right now everything is absolutely fine :-) Maybe consider adjusting http://httpd.apache.org/mod_fcgid/mod/mod_fcgid.html#fcgidmaxprocesses later on? I'm not saying it's the right parameter to tweak, just looks somewhat related to me (apache/fcgi/trac noob here)
Oct 03 00:16:10 <DocScrutinizer05> hmm dynamic.kbtelecom.net  and wa.optusnet.com.au
Oct 03 00:17:39 <DocScrutinizer05> so yep, most likely Harald?
Oct 03 00:18:29 <DocScrutinizer05> or somebody using a leete VPN ;-)
Oct 03 00:21:29 <pabs3> yeah
**** ENDING LOGGING AT Tue Oct 03 03:00:00 2017