**** BEGIN LOGGING AT Sun Oct 01 03:00:02 2017
Oct 01 06:37:44 <DocScrutinizer05> pabs3: load average: 141.76, 141.82, 140.93
Oct 01 06:38:01 <pabs3> yay
Oct 01 06:39:16 <DocScrutinizer05> a zillion of git processes eating megatons of RAM
Oct 01 06:40:59 <DocScrutinizer05> did you notice that git CVE recently?
Oct 01 06:41:09 <DocScrutinizer05> most likely unrelated, but...
Oct 01 06:42:07 <pabs3> yes
Oct 01 06:42:08 <DocScrutinizer05> I *think* it needed a git push to exploit it
**** BEGIN LOGGING AT Sun Oct 01 10:39:03 2017
Oct 01 14:59:44 <DocScrutinizer05> hmm   KiB Mem:  16156036 total, 14700984 used,  1455052 free,   997208 buffers
Oct 01 14:59:56 <DocScrutinizer05> diesn't look good
Oct 01 15:00:04 <DocScrutinizer05> doesn't
Oct 01 15:01:05 <DocScrutinizer05> the git were a red herring, they terminate in sub-seconds time when there's *some* free RAM still
Oct 01 15:07:14 <DocScrutinizer05> the apache processes seem to many and too greedy (sorted by RSS):http://paste.opensuse.org/93173259
Oct 01 15:09:16 <DocScrutinizer05> sorry, this is sorted by RES now  http://paste.opensuse.org/93533994
Oct 01 15:14:34 <DocScrutinizer05> bash: sudo: command not found  o.O
Oct 01 16:25:57 <DocScrutinizer05> 1541592 free @ 15:08 -> 886332 free @ 16:23. ETA for next lockup 100 minutes
Oct 01 16:26:01 <DocScrutinizer05> pabs3: ^^^
Oct 01 19:40:46 <DocScrutinizer05> hmm a tad longer than I thought... 718184 free
Oct 02 01:43:53 <pabs3> DocScrutinizer05: I guess I need to set some limits somewhere, will look at it today (apache restarted just now)
Oct 02 01:44:23 <DocScrutinizer05> ooh, fine :-)
Oct 02 01:44:40 <DocScrutinizer05> pabs3: bash: sudo: cmd not found
Oct 02 01:44:44 <DocScrutinizer05> known?
Oct 02 01:45:00 <DocScrutinizer05> I had suggested add whell to me, but... without sudo
Oct 02 01:45:10 <DocScrutinizer05> wheel even
Oct 02 01:45:37 <DocScrutinizer05> could you install htop please?
Oct 02 01:46:20 <DocScrutinizer05> still wondering wtf is using up 13443364k of RAM
Oct 02 01:51:01 <DocScrutinizer05> tbh I'd suggest a reboot. those python processes look fishy (330 - 430MB resident), clamd 574MB wtf? mysql always suspicious. and spamd also looks sorta weird
Oct 02 01:53:06 <DocScrutinizer05> I'd prolly blame gitweb to be the culprit
Oct 02 01:54:25 <DocScrutinizer05> wuite likely the results of somebody mirroring http:// dynamic (git?) web content
Oct 02 01:54:36 <DocScrutinizer05> quite*
Oct 02 01:57:05 <DocScrutinizer05> 953932 free   load average: 113.63, 102.97, 100.90
Oct 02 01:59:43 <DocScrutinizer05> OOMkiller kicked in?
Oct 02 02:10:56 <pabs3> made some limit adjustments
Oct 02 02:11:04 <DocScrutinizer05> :-)
Oct 02 02:13:28 <DocScrutinizer05> I wonder which python process is such a long running job to eat a 100+ minutes of CPU
Oct 02 02:14:43 <DocScrutinizer05> is trac.fcgi about bugtracker?
Oct 02 02:16:11 <DocScrutinizer05> and if it is, then who's actually using bugtracker actively anymore and for what?
Oct 02 02:16:12 <pabs3> yeah
Oct 02 02:16:26 <pabs3> basically the server is snowed under by bots
Oct 02 02:16:37 <pabs3> search engines and other bullshit
Oct 02 02:16:58 <pabs3> the Internet isn't a fun place to be for a server these days
Oct 02 02:17:06 <DocScrutinizer05> indeed
Oct 02 02:17:20 <DocScrutinizer05> though particularly nasty is all dynamic content
Oct 02 02:17:36 <DocScrutinizer05> static websites are almost sane still
Oct 02 02:17:59 <pabs3> ack
Oct 02 02:18:25 <DocScrutinizer05> for reference: neo900.org  http://paste.opensuse.org/73779723
Oct 02 02:18:31 <pabs3> you will still get things like referrer spam and people trying all manner of CMS exploits
Oct 02 02:18:50 <DocScrutinizer05> ok, I got no SMTP inbound there
Oct 02 02:19:23 <pabs3> ok, MaxClients seems to have helped
Oct 02 02:19:30 <DocScrutinizer05> but I tend to blame spiders and mirrors crawling trac and git
Oct 02 02:19:34 <pabs3> at least for now
Oct 02 02:20:43 <DocScrutinizer05> could you apt-get install htop please?
Oct 02 02:22:22 <pabs3> done
Oct 02 02:22:27 <DocScrutinizer05> ta :-)
Oct 02 02:28:44 <DocScrutinizer05> gosh, wasn't even aware how pathetic top is compared to htop
**** ENDING LOGGING AT Mon Oct 02 03:00:01 2017