Server keeps crashing, running out RAM
 

My server is crashing constantly. I've had intermittent problems with server overloads for a long time, and have never got to the bottom of them. However, recently I managed to patch up the problem by running a cron that emptied the cache every six hours. This would reduce the RAM used by a significant amount, and although it would start to climb up again, when the cache was emptied it would go back down.

I have 4GB of RAM on my server, and a site that has on average 400-500 people online at most times, and has around 2.2m posts.

However, I upgraded from 3.7.4 to 3.8.4 a week or so ago, and the problems I've had since then have been horrendous. The postindex table got corrupted as the server crashed midway through a repair, and although I managed to fix that eventually I also had to rebuild the search index. That eventually completed this afternoon, but the memory problem has worsened to the point that the server has crashed three times in the last 10 hours.

How can I find out what is eating up all the memory so quickly, as from everything I've been told I should have more than enough RAM to deal with the traffic I have to my site.

My hosts won't help other than to reboot the server for me, as I have an unsupported dedicated server.

These are the 'top' figures from the last time it crashed:

top - 21:12:46 up 3:39, 3 users, load average: 906.21, 800.53, 484.99
Tasks: 1654 total, 63 running, 1588 sleeping, 0 stopped, 3 zombie
Cpu(s): 0.3% us, 99.7% sy, 0.0% ni, 0.0% id, 0.0% wa, 0.0% hi, 0.0% si
Mem: 4148448k total, 4127304k used, 21144k free, 1920k buffers
Swap: 2040244k total, 2040244k used, 0k free, 18692k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
5470 mysql 16 0 710m 175m 3272 S 7.0 4.3 21:47.42 mysqld
1 root 17 0 3540 544 468 R 5.6 0.0 0:09.63 init
870 nobody 25 0 5816 1120 996 D 4.7 0.0 0:04.94 sh
3671 root 18 0 1716 532 440 R 4.0 0.0 0:05.71 syslogd
6259 root 16 0 5816 828 572 D 3.9 0.0 0:07.16 hald