Forum Apache going nuts daily!!!!!
 

I am running VBulletin 4.0.4

We have a very busy board with over 1.2 million posts and over 150K members. upwards of 1800 users on at
any one time. Busiest ever was 2700 online but that was on 3.6.9, not 4.0.4

Since upgrading, I now have a server that apache spirals out of control at around 1400-1800 users online each day.
It happens every single day betwen 12-1:30PM so I guess you could say that is my peak.
At that point, the machine is generally pushing around 10 mb/s of outbound bandwidth.
We server serves mostly text articles. Except for avatars, icons and a few profile pics, most content is linked
elsewhere at other locations.

The server:
2 X dual core xeon 1.60GHZ with a 1066MHz front side bus and
8 GB's of ram
73 GB SCSI 300 10,000 rpm disk

The System is running FreeBSD 7.1-RELEASE-p3

The problem is always the same, the apache processes run away and hit it's max connections.
The users online when it happens varies. I've seen it anywhere from 1200 to 1893

Server Load Averages 28.03 20.54 22.72 | 1,398 Users Online (1,048 members and 350 guests)
258 web processes running

Server Load Averages 94.87 96.00 85.25 | 1,893 Users Online (1,403 members and 490 guests)
web processes spiked to the max connections of 1000

We tried bumping max connections in MYSQL from 300 to 500 to 1000 to 1300
We have likewise set max connections on apache from a low of 300 to a high of 1600.
Always spirals to it's max connections and stop answering.

The server generally have 1-1.5GB's of ram active, 4GB's inactive a around 1 GB free
and the rest used by wired for the most part and then the rest of the system stuff.

The machine almost never uses swap. From time to time, 40k - 80k but that is rare.

Here are the admincp stats

Server Type FreeBSD
Web Server Apache v1.3.42
PHP 5.3.2
PHP Max Post Size 8.00 MB
PHP Maximum Upload Size 2.00 MB
PHP Memory Limit 256.00 MB
MySQL Version 5.5.4-m3
MySQL Packet Size 5.00 MB
Data Usage 769.94 MB
Index Usage 379.67 MB
Profile Picture Usage 16.49 MB
Unique Members Today 16,949
New Users Today 454
New Posts Today 3,584
New Threads Today 279
--------------------------------------------------------------
We are running APC Cache
$config['Datastore']['class'] = 'vB_Datastore_APC';

My.cnf:
[mysqld]
socket = /tmp/mysql.sock
skip-networking
key_buffer_size = 256M
max_allowed_packet = 5M
table_cache = 2000
sort_buffer_size = 8M
read_buffer_size = 2M
read_rnd_buffer_size = 24M
myisam_sort_buffer_size = 128M
thread_cache = 64
query-cache-size = 128M
max_connections = 2000
------------------------------------------------------------
httpd.conf:
ServerType standalone
ServerRoot "/usr/local"
PidFile /var/run/httpd.pid
ScoreBoardFile /var/run/httpd.scoreboard
Timeout 30
KeepAlive On
MaxKeepAliveRequests 100
KeepAliveTimeout 4
MinSpareServers 10
MaxSpareServers 15
StartServers 300
MaxClients 1600
MaxRequestsPerChild 1000

LoadModule vhost_alias_module libexec/apache/mod_vhost_alias.so
LoadModule config_log_module libexec/apache/mod_log_config.so
LoadModule mime_magic_module libexec/apache/mod_mime_magic.so
LoadModule mime_module libexec/apache/mod_mime.so
LoadModule dir_module libexec/apache/mod_dir.so
LoadModule cgi_module libexec/apache/mod_cgi.so
LoadModule alias_module libexec/apache/mod_alias.so
LoadModule rewrite_module libexec/apache/mod_rewrite.so
LoadModule access_module libexec/apache/mod_access.so
LoadModule auth_module libexec/apache/mod_auth.so
LoadModule unique_id_module libexec/apache/mod_unique_id.so
LoadModule php5_module libexec/apache/libphp5.so
ClearModuleList
AddModule mod_vhost_alias.c
AddModule mod_log_config.c
AddModule mod_mime_magic.c
AddModule mod_mime.c
AddModule mod_dir.c
AddModule mod_cgi.c
AddModule mod_alias.c
AddModule mod_rewrite.c
AddModule mod_access.c
AddModule mod_auth.c
AddModule mod_unique_id.c
AddModule mod_so.c
AddModule mod_php5.c
Port 80

Oh yes, almost forgot. we have noatime set.

Not sure what other info I can provide or that you need. Any ideas?

Any help would be appreciated.
Thank you in advance!