Server optimization request for a very large installation
 

<div>We are running a very large forum. More than 600,000 users and 23 million posts (in 1 million threads).<ol style="list-style-type: decimal">8 dedicated servers.
Server specifications:

1. 1 Database server, specifications: Code:
   CPUs: 4 x Intel Xeon X7350 (4-Core), 2.93 GHz, Cache 4 MB
   Ram: 64 GB
   Disk: 3 x 300 GB (Raid 5)
   OS: CentOS 5.3
   MySQL version: 5.0.45-log
   
2. 7 Web servers, specifications: Code:
   CPUs: 2 x Intel Xeon E5410 (2-Core), 2.33 GHz, Cache 6 MB
   Ram: 4 GB
   Disk: 3 x 300 GB (Raid 5)
   OS: CentOS 5.3
   Web server: Apache 2.2.3
   PHP version: 5.1.6
   

vBulletin 3.7.6
No innodb tables.
MySQL is the default mysql package from CentOS 5.3
Top stats:

1. Database server: Code:
   top - 14:17:11 up 4 days, 4:07, 6 users, load average: 0.89, 0.84, 0.88
   Tasks: 231 total, 1 running, 230 sleeping, 0 stopped, 0 zombie
   Cpu0 : 20.0%us, 12.9%sy, 0.0%ni, 64.3%id, 0.7%wa, 0.2%hi, 1.9%si, 0.0%st
   Cpu1 : 1.6%us, 9.2%sy, 0.0%ni, 88.7%id, 0.5%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu2 : 2.9%us, 7.6%sy, 0.0%ni, 87.3%id, 2.1%wa, 0.0%hi, 0.1%si, 0.0%st
   Cpu3 : 1.8%us, 8.7%sy, 0.0%ni, 89.4%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu4 : 1.5%us, 9.1%sy, 0.0%ni, 89.2%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu5 : 1.6%us, 9.2%sy, 0.0%ni, 89.0%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu6 : 2.9%us, 7.6%sy, 0.0%ni, 88.7%id, 0.8%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu7 : 1.8%us, 8.7%sy, 0.0%ni, 89.3%id, 0.2%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu8 : 6.8%us, 11.1%sy, 0.0%ni, 81.7%id, 0.3%wa, 0.0%hi, 0.1%si, 0.0%st
   Cpu9 : 1.9%us, 9.8%sy, 0.0%ni, 88.2%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu10 : 3.1%us, 7.8%sy, 0.0%ni, 89.0%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu11 : 2.1%us, 9.3%sy, 0.0%ni, 88.5%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu12 : 1.8%us, 9.8%sy, 0.0%ni, 88.4%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu13 : 1.8%us, 9.8%sy, 0.0%ni, 88.3%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu14 : 3.1%us, 7.8%sy, 0.0%ni, 89.0%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu15 : 2.1%us, 9.3%sy, 0.0%ni, 88.6%id, 0.1%wa, 0.0%hi, 0.0%si, 0.0%st
   Mem: 66000344k total, 65656356k used, 343988k free, 383372k buffers
   Swap: 8385920k total, 4k used, 8385916k free, 62347948k cached
   
2. One of the web servers: Code:
   top - 14:18:01 up 17:04, 2 users, load average: 1.10, 0.93, 0.90
   Tasks: 319 total, 1 running, 318 sleeping, 0 stopped, 0 zombie
   Cpu0 : 52.2%us, 1.0%sy, 0.0%ni, 46.2%id, 0.0%wa, 0.0%hi, 0.7%si, 0.0%st
   Cpu1 : 4.7%us, 0.3%sy, 0.0%ni, 95.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu2 : 2.0%us, 0.0%sy, 0.0%ni, 98.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu3 : 6.3%us, 0.0%sy, 0.0%ni, 93.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu4 : 5.0%us, 0.0%sy, 0.0%ni, 94.7%id, 0.0%wa, 0.0%hi, 0.3%si, 0.0%st
   Cpu5 : 7.0%us, 0.0%sy, 0.0%ni, 93.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu6 : 5.0%us, 0.0%sy, 0.0%ni, 95.0%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
   Cpu7 : 10.3%us, 0.3%sy, 0.0%ni, 89.3%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
   Mem: 4043368k total, 3559700k used, 483668k free, 220188k buffers
   Swap: 8385920k total, 25504k used, 8360416k free, 321476k cached
   

MySQL stats (and report below): <div style="margin:20px; margin-top:5px"> Code:
+---------------------------------+---------------------------------------------------------+
| Variable_name | Value |
+---------------------------------+---------------------------------------------------------+
| auto_increment_increment | 1 |
| auto_increment_offset | 1 |
| automatic_sp_privileges | ON |
| back_log | 50 |
| basedir | /usr/ |
| bdb_cache_size | 8388600 |
| bdb_home | /srv/mysql/ |
| bdb_log_buffer_size | 15750144 |
| bdb_logdir | |
| bdb_max_lock | 10000 |
| bdb_shared_data | OFF |
| bdb_tmpdir | /tmp/ |
| binlog_cache_size | 32768 |
| bulk_insert_buffer_size | 8388608 |
| character_set_client | latin1 |
| character_set_connection | latin1 |
| character_set_database | latin1 |
| character_set_filesystem | binary |
| character_set_results | latin1 |
| character_set_server | latin1 |
| character_set_system | utf8 |
| character_sets_dir | /usr/share/mysql/charsets/ |
| collation_connection | latin1_swedish_ci |
| collation_database | latin1_swedish_ci |
| collation_server | latin1_swedish_ci |
| completion_type | 0 |
| concurrent_insert | 1 |
| connect_timeout | 30 |
| datadir | /srv/mysql/ |
| date_format | %Y-%m-%d |
| datetime_format | %Y-%m-%d %H:%i:%s |
| default_week_format | 0 |
| delay_key_write | ON |
| delayed_insert_limit | 100 |
| delayed_insert_timeout | 300 |
| delayed_queue_size | 1000 |
| div_precision_increment | 4 |
| engine_condition_pushdown | OFF |
| expire_logs_days | 0 |
| flush | OFF |
| flush_time | 0 |
| ft_boolean_syntax | + ->