We lost 2 network switches at about 12am Central file.
After Restoring these switches our backup file of the switch config that was stored on another system was present but after reviewing that file it was only a partial of the data. The export of the config wasn't gathering all the data.

We had to manually reconfigure all the vlans and ips manually and took many hours to complete this task.
Service was restored at about 4am and took another couple hours cleaning up any issues with all the services.

Afterwards there was a handleful of dedicated server clients that couldn't access their servers and they changed the server password from what we had on file. This resulted in 1 to 2 more hours for those clients until we could get their updated windows passwords.

 

Starting Friday 4/27/2012 at 11PM CST the control panel service will go down for maintenance on the backend database. This is a short notice but the control panel service has started to slow down over the past couple days.

Over the past week a user from linux1 kept crashing mysql1 at odd times throughout the day. We traced it to 1 user specifically and found the database eating up the full /tmp partitian with one of their query sorts. So we created a larger 10Gig folder for /tmp and move mysql tmp to that location and they filled it up and crashed mysql again.

After going over their configuration file there was a setting wrong that was allowing the mysql instance to try and user 1200% of ram. Mysql config was set to allocate 37G of ram. This mysql server only has 4G available. After we tuned the mysql config variables where only 3gig can be used at 1 time giving 1gig to apache and the os.

We believe this now resolves the issues  with it crashing and will be monitoring mysql very closely with these new changes. This weekend will be perform an OPTIMIZE as many of the databases are fragmented. Remember Mysql2 is open for business if you want to create a database there and soon MySQL3 will be release. 

Thank you for your patience and understanding during this past week.

Over the weekend all servers had the Operating System updated with patches and reboots were applied.

We apologize to you for our short notice but our ongoing security precautions we don't like to schedule downtime for these in advance because hackers in the past like to take advantage of this and try to hit the servers with the known exploits for the exact updates we were going to patch for.

Manually upgraded proftpd on linux1, linux2 and linux3 servers because of a security vulnerability that was released recently.

http://www.proftpd.org/docs/RELEASE_NOTES-1.3.3c