The challenge appears to be PHP getting stuck and not recovering from the database running its daily backups.
Several weeks ago I changed how the backups scripts are designed due to frequent corrupt backups that I could not test restore from. During this period I upgraded PHP to the latest version in the 5.2.X string, implemented a new backup NAS, and cleaned up several of the manual scripts I had written to keep the site auto-maintained throughout the time its operational.
A few weeks after all of this happened, during the 2am backup script (which is a series of *nix shell scripts that mount a samba share (the NAS) begin a mysqldump, and clean out whatever logs I don't need anymore) the webserver (which is a separate server running Windows Server 2008 SP2) would begin getting stuck and amassing php-cgi.exe processes (microsoft and php's suggested method of using PHP on Windows is via Fast-CGI) causing the site to not respond to PHP queries. As 99% of the site IS PHP this began to cause an issue.
At that point I made changes to the w3wp IIS worker process to attempt to isolate the amount of memory PHP could use and have it automatically recycle the process in the even that it got too high. I noticed that 99% of the time a simple IISRESET would clear the issue and the site would go back to normal performance.
After making the changes to the worker process, and configuring automatic iis process recycling the problem appeared to have gone away.
Yesterday PHP stopped responding completely requiring a full reboot of the server. Due to my own negligence of leaving the Seagate NAS CD in the drive the server tried to boot from that and got stuck, requiring me to go down to the datacenter and solve the issue on site. Removing the CD quickly solved that issue and got the site running again.
This mornings issue was different. PHP was responding, however any login / database write activities caused PHP to get stuck again.
I changed PHP back to 5.2.5 (X64) changed a few other settings and we will see how it acts this evening. If not I'll try a few other things.
Sorry for the downtime. I don't like making large changes to the site as they can easily cause issues (like THESE).
I'll figure it out, one way or another I'll get it stable again.
Several weeks ago I changed how the backups scripts are designed due to frequent corrupt backups that I could not test restore from. During this period I upgraded PHP to the latest version in the 5.2.X string, implemented a new backup NAS, and cleaned up several of the manual scripts I had written to keep the site auto-maintained throughout the time its operational.
A few weeks after all of this happened, during the 2am backup script (which is a series of *nix shell scripts that mount a samba share (the NAS) begin a mysqldump, and clean out whatever logs I don't need anymore) the webserver (which is a separate server running Windows Server 2008 SP2) would begin getting stuck and amassing php-cgi.exe processes (microsoft and php's suggested method of using PHP on Windows is via Fast-CGI) causing the site to not respond to PHP queries. As 99% of the site IS PHP this began to cause an issue.
At that point I made changes to the w3wp IIS worker process to attempt to isolate the amount of memory PHP could use and have it automatically recycle the process in the even that it got too high. I noticed that 99% of the time a simple IISRESET would clear the issue and the site would go back to normal performance.
After making the changes to the worker process, and configuring automatic iis process recycling the problem appeared to have gone away.
Yesterday PHP stopped responding completely requiring a full reboot of the server. Due to my own negligence of leaving the Seagate NAS CD in the drive the server tried to boot from that and got stuck, requiring me to go down to the datacenter and solve the issue on site. Removing the CD quickly solved that issue and got the site running again.
This mornings issue was different. PHP was responding, however any login / database write activities caused PHP to get stuck again.
I changed PHP back to 5.2.5 (X64) changed a few other settings and we will see how it acts this evening. If not I'll try a few other things.
Sorry for the downtime. I don't like making large changes to the site as they can easily cause issues (like THESE).
I'll figure it out, one way or another I'll get it stable again.
Comment