Why the hell is r3v down so much lately?

[regnu]

"the webserver (which is a separate server running Windows Server 2008 SP2)"

There's your problem....




i keed I keed..


Thanks for keeping the site going James!

[slammin.e28]

Thanks James! Work is lame without R3V!

This.

This is also why we get our panties in a bunch when it is down, lol! ;)

[Eric]

, requiring me to go down to the datacenter and solve the issue on site.

You mean moms basement, right?:rofl:

[Lof8]

Thanks James! Work is lame without R3V!

[scabzzzz]

God what a pain in the ass. I dont know how you keep doing this site James. Hope its somewhat worth your time. Everytime the site goes down, I think to myself "If I was James, I'd probably be mumbling and complaining that I have to work on this shit outside of my normal work".

[Maluco]

you get what you pay for bitches!!!

[ck_taft325is]

James, shoot me a text about those lights sometime. Not sure your number saved when I got the new phone.

[djmoska]

in james we trust.

yes we do.....james for president ......

[EndreM]

I want my money back!

[Bishop]

My head exploded

[87e30]

Thanks for making it happen Criv!

[woodisgood]

Thanks for what you do James, I love this site.

[James Crivellone]

Network Attached Storage.

I'm in the same field, and I feel your pain.

Yeah. This little guy.

Cheap, simple, Raid 1. great for backups

http://www.newegg.com/Product/Produc...82E16822148485

[InuFaye]

Network Attached Storage.

I'm in the same field, and I feel your pain.

[T Verdier]

The challenge appears to be PHP getting stuck and not recovering from the database running its daily backups.

Several weeks ago I changed how the backups scripts are designed due to frequent corrupt backups that I could not test restore from. During this period I upgraded PHP to the latest version in the 5.2.X string, implemented a new backup NAS, and cleaned up several of the manual scripts I had written to keep the site auto-maintained throughout the time its operational.

A few weeks after all of this happened, during the 2am backup script (which is a series of *nix shell scripts that mount a samba share (the NAS) begin a mysqldump, and clean out whatever logs I don't need anymore) the webserver (which is a separate server running Windows Server 2008 SP2) would begin getting stuck and amassing php-cgi.exe processes (microsoft and php's suggested method of using PHP on Windows is via Fast-CGI) causing the site to not respond to PHP queries. As 99% of the site IS PHP this began to cause an issue.

At that point I made changes to the w3wp IIS worker process to attempt to isolate the amount of memory PHP could use and have it automatically recycle the process in the even that it got too high. I noticed that 99% of the time a simple IISRESET would clear the issue and the site would go back to normal performance.

After making the changes to the worker process, and configuring automatic iis process recycling the problem appeared to have gone away.

Yesterday PHP stopped responding completely requiring a full reboot of the server. Due to my own negligence of leaving the Seagate NAS CD in the drive the server tried to boot from that and got stuck, requiring me to go down to the datacenter and solve the issue on site. Removing the CD quickly solved that issue and got the site running again.

This mornings issue was different. PHP was responding, however any login / database write activities caused PHP to get stuck again.

I changed PHP back to 5.2.5 (X64) changed a few other settings and we will see how it acts this evening. If not I'll try a few other things.

Sorry for the downtime. I don't like making large changes to the site as they can easily cause issues (like THESE).

I'll figure it out, one way or another I'll get it stable again.

Nas CD?