Server outage 0823 UTC Tuesday - Fixed

As of 13:40 we are back up and running again.

We had been experiencing some delays and lockups on a server, so I applied some new Windows and ColdFusion updates and restarted the server. For some reason it did not come back online. There were 5 different data center personnel involved in troubleshooting and fixing the problem. The primary disk passed all the tests. The RAM was replaced, and the BIOS was updated. Finally we ran CHKDSK on the drive repairing any errors. At 1300 UTC we reinstalled the operating system, and that finally fixed the problem.

System is up again

The problem with the network router at our hosting facility has been fixed, and the system should be accessible to everyone again.

Network router has made the system unavailable

Our hosting facility has had a network router failure. Our servers are behind this router, and so you cannot access our servers until they fix the router.

Up-time Stats for July 2010

From our Fort Lauderdale, Florida testing center:
July 1 to August 1, 2010

2,973 attempts,
2,973 successful hits,
100.00% up-time
Average response time: 3.33 seconds for page load

System will be down 0400 to 0500 July 19 UTC

During the conversion to the new server farm, I failed to create one of the index files that is necessary for the Account Manager to work properly. Without this index, it takes about 30 minutes for the screen to load, which makes the program totally useless. With the index in place, it takes a few seconds.

Unfortunately, it will take about an hour to create this index file, during which time the entire system has to be taken out of service.

The Account Manager is the only tool that will allow you to split your account into multiple accounts, with different time periods or QTHes, and that will move the eQSLs around to the proper new account automatically.

We are going to bite the bullet and create the index tonight, July 18 at 11pm Central Time, which is 0400 UTC July 19. Hopefully it will take less than 1 hour, but we really have no idea how long the index creation will take on these new faster servers.

73,
Dave N5UP

New System is working great!

So far, the new system has been working fantastically well. With a maximum of 240 simultaneous browser connections, the application server has been running at a maximum of 10% CPU utilization. Meanwhile, the database servers is still serving over 99.5% of all database requests directly from memory instead of requiring a disk access.

Scary? I'll tell you what's scary... and sad...

Turning out the lights on 2 servers that have served you perfectly, without a hiccup, for 2 1/2 years, deleting all the files, hoping you did get everything moved over, checking your backup and your off-city backup, and then checking it all again, and then entering the service cancellation order and logging off for the last time.

Sad, and just a tad scary.