This data center has had problems with its database servers for most of a decade, reports a pilot fish on the inside.
"A few years ago, one of the cluster servers failed completely, so the disaster recovery cluster became the production servers," fish says.
"Less than a year after that, those servers started having problems. One server started showing a corrupted hard disk. Database program files went missing, and replacing the disk didn't stop the disk problems.
"I thought anyone with any hardware experience would know what that means.
"Well, it finally happened one afternoon -- the server failed.
"Let that sink in: The disaster recovery server that we were using as production failed because the main production server failed a year before that.