Blue Monday For Big Bank
by Tim Moran
"IBM Employee Sparks Massive Bank Outage," read the headline at TheRegister.com. On a Monday in early July, "one of Singapore's largest banks suffered a seven-hour IT outage that took down everything from back-office services to ATMs."
The outage began at 3 in the morning on July 5, when this "IBM employee made a procedural error in what was to have been a routine maintenance operation." In the middle of the night, a storage subsystem at DBS Group Holdings gave off an error message indicating intermittent failure. A simple fix was scheduled for 3 AM, but, unfortunately, an "'outdated procedure' was used to initiate the repair, and all IT hell broke loose. By 3:40, 'a technical command function' was mobilized, and at 5:20 a system restart was attempted" that didn't work. Nor did a restart.
Long story short -- and the long story is worth a read -- all was back up and working by lunchtime that day and no data was lost. IBM, which is handing selected IT services for the bank, issued a statement indicating that it was taking steps to make sure personnel were aware of "current procedures."
Concludes the article, "Big Blue did not note if that one unlucky IT admin was receiving the enhanced training, or if he has now become an uptick in global unemployment statistics."