An interesting technical insight into the IT angle here:
When online bank sites fail
Tom Ilube
Chief executive, Garlik
When I heard the news that crisis-hit Northern Rock's website had crashed last Friday under the weight of numbers of online savers trying to withdraw their money, I immediately thought, "Thank goodness I am not the IT director of Northern Rock today."
Having served as the head of technology (chief information officer) of Egg plc, one of the world's largest online banks for several years, I know exactly how these things work.
I can imagine banking IT directors up and down the country looking at each other nervously thinking, "There but for the grace of God..."
Banks like Northern Rock spend millions of pounds a year on their internet banking systems.
They will have big, air-conditioned rooms filled with dozens of heavyweight computers, called "servers", to do the hard work and several hundred expert IT staff to look after them.
These are big, well-funded, well-supported technology operations, not two guys eating pizza with their feet on the desk.
So one wonders why sites like this crash at all.
Well, I can say from experience that building computer systems that can cope suddenly with a completely unexpected burst of perhaps 10 or 20 times their normal processing volume is notoriously difficult to do, but by no means impossible.
Main approaches
Firstly, a bank such as Northern Rock can make sure it has enough computer servers on standby, ready to swing in to action if the need arises.
This can be very expensive, because for 99.9% of the time, the bank will have large numbers of computers sitting around doing nothing.
However, a bank can make an arrangement with a huge IT company to provide the extra capacity "on demand", ie only when it's needed.
The bank pays something to have this capacity on standby, but not the full cost, rather like an insurance policy.
Secondly, a bank can put in a throttle of some sort to slow down the number of customers hitting the system second by second.
The website will get slower and slower but will not crash suddenly in a messy heap.
Typically, when it gets unbearably slow, customers are automatically diverted to another temporary website which will take their contact details, perhaps their instructions, and get back to them as soon as possible.
Extreme occasions
Thirdly, a bank can choose to ignore the issue - and if a spike of customers actually occurs, it will allow the system to fail abruptly and sort it out quickly afterwards.
There are sound economic reasons for taking this third approach as major spikes happen rarely, if ever.
If it does happen, the damage to their reputation can be immense, but they are betting that consumers have short memories and they will ride it out.
However, some banks are focused on efficient operation and cost control above all else.
They may take the view that the risk to their reputation is outweighed by the financial benefit of saving money by not putting in place contingency plans for these rare and more extreme occasions.
Luddite executives
One of the root problems in many banks is that the business people who run the bank do not understand computers.
Some senior banking executive even take a Luddite pride in knowing nothing about computers.
There are bankers in the UK today who even get their secretary to print out all their e-mails and put them in a folder every morning! :shock:
But in the age of the internet, that is completely unacceptable. It's like bankers boasting about how little they know about accounting.
Meanwhile, the IT chaps don't know how to explain what they do or, more importantly, what risks are being taken as a result of the IT decisions being made.
But actually, the question to be asked is simple.
Every bank chief executive should ask their IT director today, "if we are hit by 10 times our normal customer volume tomorrow, what will happen to our online banking system?"
If you look outside the banking sector, there are businesses that deal with this challenge extremely well.
I recall talking to the IT executives at a major news media broadcasting organisation about how they handle unexpected peaks of traffic.
Online news media, TV and radio companies are great at designing systems specifically to cope with massive spikes.
After all, when there is a huge, breaking news story, that is precisely the time that online performance has to be at its best.
In fact, it would be interesting to contrast the performance of the big news sites last Friday that were reporting the Northern Rock news with the performance of the Northern Rock site itself.
The banking sector can learn a lot from the online news media industry.
Panic
On a day like last Friday, the scene in the IT department at Northern Rock would have been manic.
The bank will have systems that monitor the main online banking system to alert people if there is a problem.
There were probably big screens showing exactly what was happening, second by second, like a mission control centre.
That's where the IT director would have been, pacing up and down, watching developments.
The alarm bells would have started ringing long before the website crashed.
The core team members would be at their desks.
Every supplier company would have key staff in the building or on standby.
New computer servers would have been built and installed as quickly as the guys could do it.
It's a race against time to beef up capacity, but unfortunately on this occasion, the customer volumes grew just too quickly - and horror of horrors, the site crashed.
That's when the IT director picks up the phone to his chief executive to make that call.
Ouch!
Lessons
We've all been there at some stage, whether it was Egg's online systems being overwhelmed by its successful product launches years ago, the government's census data site crashing within hours of launch and being out of action for months, or Northern Rock's current challenges.
One thing you can be sure of, though - you learn fast in a crisis.
That Northern Rock IT director is the only IT executive in the UK online bank world today who has been at the sharp end of a digital run on the bank.
That direct experience is invaluable - and whatever happens to the rest of Northern Rock's executives, he will be in high demand to share his experience across the finance sector and make the whole online banking world safer.
http://news.bbc.co.uk/1/hi/business/7000203.stm