News update 2003-04: April 2003
===================
A monthly newsletter on risk management in financial services,
operational risk and user-developed software from Louise Pryor
(http://www.louisepryor.com).
Comments and feedback to news-admin@louisepryor.com. Subscribe by
sending an email to news-subscribe@louisepryor.com. Unsubscribe by
sending an email to news-unsubscribe@louisepryor.com. Newsletter
archived at http://www.louisepryor.com/newsArchive.do.
In this issue:
1. Troubles never come singly
2. Troubles come in threes (or more)
3. It’s the model that matters
4. FSA update
5. Public Holidays
6. Newsletter information
===============
1. Troubles never come singly
On 9th April the public power supply at Demon Internet’s Network
Operations Centre at Finchley failed. The standby generator started
up as expected; but a fault then occurred in the power control
system so that it couldn’t be used to run the equipment. At this
stage the backup batteries were the only source of power.
Unfortunately they ran out before the power control system could be
put back in action. This wasn’t really surprising, as they are only
intended for use while the generator is being started up. Several
services were affected.
The public power supply was eventually restored some hours later,
but meanwhile there had been a build up of email. Quotas had been
impose on customers during the outage, and some messages were
returned to sender as quotas were exceeded. There were also some
messages that were corrupted during the outage, and could not be
delivered at all.
The problem is that you can’t rely on just one thing going wrong at
a time. And even if Demon had had another line of defence (after
the batteries), there is no guarantee that it wouldn’t have gone
wrong too. However much you try to control risk by taking
preventative action, you just can’t be sure that you’ve done enough
– and it may not be cost effective, anyway.
Demon press releases can be found at
http://www.demon.net/helpdesk/announce/2003/da2003-04-10a.shtml
http://www.demon.net/helpdesk/announce/2003/da2003-04-15a.shtml
===============
2. Troubles come in threes (or more)
During March this year one of Danske Bank’s two main operating
centres was out of action for a week. During this period the bank’s
trading desks, currency exchange and communications with other
banks were shut down. Reports say that the episode had some effects
on the Danish economy; that the Nationalbanken was forced to inject
5 billion kroners into the banking sector to help push transactions
through; and that direct and indirect losses to Danske Bank could
amount to 50 million kroners ($7.2 mill USD).
It all started during the routine replacement of a defective
electrical unit in an IBM disk system. There was an electrical
outage in the disk system, which caused operations at the operating
centre to come to a halt. A few hours later, the disk system was
operational again and the overnight batch runs were started. It
soon became evident that they were not running correctly.
Apparently there was a software bug in the DB2 database system that
Danske Bank uses, and although the database system had restarted
normally after the breakdown there were inconsistencies in the
data. This bug had been present in all similar DB2 systems
installed since 1997, but this was the first time that the right
(or wrong) combination of circumstances had occurred to trigger the
problem.
Worse was to come. During the data recovery process, which in the
end took four days, three more hitherto unknown bugs were
discovered in DB2. The final one (and, reading between the lines of
the Danske Bank report, the final straw) was a problem that
“resulted in new episodes of inconsistent data that had to be
recreated by other methods. This made the process longer and more
complicated.” They eventually used back up data from their other
main operating centre, rather than wait for the software patch from
IBM.
Things could have been worse. Because Danske Bank had two operating
centres, some of their services were completely unaffected.
Moreover, it looks as if their backup (and restoration) procedures
worked when they needed to.
In 1789 Benjamin Franklin wrote “In this world nothing can be said
to be certain, except death and taxes.” Nowadays we should add
software bugs to the list. Until software has been tested under
every possible combination of circumstances, or unless it has been
so rigorously specified that it can be proved to be correct, it is
likely to contain bugs, and those bugs may cause significant
problems.
There’s a brief description of what happened at
http://www.theregister.co.uk/content/53/30095.html
Danske Bank’s report on the incident can be found at
http://frequyff.notlong.com
===============
3. It’s the model that matters
At a press conference on 8th April, Admiral Hal Gehman, Chairman of
the Columbia Accident Investigation Board, discussed the model that
was used to analyse the impact damage due to debris. If you recall,
the prevalent theory is that this was a major cause of the
disaster.
He said “It’s a rudimentary kind of model. It’s essentially an
Excel spreadsheet with numbers that go down, and it’s not really
not a computational model.” The implication seems to be that
computational models and Excel spreadsheets are incompatible.
However, this is not the case. The real problem with the model was
not its implementation, but its basic structure. Apparently it’s a
lookup table, populated with data from controlled experiments.
Unfortunately the piece of debris under consideration is thought to
have had a mass of about 1kg, much larger than any of the
experimental objects. The trouble with lookup tables is that they
are not much good when it comes to extrapolation beyond the limits
of the data.
A predictive model would obviously be more computationally complex,
but that does not mean that it would not be possible to implement
it in Excel. If the financial services industry is anything to go
by, computational complexity has never been a reason for avoiding
Excel. On the other hand, implementation in Excel might well be
inadvisable, because there are few Excel developers who have the
software engineering background to build a sufficiently well tested
and robust implementation.
The transcript of the press conference is at
http://www.caib.us/news/press_briefings/pb030408.html
===============
4. FSA update
Callum McCarthy has been appointed as the new Chairman of the FSA,
taking over from Howard Davies on 22nd September. Unlike Davies,
McCarthy will not combine the position with that of Chief
Executive. The plan is to appoint a new Chief Executive before
September.
According to my count, when McCarthy joins the Board of the FSA
joins there will be thirteen external members, seven of whom have
been in the banking industry at some time during their careers.
There are no external members from the insurance industry, and only
one from investment management.
New consultation and discussion papers out this month:
—————————————————–
CP176 Bundled Brokerage and Soft Commission Arrangements
CP177 Lloyd’s policyholders: Review of compensation arrangements
CP178 Review of prudential regulation of the Lloyd’s market
Feedback published this month:
—————————–
CP148 The FSA’s approach to the use of its powers under
The Unfair Terms in Consumer Contracts Regulations 1999
DP16 Hedge funds and the FSA
Current consultations, with dates by which responses should be
received by the FSA, are listed at
http://www.fsa.gov.uk/pubs/2_consultations.html
===============
5. Public Holidays
The annual confusion has started again. Those of you who live in
England (or many other countries) probably expect to believe their
diaries when the word “Bank Holiday” appears. Those of us in
Scotland know better.
Many bank holidays are public holidays only in England and
Wales. There are no equivalents in Scotland: when the public
holidays are depends on the city. In Edinburgh, for example, we
have public holidays this year on 1st and 2nd January, 14th April
(Edinburgh Spring Holiday), 5th May (May Day), 19th May (Victoria
Day), 15th September (Edinburgh Autumn Holiday), Christmas Day
and Boxing Day. It wasn’t entirely clear to me whether Good Friday
and Easter Monday were holidays or not
However, banks tend to stick to the English bank holidays. Some
other businesses do that too. Others use the Edinburgh
holidays. Some give their employees a choice: take any 8 days as
long as they are either English or Edinburgh holidays. The problem
for many businesses, especially in the financial services sector,
is that customers from outside Scotland expect them to be around
when their English counterparts are.
===============
6. Newsletter information
This newsletter is issued approximately monthly by Louise Pryor
(http://www.louisepryor.com). Copyright (c) Louise Pryor 2003. You
may distribute it in whole or in part as long as this notice is
included. To subscribe, email news-subscribe@louisepryor.com. To
unsubscribe, email news-unsubscribe@louisepryor.com. All comments,
feedback and other queries to news-admin@louisepryor.com. Archives
at http://www.louisepryor.com/newsArchive.do.