News update 2004-11: November 2004
===================
A monthly newsletter on risk management in financial services,
operational risk and user-developed software from Louise Pryor
(http://www.louisepryor.com).
Comments and feedback to news-admin@louisepryor.com. Please tell me if
you don’t want to be quoted.
Subscribe by sending an email to news-subscribe@louisepryor.com.
Unsubscribe by sending an email to news-unsubscribe@louisepryor.com.
Newsletter archived at http://www.louisepryor.com/newsArchive.do.
In this issue:
1. Model risk
2. More spreadsheet woes
3. FSA update
4. Time constraints
5. Newsletter information
===============
1. Model risk
A couple of weeks ago one of the big US banks, SunTrust Banks,
announced a restatement of earnings for the first two
quarters. This followed problems they had found with their loan
loss allowances, or more specifically, with the model they were
using for their loan loss allowances. Part of their press release
says: “There were numerous errors in the loan loss allowance
calculations for the first and second quarters, including data,
model and formulaic errors.” In other words, they are saying that
the data that went into the model was wrong, the model itself was
not a good fit to reality, and on top of that they hadn’t even
implemented this faulty model properly. That covers pretty much
everything that can go wrong with a model, if you count data as
including parameters (see my article and slides on how to believe
your models,
http://www.louisepryor.com/papers/actuary_04_08_08.pdf
http://www.louisepryor.com/papers/believeModelsGIRO2004.pdf).
The fall out from the modelling problem was definitely non
trivial. Q1 earnings were restated by 1%, Q2 earnings by 6%. In Q2,
the loan loss allowances changed by 90%. The loan loss allowances
had been overestimated, so this means that in the second quarter
they were out by an order of magnitude. Three people lost their
jobs as a result of the problem, including the Chief Credit
Officer. The Financial Controller was reassigned to a position
“with responsibilities that involve areas other than accounting or
financial reporting” – ie, neither finance nor control.
Moreover, SunTrust’s directors will be unable to sign off under
section 404 of Sarbanes-Oxley at the coming year end. They say that
they will likely “not be able to conclude that the Company’s
internal control over financial reporting was effective at such
date.”
So why did all this happen? Well, apparently they were bringing in
new processes and a new model in order to comply with
Sarbanes-Oxley. This evidently proved more difficult than they
anticipated. They say “The Company’s implementation of a new
allowance framework in the first quarter was deficient. The
deficiencies included inadequate internal control procedures,
insufficient validation and testing of the new framework,
inadequate documentation and a failure to detect errors in the
allowance calculation.” They also point to deficiencies in
spotting the problem, and then in doing something about it. In
particular, “certain members of the Company’s management did not
treat certain matters raised by the Company’s independent auditor
with an appropriate level of seriousness.”
The morals are fairly obvious. First, models matter, and mistakes
in models can be significant. Second, change is risky. It can be
very risky. (On the other hand, not changing also has its
risks). Thirdly, take problems seriously.
http://www.suntrust.com/common/AboutST/NewsRoom/news/20041110.asp
As a footnote to this episode, I note that many of the phishing
spams I’m getting at the moment purport to come from SunTrust. I
hope this email gets through your spam filters.
===============
2. More spreadsheet woes
While I’m on the topic of model errors, let’s get a bit more
specific about spreadsheet errors. There have been good ones
recently – Patrick O’Beirne maintains a good list at
http://www.eusprig.org/stories.htm. Just read it if you don’t
believe that there are large numbers of spreadsheet errors, and
remember these are only the ones that have both come to light and
been made public.
The one that really struck home to me was the one where the judge
said that you can’t undo mistakes. In other words, getting it right
does matter. Two parties had a disagreement about the interest on a
loan (one party failed to pay it, and the other wanted it). They
reached a settlement based on a spreadsheet prepared by one of the
parties. The settlement went through the legal process (something
called a Tomlin Order) and that should have been that.
However, some hours later a mistake was discovered in the
spreadsheet – so the parties had agreed the settlement on the basis
of information that turned out to be incorrect. However, they had
both accepted the spreadsheet and its results earlier, and the
judge ruled that the settlement was for a specific sum, regardless
of how that sum had been reached.
There are now a couple more people in the world who will always
check their spreadsheets.
http://tinyurl.com/6lzct
===============
3. FSA update
Those FSA folk continue to make speeches, some of which are well
worth reading. They are all available on the FSA web site, at
http://www.fsa.gov.uk/pubs. Last month I highlighted a speech that
had been made on financial fraud; this month a report has come out
on Countering financial crime risks in information security
(http://www.fsa.gov.uk/pubs/other/fcrime_sector.pdf). It’s worth
reading on at least two counts. First, it’s pretty interesting;
second, it contains a lot of useful information on resources you
can use to deal with financial crime. Apparently a big problem
among smaller firms is that they don’t know who to report problems
to: this report contains names and addresses.
New consultation and discussion papers out this month:
—————————————————–
CP04/18 Implementation of the Simplified Prospectus requirements
in the UCITS Management Company Directive
Feedback published this month:
—————————–
DP25 Feedback Statement on DP25 – Development of transaction
monitoring systems
CP176 PS04/23: Bundled brokerage and soft commission
arrangements
CP204 PS04/24: Insurance groups – Supplementary feedback on
CP204 and made text
CP04/3 PS04/27: Reforming Polarisation: Implementation – Feedback
on CP04/3 (A menu for being open with consumers) and made
text
CP04/10 PS04/26: Child Trust Funds – Feedback on CP04/10 and made
text
CP04/11 PS04/22: A basic advice regime on the sale of stakeholder
products – Feedback on CP04/11 and near-final text
CP04/13 PS04/25: Amendments to switch on the Integrated Prudential
sourcebook as it applies to insurers – Feedback to CP04/13
Current consultations, with dates by which responses should be
received by the FSA, are listed at
http://www.fsa.gov.uk/pubs/2_consultations.html
===============
4. Time constraints
There never seems to be enough time. The observant amongst you will
have noticed that this newsletter only just qualifies as the
November issue, and that it is shorter than usual. The two are not
unrelated: any longer and it definitely wouldn’t be November any
more. I only hope that not too many unintended errors have crept
in as I rush to write it (the intended errors are, of course, meant
to be there).
Looking at the two stories I wrote about above, SunTrust and the
wrong spreadsheet that the judge said had to stand, you have to
wonder about the role that time pressures played.
SunTrust obviously had a hard deadline; they had to comply with
Sarbanes-Oxley, and that means conforming to the time limits that
it sets. I can certainly imagine that there was a lot of pressure
to get things done quickly, and that possibly testing and review
were skimped as a result. Of course in the end the deadline won’t
be met at all.
In the second case, time pressures probably played a part, but I
bet that over-confidence did too. I can imagine someone thinking
“It’s only a simple little spreadsheet… calculating loan interest
isn’t that hard…” But mistakes can be and are made in simple
spreadsheets too, especially if you are not expecting to make them.
I always think that it’s a pity we don’t learn more about these
mistakes that come to light. Of course we can’t, because it’s all
commercially sensitive information, but it would be really useful
to know what really went wrong and why.
===============
5. Newsletter information
This newsletter is issued approximately monthly by Louise Pryor
(http://www.louisepryor.com). Copyright (c) Louise Pryor 2004. All
rights reserved. You may distribute it in whole or in part as long
as this notice is included. To subscribe, email
news-subscribe@louisepryor.com. To unsubscribe, email
news-unsubscribe@louisepryor.com. All comments, feedback and other
queries to news-admin@louisepryor.com. Archives at
http://www.louisepryor.com/newsArchive.do.