News update 2004-12: December 2004
===================
A monthly newsletter on risk management in financial services,
operational risk and user-developed software from Louise Pryor
(http://www.louisepryor.com).
Comments and feedback to news-admin@louisepryor.com. Please tell me if
you don’t want to be quoted.
Subscribe by sending an email to news-subscribe@louisepryor.com.
Unsubscribe by sending an email to news-unsubscribe@louisepryor.com.
Newsletter archived at http://www.louisepryor.com/newsArchive.do.
In this issue:
1. Year end
2. Certified models
3. FSA update
4. Seasonal greetings
5. Newsletter information
===============
1. Year end
The end of another year, and what an end it is for some! The first
ICAS for UK insurance companies as well as the first year end for
reporting under Sarbanes-Oxley for many organisations. All the new
requirements are creating huge amounts of extra work for the
auditors and consultants. There are occasional reports of how tough
it all is, and how some large(ish) number of companies are not
going to comply with the Sarbanes-Oxley section 404 deadline (75
days after their year end date, for the the auditor to identify
‘any material internal control weakness’ or ‘significant
deficiency’, in verifying that management has sufficient
operational command to produce reliable and compliant financial
reports). I saw one report that said that 300 companies have
already warned the SEC of their likely non-compliance.
So what’s going on? Why is it so difficult for companies to satisfy
the auditors that they can produce reliable and compliant financial
reports? The obvious answer is that a number of companies really
do lack the requisite internal controls, and that they are not
necessarily producing reliable results. In many cases the results
probably are reasonably OK, but it’s more a matter of luck than
judgement. And we’ve seen some cases in which the results weren’t
OK (SunTrust Banks last month, for example).
http://www.computing.co.uk/news/1159434
http://ryanjors.notlong.com
In a company of any size the process of producing financial reports
is incredibly complex, and is only as reliable as its weakest
link. Add in the effects of changes to requirements, which often
lead to people patching the gaps using manual procedures or ad hoc
spreadsheets, and the risks are obvious. Regular readers may
recognise that I am getting on to a familiar hobby horse here. One
aspect of the problem is that you have large, complex software
systems being built by people with no software engineering
expertise. Systems fail and have bugs in even when they are built
by professionals; when built by people who don’t even realise what
they are doing, they are even less likely to be foolproof.
===============
2. Certified models
Steve Rowley sent an interesting response to my piece last month
about the case where the parties concerned had to accept the
result of a spreadsheet error. Steve is a bio-informatics
researcher in the USA.
He says:
There’s an interesting twist on this in the pharma industry.
Particularly in the regulated part of the business (preparing
FDA/EMEA submissions), it’s important that the results you
derive from your data are (a) correct and (b) reproducible.
That leads to 2 tactics: extreme conservatism in study design,
and standardization of tools. Both are along the lines of “use
only what’s known to work”.
The latter is what bears on your case: good laboratory practices
(GLP) or good manufacturing practices (GMP). These are
mind-numbingly complex, rather bureaucratic specifications of
How Things Are Done. They specify every lab technique, every
software package, every bit of computer hardware, documentation
for anybody who wants to repeat the experiment step-by-step, and
so on. If you use ANYTHING else in the process, then you’re not
compliant. (There’s federal regulation in the US, known by the
charming sobriquet “21 CFR Part 11”.)
On the good side, once software has been GLP certified,
everybody trusts it. So the loan calculation story couldn’t
happen, because they’d be using certified software on certified
data. (Well, other things could happen, but not that particular
blunder.)
On the bad side, GLP procedures are completely stultifying.
It’s impossible to do research under such conditions. So often
companies split into 2 parts, only one of which is GLP/GMP
compliant (the government submission part), and one which is not
(the research part).
In fact, there are entire companies whose business is to repeat
your experiments, but this time guaranteeing and documenting
that it’s all GLP/GMP throughout. That is, it’s better to pay
for the experiment TWICE.
Most days, that even makes sense to me. You have one group of
weird, lawless researchers (hey, that’s me!) who figure out new
stuff and another group of button-down careful folk who nail
down every aspect of reproducibility for regulatory submission.
I guess we can bear that expense in pharma, when people’s lives
will depend on the result. It’s slightly unclear how that would
apply to financial services, except that I’m surprised there
aren’t standard, well-trusted, certified packages of software to
calculate things like interest.
There are some interesting comparisons that can be made with
financial services regulation in the UK.
First, the FSA is explicitly not going down the route of approving
specific models.
Also, even if a model has been certified it can still produce the
wrong results. Take the example of calculating loan interest. The
model would presumably make various assumptions about timings of
payments and the interest rate frequency (an annual rate; daily
rate annualised; or whatever). If the assumptions don’t agree with
the actual loan agreement (either because they are hard coded in
the model or someone gets the inputs wrong) then the model will not
produce the correct results. It’s not just the model that matters,
it’s the correspondence of the model to the real world as well.
Second, the division into creative but lawless scientists doing the
actual work and conforming bureaucrats producing the certified
recorded results isn’t an obvious one for financial
reporting. Though it’s a nice image… the actuary with wild hair
and staring eyes stooping over a test tube… the evil laugh… and
the mild mannered accountant clearing up the mess… No, it would
never work.
For a start, while developing a drug and getting regulatory
approval for it is essentially a large, one-off project which may
take several years, financial reporting is not. Financial reporting
has hard deadlines, which come round with horrifying frequency. The
strategy of doing everything twice, once to see what the answer
should be and the second time to prove it, is unrealistic.
It’s clear, though, that financial modelling for reporting purposes
(and probably for other purposes too) has to move more down the
tried-and-trusted, totally-reproducible route, with all processes
documented and all decisions recorded. In practice, this means more
planning ahead and much less last minute “we need these extra
results by yesterday”. It’s all going to have to be less reactive,
and so will seem less flexible.
===============
3. FSA update
At least three quarterly newsletters are issued by the FSA. The
latest issues are as follows:
Financial Crime, December 2004
http://www.fsa.gov.uk/pubs/other/fc_newsletter1.pdf
Life Insurance, December 2004
http://www.fsa.gov.uk/pubs/other/li_newsletter2.pdf
General Insurance, September 2004
http://www.fsa.gov.uk/pubs/other/gi_newsletter3.pdf
All of them include information on how to receive them regularly
(although past history, of which there is admittedly little,
indicates that the “quarterly” is perhaps more of a goal than an
achievement).
Two reports on cost benefit analysis (CBA) have been issued as part
of the N2+2 review (I can’t work out whether my predominant feeling
is “goodness, only two years,” or “goodness, two years
already?”). One is on CBA methodologies
(http://www.fsa.gov.uk/pubs/other/nera_cba_report.pdf) and the
other is on embedding CBA more deeply in the FSA
(http://www.fsa.gov.uk/pubs/other/howell_report.pdf). Before I
looked at them I was hoping to be able to say something along the
lines of “even if you don’t care about how the FSA do things, read
these reports for their more generally applicable comments.” I
haven’t read them very thoroughly, I admit, but so far haven’t come
across many particularly juicy nuggets.
However, I did like the FSA’s comment that “we recognise that
cumulative CBA, in the pure sense of the term, is in practical
terms impossible. This reflects the extreme difficulty of
modelling, with any reasonable certainty, what the net cost-benefit
effect of regulation on UK’s financial services markets would now
be like, compared with a position prior to regulation.”
New consultation and discussion papers out this month:
—————————————————–
None
Feedback published this month:
—————————–
PS04/28 Lloyd’s: Integrated prudential requirements and changes to
actuarial and auditing requirements – Including feedback
on CP04/7, CP04/13 (part) and CP04/15 (part) and ‘made
text’
Current consultations, with dates by which responses should be
received by the FSA, are listed at
http://www.fsa.gov.uk/pubs/2_consultations.html
===============
4. Seasonal greetings
No feeble jokes about the risks of the holiday season – just my
wishes for an enjoyable and relaxing holiday and a peaceful 2005.
===============
5. Newsletter information
This newsletter is issued approximately monthly by Louise Pryor
(http://www.louisepryor.com). Copyright (c) Louise Pryor 2004. All
rights reserved. You may distribute it in whole or in part as long
as this notice is included. To subscribe, email
news-subscribe@louisepryor.com. To unsubscribe, email
news-unsubscribe@louisepryor.com. All comments, feedback and other
queries to news-admin@louisepryor.com. Archives at
http://www.louisepryor.com/newsArchive.do.