In late 2004 one of the big US banks, SunTrust Banks, announced a restatement of earnings for the first two quarters. This followed problems they had found with their loan loss allowances, or more specifically, with the model they were using for their loan loss allowances. Part of their press release says: “There were numerous errors in the loan loss allowance calculations for the first and second quarters, including data, model and formulaic errors.” In other words, they are saying that the data that went into the model was wrong, the model itself was not a good fit to reality, and on top of that they hadn’t even implemented this faulty model properly. That covers pretty much everything that can go wrong with a model, if you count data as including parameters (see my article how to believe your models).
The fall out from the modelling problem was definitely non trivial. Q1 earnings were restated by 1%, Q2 earnings by 6%. In Q2, the loan loss allowances changed by 90%. The loan loss allowances had been overestimated, so this means that in the second quarter they were out by an order of magnitude. Three people lost their jobs as a result of the problem, including the Chief Credit Officer. The Financial Controller was reassigned to a position “with responsibilities that involve areas other than accounting or financial reporting” – ie, neither finance nor control.
Moreover, SunTrust’s directors were unable to sign off under section 404 of Sarbanes-Oxley at the next year end. They said that they were likely “not be able to conclude that the Company’s internal control over financial reporting was effective at such date.”
So why did all this happen? Well, apparently they were bringing in new processes and a new model in order to comply with Sarbanes-Oxley. This evidently proved more difficult than they anticipated. They say “The Company’s implementation of a new allowance framework in the first quarter was deficient. The deficiencies included inadequate internal control procedures, insufficient validation and testing of the new framework, inadequate documentation and a failure to detect errors in the allowance calculation.” They also point to deficiencies in spotting the problem, and then in doing something about it. In particular, “certain members of the Company’s management did not treat certain matters raised by the Company’s independent auditor with an appropriate level of seriousness.”
The morals are fairly obvious. First, models matter, and mistakes in models can be significant. Second, change is risky. It can be very risky. (On the other hand, not changing also has its risks). Thirdly, take problems seriously.