Data may not lie, but it sure can conceal…
In our Insights blog from September, we explored the value of independence when considering a review of dealing desk performance, leveraging the events surrounding a well-known historical incident to make some hopefully valuable points. In this post I follow-up on the topic by sharing some insights into how a simplistic acceptance of a firm’s existing performance and/or risk data, justifiably derided as the ‘tick-and-flick approach’, can lead to downstream problems.
Spot the elephant
Lee Baker
Lee Baker is an accomplished data scientist, software creator, CEO of ChiSquared Innovations, and author of at least fourteen books on how to work with data. His almost perfectly-named firm has a vision to “produce statistical analysis systems that take you ‘from data to story’ with as little fuss as possible.”
I was drawn to Lee’s earliest work in 2017 when his hit, Truth, Lies & Statistics: How to Lie with Statistics seemed to confirm a former boss’s suspicion that when Powerpoint was used in business the content was nine-tenths deception. But Baker’s clever rearrangement of a quote often attributed to Mark Twain was also catchy; though Twain borrowed from Benjamin Disraeli who is reputed to have remarked that: “There are three kinds of lies: lies, damned lies, and statistics.”
Truth, Lies & Statistics is an easy and fun read, and with chapters like “Pirates Caused Global Warming” (debunking the Post Hoc fallacy) it’s like a crib book for the statistically devious. But while Twain and Disraeli risked painting actual statisticians in a bad light, Baker mounts a partly-convincing defence, turning his sights on those who might use statistical tricks to “hoodwink and otherwise dupe the unwary.”
What Baker’s works reveal is that there are lots of ways data can be manipulated or misrepresented for gain or deception.
The data challenge
In modern finance the average firm produces quite staggering amounts of data, and there are many ways it is ‘manipulated’ for good or not-so-good, but with the rather obvious rider that it is overwhelmingly intended to be for good. For those faced with the daily consumption of Fantasia-inspired bucketloads of information it can be useful to ask if you truly understand what the information is telling you? But we also suggest you ask: what is it not telling you?
At Martialis, our view is that practically none of the data ‘manipulations’ used to create 1st and 2nd line reporting were ever devised to “dupe the unwary” as Lee Baker put it. In truth, they’re purpose-built, specifically designed to guide and inform, and firms spend millions trying to ensure this. Despite this, our experience has shown that even the most perfectly presented reporting, containing data of the highest quality (clean), can still dupe both the wary and the unwary, and this can create problems if left unchecked.
What has led us to this seemingly contentious conclusion?
It would be easy to answer this by simply reminding readers that humans are fallible, but there are often deep-seated reasons why data can cause problems, which we broadly catalogue as:
Complacency, inertia and variability of experience;
Misinterpretations, and/or misunderstandings;
Misadventure, not exactly “the dog ate my report,” but variants that might surprise, including reporting-line and staffing changes, particularly of management;
Information overload, a not uncommon problem in a heavily regulated industry; and
Lack of reporting completeness and/or modernity.
And it’s this last category that I will expand on today: the lack of completeness and/or modernity in the 3rd-line space, drawing on recent work with clients and the directions we have been taking in our Dealing Desk Review practice.
Lessons from 2004
In my September blog-post I referred to a difficult moment in 2004-finance that shocked dealing desks across Australia and elsewhere (and upon which information can be readily found with any search-engine). What we know from the incident is that dealing staff sought and found ways to conceal unauthorised activity. We also know that senior dealing management’s interest in the underlying activities of the desk proved insufficient, and there were elements of complacency, misinterpretation of data, and some information-overload thrown in for good measure (regulators did eventually note some 800 individual limit breaches had been concealed; so, data was likely piling up somewhere).
However, and this is key; the reporting at the firm which suffered the eventual loss did not produce red-flags of sufficient magnitude at high-enough points on the management-hierarchy to back-up those brave risk management professionals who did alert management to a suspected desk-out-of-control problem.
At this point I will re-emphasise the point I made in my September blog: an independent review of the flags that were being raised in 2003 and ‘04 would likely have saved millions.
Bigger lessons still
Through relatively simple research, we found that since Y2k there have been at least 47 dealing-desk losses of greater than US$100m magnitude, including the Australian loss of our blog. What’s striking is that a clear majority of these were experienced by well-regarded and major global financials.
Of these:
Total losses (of the 47) amounted to U$101.6B at mid-2021 values,.
The largest amounted to U$11.4 billion (2008).
A great many more (that we couldn’t research) comprised losses of less than US$100 million.
Of losses (by the amount lost):
o 78.7% were generated by firms that had a Big-4 audit relationship;
o 39.3% were generated by a Systematically Important Bank; and
o 10.4% involved non-linear derivative products (somewhat surprisingly)
Not all involved fraudulent dealing activity.
Which has, at least in part, encouraged us to expand on the client-driven demand for Martials approach to data-analysis completed in prior years in this specialist field.
What can we deploy (now)
Having commenced desk reviews in 2019, we have steadily grown the areas of this interesting capability, with focus on three broad fields:
1. Discovery – developing a proprietary base-line view of the desk on which the need for deeper review can be determined:
a. Type of desk
b. Unusual or atypical attributes – returns/risk/other
c. Peculiarities versus norms/peers
2. Desk Conformity, with/to:
a. The implied or present regulatory environment
b. Industry standards
c. Market best-practice
d. Policy
3. Desk Alignment, with/to:
a. Strategic intent, e.g., sources of revenue and/or risk, type of desk
b. Adjacent businesses, as stand-alone or as a component with
c. The degree of complementarity
Answering interesting questions
Does the desk generate actual outcomes consistent with its mandate and stated strategy?
Does the desk demonstrate sustainability from a range of Martialis proprietary standpoints?
Are there elements of performance/risk/funding that warrant deeper review?
In generating revenue, is the desk conforming with regulatory and industry standards and expectations?
Does the activity entail supervisory corner-cutting, individual or entity overreliance, or elements of questionable activity?
In the process, we catalogue and flag areas where a desk may be adrift from industry best-practice standards and methods. Our focus is on the alignment to your strategy and how this compares with similar desks at other firms.
We look at the performance over a number of years and your assumptions of revenue sources against the actual revenue in the Discovery phase. Our experience suggests assumptions and performances are often poorly correlated and a detailed investigation can improve business performance.
There are insights, then there’s real insight – the value of independent review
2004 was not a year of pandemic, nor of global financial crises, but it was a truly difficult year for a group of financial professionals. In this blog we ask a simple question – could their truly difficult year have been avoided?
Here I look closely at a major trading desk fraud and interview a senior risk executive at the heart of the investigation. One major failing was a lack of effective oversight and particularly some independent review which could have detected the issues well before they detonated.
So what happened?
The 3rd year of the new millennium began with few ominous portents. The Rover lander was settling itself delicately on Mars, a last VW Beetle had rolled off its production line in Puebla, Mexico, and decision-makers in the United States, having chosen to invade Iraq did not appear intent on doubling down on their chase for weapons of destruction, mass or otherwise; financial markets were simply dozing.
Becalmed in the charm of early-year, the S&P 500 had drifted quietly from 1112 and would fail to crack 1200 until December, ten-year US paper opened at 4.25% and would end the year a mere 2 basis-points lower; currency markets snoozed.
Yet, for a relatively small group of Australian financial markets executives 2004 would be a truly defining year.
At 1:53 pm on January 13th the then largest of Australia’s financial services leviathans made a startling market announcement on the Australian Stock Exchange; its risk teams had identified “losses relating to unauthorised trading” and that immediate action had been taken to close out and therefore “minimise any further losses being incurred.” The Bank did venture a little further, curtly confirming that losses were “not expected to exceed $180 million.”
For those whose year the announcement would define the shock of such a short statement must have come as a mixed blessing; the announcement was brief, the loss seemed contained, could there be much more to see and hear on the matter? Would there more to it?
And of course, the answer was yes, a great deal more.
Beyond wild estimate the ultimate cost to the long-proud firm is impossible to calculate, but we know today that it was more than the $180 million posted in that first ASX announcement. It is very hard to account for the costs of a disrupted business and the lost custom that comes when normal offerings are curtailed by regulatory order. Increased capital charges can easily be estimated, but what of distracted management and teams, and what happens when a full-service business can’t offer a full suite of products to customers? Will risk-managing customers wait until you’ve regained composure?
What we can say is that in less than a month the firm in question conceded that the loss was double that conceded in the original announcement. Double. How was that possible? We can’t be sure.
Within six months the firm exchanged its CEO and Chair, sacked the traders deemed responsible, sacked or forced the resignation of eight senior staff, disciplined or moved on 17 others - and carried through on a plan to restructure its board.
What went wrong?
The cheeky answer is lots, but since the prudential regulator devoted undivided attention to the irregular activity and losses it’s perhaps best to let their catalogue of the problems identified stand. In their report on the matter, we find the following:
traders choosing to conceal their true positions;
missed opportunities to detect and close down the irregular activity;
expressions of concern by counterparties at large ignored (by management);
collusive behaviour of the traders involved succeeded in suppressing many of the bank’s early warning signals; and
Line Management turned a blind eye to known risk management concerns.
Which points to too many things for one blog post to mention, but in trying to answer our question (could this have been avoided?) we’ve turned to a former senior risk executive whose intimate familiarity with this story remains very real after seventeen years.
Former Senior risk executive involved in the investigation –
I was involved as a risk executive and dealt with the desk involved. As the APRA review found, I attempted to bring the problem to the attention of senior and executive management, however my concerns were ignored.
Eventually, they were considering moving me away from my position. I was compromised and warned them that without an independent review the desk situation would end disastrously.
Once I stepped away from my duties there was no one of any experience closely monitoring the desk activity.
We asked for his opinion on another simple question – could independent review have made a difference in unveiling the irregular behaviour?
Former Senior risk executive again –
An independent review of that desk at that time may well have allowed genuine discovery.
When management and staff in the depths of any business are blind to fraudulent activities, independent reviewers have the capacity to provide frank and factual advice to senior management and the board.
Independent reviewers bring a fresh set of eyes to issues that are complex in nature and clouded in “noise” and jargon. An independent review dispenses with reporting alignment, routine familiarity and complacency. Independent reviews can highlight deficiencies in reporting requirements and shine a light on those who should have known better.
I truly believe that if this style independent review had been conducted then 2004 would have been a better year.
What’s clear as we look back through the records since this story first emerged is that it’s worth thinking about the right questions and it’s worth thinking about discovery.
Is that business unit, that desk, doing what you expect it to?
Is it aligned to your strategy and complementing the wider business?
Is it conforming to regulator expectations – and the letter of the law?
An independent review can assist with this discovery and help uncover potential issues that may have been overlooked previously.
Supervision of Markets businesses – the independent review
The previous blog on 10 August 2021 looked at some of the measures we use to establish the performance of desks and larger units in market businesses. We use a range of calculations from readily available data to establish the type of desk and therefore the expected performance. In some cases, this aligns well with the expected strategy while at other times it may not!
This blog looks at the ways the performance reviews can assist managers and senior managers.
Effective supervision
The actual performance and sources of revenue are sometimes difficult to uncover using established processes. Put simply, some experienced staff can find very effective ways to disguise the true strategies to create revenue when oversight processes are well understood and any gaps are well known. This can be as benign as internal arbitrage, exploiting a monopoly or a dominant market position. In other cases, the revenue is derived from ‘scope creep’ where the original purpose of the desk has drifted over time to follow other strategies without explicit agreement from managers.
Alternatively, this can result less innocent practices such as fraud. There have been a number of well documented cases where knowledge of internal oversight practices can enable fraudulent activity.
Regulatory supervision such as the Senior Manager Regime (SMR, UK) and Bank Executive Accountability Regime (BEAR, Australia) have increasingly focussed on effective oversight from senior managers who are expected to be informed about the businesses under their control. Unless senior managers take reasonable steps to understand and test businesses under their supervision the regulatory system may find them responsible for any problems.
The challenge for senior managers is how to get information and analysis on their supervised businesses.
Independent review
Independent review is an important method for gathering information by looking at existing processes with new insights. The usual internal reviews are not simply re-used and adapted slightly for the current focus areas of the business. Rather a different approach is typically employed by independent reviewers which can add to the information generated from the regular oversight.
In addition to new ways of looking at existing businesses, an independent firm will not have the restrictions which can be problematic for staff members. Internal oversight can often follow a very predictable path which has been established for some time. Lack of time and a focus for efficiencies can leave businesses using analysis and oversight practices which are out of date and/or without review.
Internal relationships and politics can also make reviews difficult for staff. Who wants to create internal resentment and additional work?
Independent firms are not aligned with internal parties and bring external experiences to the review process which can provide different and unique insights. It is this approach which can greatly assist senior managers demonstrate effective oversight.
Changes over time
Most desks change their trading and sales approach over time to suit changes in market conditions and customers. This is normal and expected. But is the current strategy and performance still within the expectations of senior management?
Our independent performance reviews look at these changes over time. The product mix, customers, trading patterns and key days are all analysed for their impact on revenue and whether they have adjusted to different trading conditions.
For example, the relative use of linear and non-linear products is quite dependent on market volatility and direction: is this reflected in the desk performance? Is there preference for a desk to be ‘long’ or ‘short’? This can be quite indicative of trading strategies and often the use of the balance sheet advantages to augment revenue.
Does being in the office or working remotely matter?
This section of any review is relatively new because pre-COVID, working remotely was very rare for trading and sales staff. But now it is a reality for many businesses with all or part of the desk regularly working from home.
In the highly scrutinised environment of a trading room, oversight is clear and immediate. Despite the increasing use of monitoring software for computers and other devices, it is much easier for staff to use unauthorised information sources to assist in trading and sales. A private phone or tablet is very much ‘off the grid’ and cannot be effectively banned from a home office in the way they can from the trading room.
If there has been a change in trading or sales approach (and this can be very subtle) it can often be detected in the change of performance from in-office to remote location.
This is one test we have recently added to the review based on changing market conditions.
Effective and demonstrable oversight
The focus of the performance review is to provide a view of the past, current and change in desk performance and strategy of desks. The independent review does not replace the internal reviews: rather it is designed to compliment them by adding a different analysis of the same data.
The obligations on senior managers for effective and demonstrable oversight can better met with this combination of internal and periodic external review.
Summary
An independent performance review of a desk or small combination of desks is where the Martialis approach makes most impact.
Many reviews are done across wide businesses and focus on opportunities and risks for those businesses.
We look at the micro level. Close, forensic investigation of desk performance can assist senior managers making strategic decisions on products and customer groups in the context of the whole business while providing clear evidence of effective supervision.
How do Markets desks really perform?
Almost every firm with a trading-desk capability periodically reviews the performance of their sales and trading teams. Typically, this is done using internal 2nd line (risk) and 3rd line (audit) resources at regular intervals following a routine process of checking controls and a range of related processes.
In many cases, underlying performance is not regularly compared with internal and external measures.
We have done some recent work to independently examine the performance of trading and sales teams at firms and arrived at some interesting results. The true source of revenue of certain desks was quite different to that assumed or expected by management, risk and audit.
In some cases, the daily analysis of the sources of revenue was not complete due to limited data sets while in other cases the analysis lacked critical components to highlight inefficient use of balance sheet and/or capital.
In other examples, analysis showed the expected changes to the capital calculations (FRTB and SA-CCR) will also impact the performance of Markets businesses. The efficient use of resources such as capital and balance sheet will be imperative for future profitability of desks and businesses.
Careful, independent analysis of performance is very effective in identifying changes to trading and sales strategies over time. When many firm’s staff are still operating out of the office environment, fully understanding these changes can prevent desks straying from the agreed strategies.
Strategy versus performance
It’s rare to find desks without a reasonably well defined strategy which is agreed with management and risk reflecting the preferred levels of customer engagement, trading performance and risk limits.
But are desks actually following the strategy? We have found some significant deviations from the strategy which can be found in the true performance of the desk. While this may not be a problem for an individual desk, it can lead to duplication and inefficient use of resources across a business.
Statistical analysis
An effective starting point for our performance analysis is a statistical analysis of the trading and sales results over a number of years. Key statistical parameters can be readily calculated from the daily P&L data which can then be aligned to the expected outcomes for desks with similar strategies.
Desks can be broadly classified as ‘flow’, ‘proprietary’ and ‘hybrid’. Although each desk is not uniquely one type only, they typically have strategies which place them predominately in one or the other. Each type has quite distinctive statistical markers which can be tested against the actual desk performance.
In a number of cases, we have found significant differences between the stated strategies of desks and the actual performance. In some cases, reviews have unearthed concerning patterns that lead to important questions and further investigation. This can better inform managers of the daily activity of their teams and help them align their business to their preferred goals.
Runs of positive performance
Long or short runs of positive performance are very insightful when combined with the type of desk and the statistical calculations for that desk. In some cases, long runs of positive P&L are expected while in other cases a more random dispersion of results is typical.
Anomalies in the expected performance runs can show either poor performance and/or the existence of internal or firm advantages. For example, significant balance sheet used to fund certain assets at comparatively cheap rates may be within the strategy of the desk; or it may not. Either way, the true performance might show inefficient or inappropriate use of cheap funding rates.
We have found this measure helps identify aspects of performance that can be compared with the desk strategy and expected performance.
Different time frames of reference
Most performance analysis is done on a monthly and annual basis. Budgets are set and monthly results are often used to identify and adjust expected targets across businesses for revenue, cost and capital.
While this is a perfectly reasonable management approach, we have found looking at performance on a more micro scale can yield significant information to better manage the monthly and annual performance as well as identify areas of improvement. In particular, the daily and weekly P&L data can be used to effectively identify these areas.
Potential misconduct?
While this is rare it still exists. In many of the public cases, ‘post-mortem’, detailed reviews of daily performance compared with the expected performance have shown that preemptive actions could have averted the fraud and saved considerable losses, reputations and remediation costs. We have all seen how modest business units have led to substantial losses far exceeding long-run profitability.
No manager (front office or risk) wants to find fraudulent activity and an independent review can uncover both potential and actual opportunities for unapproved trading that might otherwise remain hidden.
Summary
Martialis has skilled market professionals with significant experience in desk performance-analysis covering most asset classes and types of activity.
Although the emphasis is on improving efficiency though careful analysis to ensure resources are used according to the strategy, we have also found serious concerns about potential and real fraud.
Independence is key. A fresh look at familiar data and some different techniques of analysis can add significantly to the understanding of desk performances and, crucially help management unearth and understand anomalies.