End User Data Analysis: the Good, the Bad, and the Future
Or why I am building Jig.
“Don’t tell your boss about the BB sheet yet”, I was told at the end of the meeting.
It was early 2009, the bank had been hit really hard by the crisis, and I was a junior developer hoping to make it through the next wave of terminations. I had recently been assigned to the Commodities tech team to help build a new risk management platform. Commodities had suddenly become an asset class. New traders from competing firms had been hired, and operations had to be up and running in a matter of weeks.
At the time the bank was well aware that it should not be running trading operations with Excel. It had been doing so for years, and the risk and tech departments finally had had enough. Measures were imposed to try and curb Excel usage on the trading floor. Trading desks would be fined two million dollars per year if they kept running critical operations from Excel sheets. New projects were downright barred from using Excel at all.
Traders didn’t care. They didn’t trust that the tech guys could pull off building a platform fast enough. And even if they did, they thought that it could not be good enough. They called me in a meeting, told me that they had brought me on the project because they had heard that I was good with Excel. They said that there was a spreadsheet that I could leverage to build up the new risk management platform. It was called the BB sheet. Traders would convince senior management to let them use Excel, that one last time, because they needed to trade, and they needed to trade right now. Until then, I wasn’t to tell my superiors what I was working on.
I left the bank two years later, and in 2013, when the commodities trading activities were finally shut down, the BB sheet was still running the show.
The Motive
Business teams don’t need applications, they need answers to questions. How one gets to an answer does not matter; what matters is 1. how accurate the answer is, 2. how long it took to get it, and 3. how much it cost.
There is usually not much room for improvement regarding the accuracy of an answer. Most questions from business teams involve basic statistical calculations (quantiles, growth rates, etc.), which can either be right or wrong. Similarly, cost is not a very important factor. If a team needs an answer, they’ll pay for it.
Which leaves us with the crux of the problem: speed. How long should it take for a decision-maker to get a question answered? Depending on the choices you make as an organization, this metric can change from minutes to months.
The Path
End user development tools like Excel promise to take all the unnecessary burden out of the application development process. And to a large extent Excel does fulfill that promise.
The biggest selling point for these tools is that they make applications simple to build. But to be honest, that’s not quite the case. Spreadsheets for instance have a tendency grow into incredibly convoluted beasts very quickly. It requires real expertise (and discipline) to be able to create a spreadsheet that doesn’t turn into total chaos after a few iterations.
Instead, my view is that the killer feature for this kind of software is that they allow business teams to be self-reliant rather than depending on external IT departments. “Proper” application development requires at least a UI expert to build the frontend, usually a backend person to code the calculations, and sometimes a “business analyst” to translate between people who don’t speak the same language. Since developers don’t normally report to business teams directly, managers need to get involved too, budgets need to be discussed, and development time scheduled.
Projects following that path are not only long to develop, they are also incredibly stiff. Each iteration is a new trip on the merry-go-round: meetings, specs, dev, test, qa, uat, deployment. What I find particularly telling is that in all cases that I have witnessed, the first feature that business teams ask for after seeing a working application is a button to download the data to Excel.
The Issues
So, why not use Excel for everything? Well sooner or later, people run into one or all of these things:
Spreadsheets get too complicated to work with,
Spreadsheets have bugs,
There are calculations that are hard to do in spreadsheets,
Traditional applications mitigate these problems because:
Code can be modularized,
Code can be audited and tested,
Code can handle arbitrarily-hard calculation.
Unfortunately, none of these properties can be applied to spreadsheets.
You have to realize that these problems are so ubiquitous in the business world that a whole discipline dedicated to mitigate those risks arose. They called it “spreadsheet risk management” and it has its academics, its research papers, its conferences and even its very own EU-funded working group!1
The Present
Today’s fashion is no code and low code tools.2 In the data science world, they’re known as BI platforms. There are a bunch of them. They’re really expensive, and they’re not that amazing. If you look at their demos, what you will inevitably see is something that looks like a glorified pivot table. Someone will get some data, drag and drop fields onto columns and rows, add filtering, choose a metric, tweak the colors, make a chart, etc. All of these are basic Excel features. There is not a single tool out there that demoes how to create a inner join, or create an interpolator, or bin data in buckets that you created. And that’s because these tools drastically lack the expressiveness of code. You can do everything, they say, but my experience is that the only way to do anything out of the mundane is to hire an expert to make the reports and applications that you need.
The Future
My bet is that in the near future, many more people will learn programming. Economy graduates will know Python. Architects will know CAD scripting languages. We will see more and more hybrid profiles emerging out of the blue. We already have DevOps, DevRels, Quant Traders (DevFin?), Digital Artists (DevArt?). I don’t see why it should stop here. A salesman who knows how to script his own tailored CRM becomes superhuman. A product manager who can code can automate its metrics calculations and monitor their evolution. I code stuff for optimizing my real estate searches, for designing my bookshelf, for building toys for my kid.
I envision a future where developers are not siloed away in their own department, but where they work within other teams to automate tedious tasks or enable complex ones. In this future, the coding knowledge of a hybrid DevX person won’t be as broad as that of today’s developers. They won’t know about concurrency problems in multi-threaded systems. They won’t have time to learn the quirks of CSS layouts. But they will know how to generate, load, iterate, and transform data, and they will apply these skills to a field that they master.
We need to make tools for these people. Excel showed us that non-developers could pick up VBA and make incredibly rich applications. I think it is time to push it a step forward. I believe that blending an Excel-like interface and a full IDE will enable an awful lot of different people to boost their productivity through automation, all the while eradicating all the issues that have plagued spreadsheets over the years.
At least this is what I am betting on. And this is why I am building Jig.
http://www.eusprig.org/horror-stories.htm
https://trends.google.com/trends/explore?date=all&q=%22low%20code%22