QuestDB – Why we invested

Hardware Strikes Back

The biggest thing to have happened in IT for a long time has been the evolution of the public cloud – that is the centralisation, standardisation and industrialisation of computing capability to massively lower the cost per transaction. This is nothing less than the industrial revolution of IT – if you can lower the unit cost on the supply side, you unleash a demand side shock, astronomical application growth. So much is well documented.

What is less well known is that there has been a second revolution under the hood that has enabled this growth – so called SDx, software defined everything.  This can be thought of as the implementation of the famous Marc Andreesen quote that “software is eating the world”. Whether it is software defined data centre, networking or infrastructure, moving functionality into code and abstracting it away from increasingly generic hardware allows you to scale hardware with software economics. Choose any workload you want as long as it runs on x86.

However, it turns out there are limits to this process. There are physical limits in terms of transport of bits – it makes sense to process some workloads closer to where they are created – there are security limits as developers seek to restrict an attack surface, and there are legal limits as people are increasingly concerned about what data gets processed by who and where. Most importantly, there are speed limits as underlying chip architectures reach the limits of physics. Software’s greedy appetite has made developers lazy – but now hardware is fighting back.

https://www.technologyreview.com/s/615226/were-not-prepared-for-the-end-of-moores-law/amp/

First kid on the block to notice this was AI and its favourite cousin Machine Learning. Training neural nets turns out to be really time consuming and compute intensive, so people started running their models on specilaised chips called GPUs – originally designed to manipulate huge amounts of vector data in the video gaming industry – with better results. However, with the extraordinary growth of this application, more specialized chip architectures like Google’s TPU, and Graphcore’s IPU have been emerging. Public clouds now increasingly allow targeting workloads to the right kind of compute architecture, and must maintain differentiated compute capabilities to support that.

It’s About Time

Similarly, infrastructure software has become fat and bloated, built with little regard for efficiency and elegance on the grounds that compute would just continue to get cheaper and faster ad infinitum. One such class of application is databases, an area which every application in the world relies on but which has seen little innovation since SQL was invented in the 70s. Big data leads increasingly to big databases – and management of how applications find, manipulate, update and present that data across the application space has to be increasingly mindful of underlying infrastructure and choosing the right tools.

The particular area where QuestDB play is known as Time Series – TSDB. This is a database designed specifically to handle events – data that carry a time stamp and are records of values at a specific moment. You can think of a sequence of stock prices, temperature samples from an industrial sensor, or internet log data. Time series data is typically sequenced, is a permanent record – rather than being updated or deleted – and is searched for patterns rather than queried. For example, users look for the trading pattern in a series of stock quotes, the anomalous temperature variation in a sensor, or an unusual event pattern that might suggest a cyber-attack in logfile.

But here’s the thing about time series data – it’s growing way faster than any other kind of data. There are more kinds of things issuing more kinds of event information and at ever shorter intervals. We’re looking at devices or pieces of code that may be creating millions of events per day. Tracking and analysing those data, and tracking them over multiple such sources, is becoming impossible for mainstream database software. Existing TSDB software is either highly specialised and hard to use, or built on existing software platforms for ease of use but with all the performance compromises that entails.

QuestDB have come up with a tool that offers the best of both worlds – a software architecture that has been designed from the ground up to be as close to the hardware structure as possible, offering breakthrough performance – and a developer-first approach integrating common query languages and features, and engagement with the developer community in an open source model. The team – with a background in trading, financial markets and blockchain – has been working on this challenge for a number of years and are exceptionally well suited to this task.

We are excited about the opportunity, about the team creating an important piece of the next-generation infrastructure for the Internet.