Long way to Mercury
This post is a summary why and how we came to the conclusion that there is a need for our Mercury data processor engine.
The automotive industry is in a huge transformation at the moment, which we call digitalisation sometimes. It means in reality that vehicles provide more and more connected services, and they become more and more intelligent. The whole industry is shooting for the new holy grail called autonomous driving.
Under the hood a car produces gigabytes of data quickly. To test the intelligent functionalities, connected services, this data needs to be gathered from the cars, processed, analysed, transformed, categorised, tagged, etc...
This is where our Mercury Engine comes into play.
We met many use cases, where huge binary datasets had to be processed or streamed. We also met use cases in which vehicles, or IoT devices sent data realtime over MQTT. How does our Mercury Platform serve these needs?
Processing large binary datasets
One of the biggest pro in Mercury, that it can be run on a single core and it can still process
a huge amount of data. Our first test was processing around 300 GBytes of trace files from cars. Mercury processed it in 58 seconds on a single core. Scaling it up in the cloud it can handle a huge amount of data running parallel on many instances.
Mercury is just a piece of software built on state of the art industry-wide well known technologies, like Java and Spring for example. We decided to use Kubernetes to maintain a cloud agnostic approach. An important goal was to provide features so that any developer can add plugins and scripts easily to be executed by the engine. We also provide developer tools enabling developers or testers to execute their local code in a remote Mercury instance. Deployments are also made easy.
We provide default plugins for persistence, filtering, visualisations, and of course our other products can be part of the whole ecosystem. A good example is to filter binary data with the help of the IP Camp AUTOSAR Parser, meaning easy signal data extraction from complex almost cryptic binary dumps.
The following picture shows a possible use case handled by our engine, a reaction time regression test for autonomous driving safety functions.
The possibilities are limitless, but an important feature to note is the realtime operation. Mercury can receive data over MQTT and do all the filtering and processing realtime, enabling automated testing with multiple vehicles on the fly. Data coming from cars are persisted, and with our simple plugin architecture it is also possible to replay the captured test data once again with different configurations.
Check back for more information on our product, we will publish demos and videos on the site later. We are working on AI integration as well at the moment.