2206: Dynatrace Grail and a Mission to Unify Observability Data

Continuous digital transformation has created a data explosion that's overwhelming many organisations. Every tap, click, or swipe from a user, new code deployment or architecture change, and attempted cyberattack generates more data that can be captured and analysed. However, there is too much data to store and use cost-effectively.

Organizations are therefore forced to discard most data or lock it away in cheaper storage layers where it can't be analyzed. This has led to a tipping point, where the datasets used for observability and security analytics are too incomplete to be truly valuable.

The value of observability and security data is in the insights it can provide to help organizations optimize their digital services. However, current approaches to analytics mean they often need help to maximize the detail those insights contain. As well as having an incomplete dataset, most of what they keep is stored and analyzed in silos, so it lacks a crucial ingredient – context. This means any answers from data analytics are often incomplete, imprecise, or even incorrect – limiting its value to the business.

Grail is a data lakehouse, purpose-built to solve these challenges. It combines the structure, management, and querying capabilities of a data warehouse, with the low-cost benefits of a data lake. Bob Wambach shares the story behind Grail and how it enables organizations to store vast quantities of observability and security data cost-effectively and analyze it together in context.

I also learn how Grail uses a massively parallel processing (MPP) analytics engine, enabling organizations to run thousands of queries simultaneously rather than processing them sequentially. As a result, they can get instant, more precise, and cost-efficient insights from analytics.