Joe Lichtenberg, product and industry marketing at InterSystems, explores how businesses can utilise data fabrics to build a better bridge to their murky data lakes.

In the data world, a lot can change in five years. It wasn’t that long ago that data lakes were being used by businesses to try and gain an overall picture of their data health and use these insights to drive crucial decisions.

Yet, the rising prominence of different data types has meant the increasing unreliability of data lakes for their use in analytics and visibility, and have muddied the data waters.

To gain true business insights, intelligent data fabrics are able to connect siloed data to transform and harmonise data, enabling them to be woven into future strategies.

Harmonising different data formats

Different taxonomies, metadata, and structures have historically made up data lake structures in the past. This makes it difficult to integrate, transform, normalise, and harmonise all of these data points so that organisations can gain a consistent and comprehensive overview and are able to use it.

Further complexity has been added due to the growing availability of real-time data and the subsequent requirement to harmonise this alongside batch data. Additional complications arise when businesses need to use real-time and historical data to make decisions in the moment.

Ultimately, data lakes have shown themselves to be incompatible with these requirements, and many organisations are now looking for a way to combine both real-time data and batch data in a way that allows them to gain actionable insights.

Due to the cost and inconvenience of ripping and replacing aging data systems, organisations are looking for a solution that can enhance their current technology arrangements and continue to extract value from them.

In many businesses that are operating in a highly siloed, distributed environment with many legacy applications and data stores, this need is coupled with the requirement for technology that can create interfaces to their existing infrastructure.

They also need to be able to aggregate, integrate, transform, and normalise the data on demand. With data lakes proving themselves to in effect be just another silo, a new approach is needed for businesses to get the most out of the data at their disposal. This is where intelligent data fabrics can provide a steppingstone to the next generation of data architecture.

The role of intelligent data fabrics in the future

Uniquely, intelligent data fabrics have an ability to transform and harmonise the data so that it is actionable, and incorporate a wide range of analytics capabilities, from analytic SQL to machine learning, to support the needs of the business.

By allowing existing applications and data to remain in place, intelligent data fabrics enable organisations to get the most from previous investments, while helping them gain business value from the data stored in lakes quickly and flexibly to help meet the needs of a variety of business initiatives. This includes everything from scenario planning and risk modelling, to running simulations for wealth management to identify new sources of alpha.

Multiple architectural layers would be needed while using traditional technologies to gain such capabilities from data lakes, including scalable data stores, an integration layer, transformation, normalisation, and harmonisation capabilities, a metadata layer, as well as a real-time, distributed caching layer.

Then, there is also a need for an intelligence layer, with application logic and analytics capabilities, and a real-time layer. Building such an architecture traditionally required a wide range of products as well as integrations and maintenance of the products, making it extremely complex and costly to build and maintain.  

By simplifying and streamlining the stack through new technological capabilities, businesses can make it much cleaner architecturally, and simpler from an implementation, maintenance, and application development standpoint.

There is no longer a need for different development paradigms to manage the various application layers.

It is also higher performance as latency is reduced due to the removal of interfaces to connect the different layers of the architecture, allowing organisations to incorporate transaction and event data into analyses and processes in near-real-time.

How data fabrics can enhance digital transformation

On top of these benefits, modern intelligent data fabrics are able to scale out dynamically to help accommodate increases in data volumes and workloads. This is crucial in the finance sector, for example, where markets and volatility levels have spiked during the Covid-19 pandemic.

Furthermore, it can assist a business’ long-term goal for digital transformation as it breaks down data siloes, helping to remove operational inefficiencies and streamline processes which are the central aims of all digital transformation strategies.

Once siloes have been broken down, organisations gain an overarching view of the enterprise data from internal and external sources, and with that comes the synergy to be able to use that data for a wider range of purposes.

Not only does this allow all-important metadata to be accessed along with other insights across the organisation, it also enables data provenance and lineage. This is critical for businesses to be able to understand the source of the data and what actions have been applied to it so that they can validate and trust the data which is being used to make significant business decisions.

Insightful and actionable real-time insights

The benefit of a real-time ‘dashboard’ awaits businesses that successfully incorporate an intelligent data fabric into their operations. To see the value of this in practice, look at flying a plane – a scenario in which pilots need to synthesise a variety of data to do so safely.

Thanks to advances in technology, pilots now have all the signals they need being combined and analysed in real-time and presented in a display and with alerts that can predict the risk of incidents and suggest corrective actions in real-time, without requiring the pilot to manually interpret different signals from the various parts of the plane. In times of crisis, such as an imminent stall, these capabilities become critically important.

Similarly, businesses today want this same capability to filter out the data that isn’t important and to bring the information that is to the surface. These capabilities can steer the business in normal times and become critically important in times of crisis as we’re seeing now.

Looking to the future of data architectures

The world of data is transforming faster than many could ever have imagined, and businesses need to utilise the data they have available to them, both batch and real-time in order to keep up with the rapidly changing pace.

By incorporating data fabrics, businesses can benefit from real-time data insights while allowing them to get the most out of their existing data architecture setup. With macro-economic influences such as Covid-19 impacting on current business operations, greater access to actionable data allows organisations to tackle the unknown now and in the future, with a more comprehensive set of tools as their disposal.