Let’s face facts. The movement to cloud, which includes digital enablement in general, leaves systems less integrated. The legacy data integration tools of the past no longer cut it, and that results in less value for the business. As we move more workloads into clouds, as well as add stand up IoT and mobile devices, this data integration problem will only get worse.
The problem is easy to understand, but much harder to solve. We need a platform that enables data portability for hybrid cloud operations and real-time cloud analytics, with the ability to connect clouds to Edge computing while maintaining best-of-breed security. Moreover, we need to understand how to view our own requirements, and back those requirements into the right technology the first time.
Keep in mind as you read this post that we’ll be talking about emerging data integration patterns, especially “synchronized integration” to improve the emerging digital enterprise. These patterns, and the technology that adheres to these patterns, differ from those of traditional API to API integration, which are programming intensive and more complex.
Also, the hybrid cloud approach is the de-facto way enterprises now approach digital enablement of enterprises. This means that, in all hybrid cloud instances, we’re adding more silos and increasing complexity.
As we move to cloud-based systems, we are also moving to more silos. The reason is that enterprises are in flight to the cloud, as well as other platforms, such as IoT and Edge computing. Thus, data integration requirements are evolving, and expanding. Things are becoming more complex.
Indeed, the global data integration market is expected to grow from $7.45 billion in 2017 to reach $24.95 billion by 2027 with a CAGR of 14.3%. Factors include the rising use of IoT, smartphones, and the increased usage of cloud computing. Combined, these factors create a demand for effective data integration tools and propel market growth. However, the lack of efficiencies between modern data integration requirements and legacy systems limitsproductivity for the enterprise, and thus impedes hybrid cloud data integration implementation.
Unfortunately, data integration is often an afterthought that was not considered a strategic factor of digital enablement. The end result is the scattering of data that is difficult to sync with other data stores. Thus, there is no “single source of truth,” and operational data is missing key pieces that would allow analytics to take place.
Understanding the damage done by the lack of data integration can be defined as the difference in the potential value that can be delivered to the business versus the existing state. It’s about leveraging technology to maximize the value that enterprises can obtain from their data, including strategic uses such as:
- Frictionless supply chains that allow the state of inventory and production to drive automated reorders and adjustments to suppliers.
- Real time data analytics and decision support that allow leadership to understand the exact state of the business and make agile adjustments.
- Proactive business processes that automate adjustments to maximize outcomes.
The reality is that most businesses are looking at a 10-20 percent loss of value, considering the lack of data integration technology or its usage. While this number will vary from sector to sector, it will always be present at a number greater than 10 percent, even if some data integration technology is leveraged. 100 percent value is found when all systems leverage synced data, no matter if the data and systems are cloud-based, IoT-based, or within all applications that leverage legacy data. By not having a modern data integration approach for digital and multi-cloud enablement, these technologies will fail as well.
A core set of steps to enable data integration for your emerging digital systems is referred to as Defining a Path Forward. These steps include:
Step 1: Assess the degree of separation for data within your organization, or how many silos you’ll create.
You can look at this as a percentage that takes into account data that exists in silos or micro-silos that is not readily shared with other systems. This includes new digital technology, such as IoT, Edge, and cloud computing-based. In most cases, the percentages have gone up 4-fold in the last 3 years.
Step 2: Define the semantics of each source system.
This means the type of data, the structure, the management, the security, and governance, but mostly how data is represented in each data store. Keep in mind that we need to deal with the differences in semantics using transformation mechanisms that are part of the data integration solution.
Step 3: Pick the technology.
Perhaps the most important step is to partner with the right data integration technology provider. While there are many choices, the approaches that each takes to data integration are very different from product to product. Moreover, few have been purpose-built with digital enablement in mind.
The lists of required capabilities are beginning to emerge, including integration with advanced security systems that live in the cloud, the ability to support real-time data for operations and analytics, and finally the ability to define a “single source of truth” for all important data with the organization, such as customers, invoice, product, etc., even if distributed across several different physical databases. Remember the importance of transactional integrity and the flexibility needed when connecting many different data sources and targets to many different clouds.
Step 4: Call to action.
Your digital business means more data in more places on more platforms. Evidence of the expanding use of IoT and cloud computing platforms exists everywhere. However, as database architects accelerate the use of public cloud, they are often constrained by legacy data integration tools, such as those built in the 90s. Keep in mind that both hybrid and multicloud are de-facto ways that enterprises now approach digital enablement.
Fortunately, good solutions are beginning to emerge that are both easy to obtain and easy to use. One such solution that’s available for evaluation is the Griddable.io platform. It enables data portability for hybrid and multicloud operations, offers real-time analytics, and can connect clouds to Edge computing while securing private data.
Griddable.io is a scale-out grid platform that synchronizes data across any set of data stores, on any cloud. This technology uses a policy engine to control the flexible definition of what data is filtered, masked, or transformed in transit. The grid guarantees transactional data is always consistent and recoverable from failures.
Data integration technology has the potential to save you millions of dollars in operational costs, as well as provide strategic savings that can be 10 times that amount. If data integration technology is not on the radar of organizations that are being digitally enabled, it should be.
David Linthicum is the Chief Cloud Strategy Officer at Deloitte Consulting, and was just named the #1cloud influencer via a recent major report by Apollo Research. David is a cloud computing thoughtleader, executive, consultant, author, and speaker.