Six Ingredients of Data Management Intelligence

In recent years, many enterprises have been heavily investing in data management technologies to make their business processes and their managerial decision-making data-driven. It is nowadays common to find a multitude of data management systems in the enterprise environment, ranging from conventional relational databases and data warehouses to data lakes that manage BigData.  This variety of data management systems is largely due to that modern enterprises must manage very large volumes of data from diverse sources, including for example data at rest, data streams with very high ingestion rates, as well as large amounts of transactional data for batch processes.

Managing different types of data in a scalable and cost-effective way is still a great challenge, especially when performing analytics processing over multi-source data. Confronting this challenge is the first important step for turning raw information to intelligent business insights. Nevertheless, while it is a prerequisite step, it is not sufficient. The extraction of useful business insights requires linking, integration and consolidation of datasets from diverse data sources, including structured, unstructured, and semi-structured data. Furthermore, the application of advanced analytics technologies, including different forms of Machine Learning, is essential for tackling complex business problems. Likewise, there is always a need for involving the right people, including experts in IT, databases, data science, as well as subject matter experts in the problem domain at hand. These are the main ingredients of modern intelligent data management.

1. Catalogue Your Data Sources

The development of an intelligent management infrastructure starts from the development of a catalogue of all the available data sources and datasets. Building an intelligent data management infrastructure requires 360 degrees view of the data that are available in the organization. The catalogue is destined to provide such a view, while also enabling dynamic data management as new data sources are added to the infrastructure. Each new dataset must be registered in the catalogue. In this way, data management stakeholders, including data administrators, IT developers, and data scientists can become instantly aware of new data sources as they become available. The catalogue should typically provide metadata about each data source, include ways for accessing the data such as URLs (Uniform Resource Locators) and URIs (Uniform Resource Identifiers).

2. Link and Integrate Your Data Management Systems

In most cases, data management intelligence requires the integration of data from different sources. Take customer profiling and customer analytics application in banking as an example. To construct a credible profile for a customer there is a need for combining data from different operational systems (e.g., e-banking, loans management, and investment systems) and data lakes (e.g., the customers’ social media interactions with the financial organizations) to analytical databases such as data warehouses. This is the reason why there must be a need for linking and integrating diverse data management systems such as relational databases and BigData data lakes. This integration is always challenging, as different systems may use diverse formats and semantics to describe the same data.

Another important concern is the speed and responsiveness of the integration. Once upon a time, it was sufficient to perform batch Extract Transform Load (ETL) operations overnight towards consolidating diverse data sources in an analytics database. Nowadays, there is an increasing number of use cases that require analytics to be run on “fresh” (i.e., real-time) data, which makes the data systems integration process more challenging.

3. Employ Advanced Data Mining and Business Analytics

No matter how efficient the infrastructure is, the intelligence will eventually come from the analytics. To this end, there is a need for designing and deploying a proper analytics infrastructure, including tools and algorithms for advanced analytics. Nowadays, machine learning techniques are increasingly used for extracting knowledge from the raw data. Given the vast amounts of data currently available, deep learning algorithms are gaining momentum and used more frequently than traditional machine learning techniques. Furthermore, in some industries (e.g., manufacturing, transport, gaming) there is a strong interest in reinforcement learning techniques. Overall, to produce useful insights, enterprises must build and integrate a modern analytics infrastructure that supports different machine learning techniques over their data management systems.

4. Prioritize Security and Data Protection

To secure the ROI (Return on Investment) of the intelligence data management infrastructure, there is always a need to protect sensitive data from data breaches and other security incidents. Therefore, security and data protection must be integral elements of an intelligent data management infrastructure. Security deployments should aim at protecting the integrated intelligent management infrastructure from known cyber-security risks while providing strong authentication, authorization, and identity management. Nevertheless, depending on the use cases supported by the intelligent data management infrastructure, additional security concerns may arise. For example, in cases where deep learning techniques are used, the cyber-defense infrastructure must safeguard the analytics infrastructure against specialized adversarial attacks for AI systems such as data poisoning attacks.

5. Visualization of Data Insights

Business insights must be made available to end-users in ergonomic, easy-to-understand, and easy-to-use ways. Therefore, the visualization of business insights in a proper front-end is equally important to the complex back-end developments of the data management infrastructure. As a minimum, intelligent data management applications must include user-friendly dashboards that help end-users understand and use data-driven processes. In recent years, more advanced and versatile visualization techniques are available, including for example augmented analytics, interactive visualizations, and augmented reality.

6. Build the Right Team for Data Management Intelligence

Intelligent data management cannot come without the right people on board. Engaging the right people is one of the greatest challenges, given the proclaimed talent gap in technologies like BigData management and machine learning. Another challenge is that the data management team must comprise people with diverse profiles and skills, including for example database experts, database administrators, software developers, data scientists, business analysts, and domain experts in the use cases at hand.

Overall, establishing an intelligent data management infrastructure at the enterprise level requires a balanced combination of various ingredients, spanning technological, organizational, security, and human factors. Excelling in all areas of data management is very difficult. However, being aware of the steps is important for preparing the transition to intelligent data management. It will not happen overnight, but it must take place in the right way.