How to build a great Data & AI organisation Part 1: Current state, the future, and gaps.
8/13/2024 | nmb@konfitech.com
Speed is of the essence. Many business leaders across the world have recognized the invaluable power of data and AI, especially if provided to decision-makers in real-time so that they react precisely to the facts and circumstances that the data tells them.
The increase of cloud platforms and the capabilities of advanced analytics on the cloud platform has become a strategic priority for many companies. These cloud platforms being able to ingest, process and provide back clean actionable data to people in the company, especially concerning data and AI.
New technologies and data analysis tools have traditionally just been placed on-top of the old legacy data silos. The emergence of new data streams needing to be integrated into legacy data streams and silos has become key challenges for many companies and has created problems related to management and governance of the data.
Many are adopting a hybrid cloud approach as they have slowly migrated workloads into the clouds. Creating a difficult mix of tech-stacks to manage, and unnecessary overhead. Business leaders and data officers are facing challenges with both building newer and better solutions, consolidating systems, and teaching the workforce how to efficiently make use of the data and AI capabilities.
Data is showing that few are successful in this venture, with 87% of companies failing to execute on their data strategy. The potential of data only increases with the increased adoption of cloud computing. According to MIT, the challenge over the next few years lies within three main areas:
- Improving data management
- Enhancing data analytics and machine learning
- Expanding use cases of enterprise data
Facing these challenges, IT needs to either utilize current resources more effectively or source new ones. Creating the demand for skills to close the gap between business requirements and technology enablement, especially in the domains of data and AI. The need for executive business leaders to find the right platform that can dedicate enough attention and have the right capabilities to support the improvement of data management, data and analytics, and machine learning use cases is the most important.
Finding great implementation partners for each is also important, as to find someone who can deliver the expected solution without more overhead.
One of the major gaps in the technology operating model in terms of data is the talent challenges. Lack of skilled Machine Learning resources that can successfully hand-off to the business is difficult. One must source the right suppliers for these kinds of projects, who has familiarity with building data literacy and successfully implementing ML projects involving data and AI.
All these business challenges mentioned above will bring with them an array of technical challenges as well related to data architecture and strategy. As mentioned above, many are failing to execute on their data strategy, but why? Databricks has stated that within the data architecture many are failing, as setting up the requirements for a modern data architecture requires different workloads:
- Business analytics (Visualizations and usage)
- Data engineering (Cleaning data and providing business users with the necessary data)
- Streaming (making sure data is live and updated)
- Machine Learning (Automation and advanced analytics)
All these which are very different technologies and topics, which require specialized talent and infrastructure. Focused efforts on data and AI can then enable the reliability, scalability, and low cost for all the work this would require.
However, the trends of multi-cloud, hybrid-cloud, and the mixes of both could spawn complexities. As some can’t seem to find the one vendor that suits all their business needs or risk profiles. Which has overall just created more unnecessary complexity, while if you source the right cloud vendor and implementation technology partner, one could stick with them and let them manage overhead.
Avoiding the unnecessary costs and potential losses of potential in returns from AI and gen AI. To build a great data organization, the business must look itself in the mirror, and ask itself if it is addressing problems outlined above, especially in leveraging data and AI effectively.
Stay tuned for part two where we look into what the best data organizations do in terms of alignment and strategy.
Make sure to also give us a follow on: