Using unstructured data to fuel enterprise AI success
Unlocking the Power of Unstructured Data: How Enterprises Can Fuel AI Success
In the digital age, enterprises are sitting on a treasure trove of unstructured data, from call records and video footage to customer complaint histories and supply chain signals. This invaluable business intelligence, estimated to make up as much as 90% of the data generated by organizations, has historically remained dormant due to its unstructured nature, making analysis extremely difficult.
However, with the right tools and strategies, this messy and often voluminous data can be transformed into a precious asset for training and optimizing next-generation AI systems, enhancing their accuracy, context, and adaptability. A compelling example of this can be seen in the US NBA basketball team the Charlotte Hornets, who successfully leveraged untapped video footage of gameplay to identify a new competition-winning recruit.
The Challenges of Organizing and Contextualizing Unstructured Data
Unstructured data presents inherent difficulties due to its widely varying format, quality, and reliability, requiring specialized tools like natural language processing and AI to make sense of it. Every organization's pool of unstructured data also contains domain-specific characteristics and terminology that generic AI models may not automatically understand.
For instance, a financial services firm cannot simply use a general language model for fraud detection. Instead, it needs to adapt the model to understand regulatory language, transaction patterns, industry-specific risk indicators, and unique company context like data policies. The challenge intensifies when integrating multiple data sources with varying structures and quality standards, as teams may struggle to distinguish valuable data from noise.
How Computer Vision Gave the Charlotte Hornets an Edge
When the Charlotte Hornets set out to identify a new draft pick for their team, they turned to AI tools, including computer vision, to analyze raw game footage from smaller leagues. "Computer vision is a tool that has existed for some time, but I think the applicability in this age of AI is increasing rapidly," says Jordan Cealey, senior vice president at AI company Invisible Technologies, which worked with the Charlotte Hornets on this project.
By deploying a variety of computer vision techniques, including object and player tracking, movement pattern analysis, and geometrically mapping points on the court, the team was able to extract kinematic data, such as the coordinates of players during movement, and generate metrics like speed and explosiveness to acceleration. This provided the team with rich, data-driven insights about individual players, helping them to identify and select a new draft whose skill and techniques filled a hole in the Charlotte Hornets' own capabilities.
Annotation of a Basketball Match
Before data from game footage can be used, it needs to be labeled so the model can interpret it. The x and y coordinates of the individual players, seen here in bounding boxes, as well as other features in the scene, are annotated so the model can identify individuals and track their movements through time.
Taking AI Pilot Programs into Production
From this successful example, several lessons can be learned. First, unstructured data must be prepared for AI models through intuitive forms of collection, and the right data pipelines and management records. "You can only utilize unstructured data once your structured data is consumable and ready for AI," says Cealey. "You cannot just throw AI at a problem without doing the prep work."
For many organizations, this might mean they need to find partners that offer the technical support to fine-tune models to the context of the business. The traditional technology consulting approach, in which an external vendor leads a digital transformation plan over a lengthy timeframe, is not fit for purpose here as AI is moving too fast and solutions need to be configured to a company's current business reality.
Forward-Deployed Engineers (FDEs)
Forward-deployed engineers (FDEs) are an emerging partnership model better suited to the AI era. Initially popularized by Palantir, the FDE model connects product and engineering capabilities directly to the customer's operational environment. FDEs work closely with customers on-site to understand the context behind a technology initiative before a solution is built.
"We couldn't do what we do without our FDEs," says Cealey. "They go out and fine-tune the models, working with our human annotation team to generate a ground truth dataset that can be used to validate or improve the performance of the model in production."
Fine-Tuning Models for Context-Specific Data
Second, data needs to be understood within its own context, which requires models to be carefully calibrated to the use case. "You can't assume that an out-of-the-box computer vision model is going to give you better inventory management, for example, by taking that open source model and applying it to whatever your unstructured data feeds are," says Cealey.
"You need to fine-tune it so it gives you the data exports in the format you want and helps your aims. That's where you start to see high-performative models that can then actually generate useful data insights." For the Hornets, Invisible used five foundation models, which the team fine-tuned to context-specific data.
Clear Goals and Commercial Metrics
Lastly, while the AI technology mix available to companies changes by the day, they cannot eschew old-fashioned commercial metrics: clear goals. Without clarity on the business purpose, AI pilot programs can easily turn into open-ended, meandering research projects that prove expensive in terms of compute, data costs, and staffing.
"The best engagements we have seen are when people know what they want," Cealey observes. "The worst is when people say 'we want AI' but have no direction. In these situations, they are on an endless pursuit without a map."
Conclusion
Unlocking the power of unstructured data requires a combination of specialized tools, strategic partnerships, and a deep understanding of the business context. By fine-tuning models to context-specific data and working closely with customers to understand their needs, organizations can transform their unstructured data into a valuable asset for training and optimizing next-generation AI systems.
As the AI landscape continues to evolve, it is essential for companies to stay focused on their goals and commercial metrics, avoiding the pitfalls of open-ended research projects. By doing so, they can unlock the full potential of their unstructured data and drive real business outcomes.
