Embracing Semantic Technologies to Elevate Storytelling

Creative storytelling is at the heart of everything we do in our dynamic world of media. Today, that artistic, imaginative process is experiencing transformation due to semantic technologies. This is a topic our team at Avid has been focused on, so I was honored to address it in a keynote presentation, “Elevating Storytelling with Semantic Technologies,” at the recent Dublin Tech Summit 2024.

Let’s explore key points discussed, concentrating on the role of semantic technologies, challenges faced by the media industry, and the intersection of human creativity and AI. You can also view the related podcast interview from the Summit.

The Challenge: Growing Pressure on the Media Industry

With a 6.7% year-over-year increase in content production,* largely driven by the booming streaming market, the demand for fresh, engaging content has never been higher. As businesses struggle with declining profit margins, editors and artists are faced with the prospect of producing more with fewer resources, making efficiency in the creative process more important than ever. The challenge is compounded by a decrease in the number of new professionals entering the industry, which means that we need to ensure that the user experience of our tools is simplified and appeals to a more geo-diverse creative industry.

To keep up with the demand, media companies are exploring various strategies to monetize their content. This includes repurposing content from archives and distributing it to different markets, and attracting skilled artists based in different geographies to produce content. With so much creative energy spent in operations such as curating and researching content for re-editing and localization, these operations are the ones to streamline.

The Solution: Better Content Understanding

Simply asking creatives to do more isn’t sustainable. What’s needed is a fundamental shift in how we understand and manage content. We must provide creatives with tools that allow them to express their creativity more effectively, letting the tools come to them rather than the other way around. This is where semantic technologies and knowledge management systems come into play. Semantic technologies are tools that enable machines to understand and interpret human language in a meaningful way. They can be used to store contextual knowledge of the content and production information. They help enhance storytelling by matching the creative user’s natural language queries against the knowledge, and execute actions that the creative user specified using their creative wisdom.

The Role of Knowledge Management Systems

To understand the impact of these technologies, let's first define what we mean by "knowledge." Using the DIKW (Data, Information, Knowledge, Wisdom) pyramid from information science, we can break down the process as a computer would understand it: 

• Data: In media, data is the raw sensory input—audio or video files that a computer can recognize but not understand.
• Information: This is data enriched with context, such as metadata indicating who is in a video, audio, or any other content type, when it was shot, etc.
• Knowledge: This is the result of connecting various pieces of information contextually, creating a deeper understanding of the content. Semantic technologies now help with this goal.
• Wisdom: Wisdom involves knowing how to apply the fragments of knowledge in a specific context to tell a compelling story. This level of understanding remains a uniquely human trait, as it requires cultural context and creative intuition that computers currently lack.

A knowledge management system helps bridge the gap between understanding content and facilitating creativity. By organizing data and information contextually, these systems allow us to create a more intuitive and efficient content creation process. This is crucial in the media industry, where traditional post-production pipelines often operate in isolated silos, each adding different layers of content and information without effectively communicating with one another.

Semantic Technologies: The Building Blocks

Several core technologies underpin the move toward knowledge-based content management:

• Large Language Models (LLM): These models facilitate natural language processing, allowing systems to understand and respond to queries in various languages. Using Retrieval Augmented Generation (RAG) they can be used to augment the LLM’s knowledge with local content, and even API documentation that specifies how to execute certain actions on applications or services. This technology plays a crucial role in content analysis, helping creatives quickly find and summarize relevant content, and execute actions based on natural language prompts.
• Semantic Embeddings: These technologies create multi-dimensional vector representations of media content, enabling semantic matching between different media types (audio, video, text) and allowing for more intuitive content discovery without the need to create metadata.
• Knowledge Graphs: These systems extract and organize information from diverse sources, creating a web of interconnected data that facilitates the discovery of relationships and patterns. This helps in piecing together fragmented information into a coherent narrative.

Real-World Applications and Benefits

Imagine you’re working on a news story about a fire at the Glasgow School of Art. Using semantic technologies, you can quickly find related video content that may not have metadata associated with it, match it with your story, and prepare it for editing—all in a matter of minutes. This not only saves time but also enhances the quality of your storytelling by providing richer, more contextual information.

Another example is in film production, where you can easily track character names, roles, and relationships across multiple databases, scripts, and production logs. This seamless integration of information helps creatives focus on their craft rather than getting bogged down by administrative tasks.

The Future of Media Asset Management

The prototype systems we’re developing, such as Avid Ada, demonstrate the potential of moving from traditional media asset management to advanced knowledge management. These systems will help us better understand user input and content, providing a more integrated and efficient approach to media production.

Collaboration and Innovation 

None of this progress would be possible without collaboration. From the Entertainment Technology Center at USC providing content and data, and technology partners such as The Rebel Fleet collecting the data required using MovieLabs Ontology for Media Creation (OMC), to advice and direction from editors and technologists provided by major Hollywood studios, our partnerships are driving the development and adoption of these technologies. This collaborative spirit is essential as we navigate rapid technological advancements in AI to benefit the media industry.

 

The Human Element in Creative Storytelling

While semantic technologies are transforming how we manage and understand media content, the essence of creative storytelling remains a uniquely enduring human endeavor. AI technologies, specifically semantic technologies, hold promise for elevating storytelling in the media industry by going beyond managing information (metadata) to now holding knowledge. However, the ultimate storytelling still requires human input and cultural context, which machines currently cannot replicate. It’s the wisdom, creativity, and cultural insight of humans that bring stories to life.

As we continue to integrate these technologies, our goal is to empower creatives to tell better stories, more efficiently and effectively. These knowledge management technologies enhance efficiency and creativity by providing tools that allow creatives to curate the various fragments of knowledge using their creative wisdom expressed in their natural language.

For content creators and media professionals looking to integrate AI into their operations, understanding and leveraging semantic technologies can lead to significant improvements in storytelling. By staying informed about the latest advancements and exploring new tools, you can enhance your creative processes and stay ahead in the competitive media landscape.

* According to Carreta Research’s Dec 2023 IABM BAM Market Trends report [1], the Streaming market is driving the most content with an expected Compound Annual Growth Rate (CAGR) of 6.7% continuing into 2027.

Discover how semantic technologies can elevate your storytelling with Avid Ada!