Future of TV Briefing: The guide to generative AI tools for video making

Love it or hate, generative AI is inserting itself into the future of video making. There are already examples that stretch beyond test balloons to show how artificial image creation tools can assist professional productions. And the types of generative AI tools becoming available for video productions is also expanding. 

So during this year’s summer slowdown — which seems set to actually be a slower solstice in light of the writer’s strike and ebbing upfront market — let’s look at the types of generative AI tools being used to make videos today.

Disclaimer: Before we get into the guide, let me be clear that generative AI poses both opportunity and threat. As covered last month, video makers and VFX specialists are showing how these tools can aid humans in the production and post-production processes. But as the “Joan Is Awful” episode of “Black Mirror” showed, these tools also put people’s jobs at risk. The latter threat is real, and the concerns being raised by entertainment union members are legitimate. I’m not highlighting these generative AI technologies here in order to hype them but to help people to understand them.

Alright, having said all that, on to the guide.

Text generation

This is the most basic and, I’ll say, least creatively interesting of the generative AI applications. Ask an AI tool to come up with an idea for a video and then write out the character dialogue. The tool can also be prompted to produce a shot list, which can include writing out image prompts for another generative AI tool to produce the actual images, or to spit out SEO-friendly YouTube title ideas. The Writers Guild of America has reason to be concerned about this technology’s potential threat to their livelihoods, but at the moment these tools are more akin to a writing assistant than a writer in their own right.

Uses: Writing scripts, developing shot lists, designing image prompts, brainstorming YouTube video titles

Limitations: Skews toward the generic, limited sense of humor

Example tools: OpenAI’s ChatGPT, Google’s Bard

Image generation

AI tools capable of producing still images are where generative AI really starts to get interesting for video purposes. Much has been made of how image-generating AI tools have been used to produce short videos in the style of famous directors like Wes Anderson. But the more practical use cases are the more subtle ones, such as producing entire backgrounds for scenes, expanding existing backgrounds and generating illustrated and photorealistic images and objects for use in graphics and YouTube thumbnails.

Uses: Generating background images, expanding and filling frames, producing individual object images, creating YouTube thumbnails

Limitations: Difficult to control for specific outputs, difficult to produce consistent images, hands 

Example tools: Midjourney, OpenAI’s DALL-E 2, Stable Diffusion, Adobe Photoshop’s Generative Fill feature

Video generation

If AI tools can generate single images, then it’s not too far of a leap for them to generate a sequential series of images to comprise a video. But it’s still a sizable leap. And while these tools are capable are producing full clips all from a string of words in a text prompt, the bigger possibilities seem to lie in the options to have the tools take an existing video clip and combine it with a separate image to produce a new video, such as taking a clip a computer mouse outfitted with bottle caps and blending it with an image of a car to create a stylized clip of a racing car.

Uses: Producing entire video clips, converting existing clips into computer-generated imagery

Limitations: Resulting clips tend to vary frame to frame, difficult to control for specific outputs, limitations on resulting video lengths

Example tools: RunwayML’s Gen-1, RunwayML’s Gen-2, Kaiber, Pika Labs

Voice generation

Voice actors’ AI anxieties are somewhat well-founded. These tools are capable of taking text and turning it into spoken word that sounds closer to an actual person than to Siri. The generated voices can be infused with different accents and inflections, such as an elderly British woman, or — red flag alert — be modeled from actual people’s voices. They can even adjust their delivery to reflect the meaning and emotion of the text. Someone listening closely will likely still pick up on the somewhat stilted delivery, though.

Uses: Producing voiceovers, creating voices for computer-generated and hand-drawn characters

Limitations: Vocal delivery can skew stilted and stiff

Example tools: Eleven Labs, Synthesia, Speechify

Character animation

All of these generative AI tools are pretty bleeding-edge technologies, but this category seems to be on the sharpest end of the spectrum. These tools can mimic the work of specialized animators to bring 2D and 3D characters to life. They are capable of animating the facial expressions of a pictured person as well as processing a video clip for motion capture data in order to replace its human subject with a computer-generated character. The tools are not anywhere near as capable as a Pixar animator, but as is the case with all these tools — and the justification to fear them — their iterations today are the worst they’ll ever be.

Uses: Animating 2D and 3D characters’ facial expressions and movements, capturing motion capture data for 2D and 3D scenes, generating alpha mattes for masking

Limitations: Random distortions in facial expressions, limited controls over expressions, motion capture processing best suited to videos filmed without camera movement, motion capture output may not conform to a consistent ground plane

Example tools: Wonder Dynamics’ Wonder Studio, D-ID’s Creative Reality Studio, MyHeritage’s Deep Nostalgia feature

What we’ve heard

“It used to be a 33-33-33 split between brand deals, YouTube ad revenue and then affiliate money. That has pivoted a little bit. YouTube is now down to maybe 10%. The revenue has definitely been cut, same for Facebook. But it’s still thousands of dollars every month.”

— YouTube creator Cassandra Bankson

TikTok vs. Instagram Reels vs. YouTube Shorts

During VidCon last month, I asked more than two dozen teens, tweens and twentysomethings how they feel about TikTok, Instagram Reels and YouTube Shorts. Here’s what they had to say.

Numbers to know

21%: Amazon Prime Video’s share of the U.S. streaming market in the second quarter of 2023, edging out Netflix by a percentage point.

43%: Percentage of survey respondents who said they currently subscribe to the maximum number of streaming services they’re willing to pay for.

0: Number of scripted TV shows that were shooting in Los Angeles in the final week of June.

>$280 billion: Decline in combined market value for Comcast, Disney, Netflix and Paramount since the end of 2020.

What we’ve covered

Buyers ask for, and get, pricing rollbacks in the upfront:

• TV network and streaming service owners are agreeing to lower ad prices in this year’s upfront deals.
• Netflix and Fox are among the sellers still yet to close their upfront market.

Read more about the TV upfront here.

Brands take their intellectual properties to Hollywood — with marketing mixes to go along with them:

• Mattel, Nike, Tetris, Nintendo and Frito-Lay are among the brands to have had movies premiered showcasing their products this year.
• Hasbro is also in on the act, though it’s been reportedly trying to sell off its film-and-TV studio Entertainment One.

Read more about branded entertainment here.

Cereal company builds millennial, Gen Z following on TikTok, Instagram as social media behavior and algorithms change:

• Magic Spoon found that product shots and videos don’t perform well on TikTok.
• The brand has pivoted to comedy skits and behind-the-scenes footage.

Read more about Magic Spoon here.

Meta’s Alvin Bowles on next steps for Reels ads:

• Bowles said Meta is “really focusing now on the monetization aspect of” Reels.
• Meta recently changed up its monetization program for Reels creators.

Read more about Reels ads here.

What we’re reading

Upfront crawls to a close:

Some TV network owners are getting close to wrapping up this year’s upfront deals, though the results are far from stellar with declines largely across the board, according to Variety.

Amazon’s studio pressure:

Amazon CEO Andy Jassy is pressing its film-and-TV studio to make a case for the programming budgets of some of its most expensive shows, the latest example of Hollywood’s austerity era, according to Bloomberg.

NBCUniversal’s leadership changes:

Interim CEO Mike Cavanagh has rejigged the Comcast-owned media conglomerate by expanding the purviews of film boss Donna Langley and TV and streaming chair Mark Lazarus, with TV and streaming exec Susan Rovner will depart, according to The New York Times.

Influencer marketing’s new disclosure requirements:

The U.S. Federal Trade Commission’s updated endorsement guidelines will effectively require video creators to disclose sponsorships in the video (as opposed to writing “#ad” in the post’s caption), according to The Wall Street Journal.

Streamers’ adversarial advertising:

Disney’s aversion to letting rival streamers advertise to its audience has been adopted by Netflix and Warner Bros. Discovery’s Max, though NBCUniversal’s Peacock and, to an extent, Paramount’s Paramount+ are willing to take competitors’ money, according to Variety.