At CES, new AI tools for TVs offer new features for both content and commerce

Earlier this week, Telly — a startup that gives people a free 4K TV in exchange for more ads — debuted a new voice AI assistant called “Hey Telly.” Built with OpenAI’s large language model, the chatbot helps operate the TV, chat with viewers and behave in ways similar to ChatGPT. With three possible characters to start — and more to come — the chatbot can also provide personalized recommendations based on who’s watching. It’s also exploring ways of using generative AI platforms like Midjourney and DALL-E to let people make generative AI images for TV screensavers.

Besides using OpenAI’s LLMs, Telly also employs others for various tasks. For example, it uses a separate voice engine that can translate speech to text and then use that raw text for various AI models depending on uses.

“We have brought in the models, all the processing, that we see fit for the use case,” said Telly chief product officer Sascha Prueter. 

Beyond startups, giants like Samsung, LG and Hisense are all adding AI to new TVs to improve picture and sound quality. For example, some use visual recognition to optimize the screen based on the content someone is watching or playing. Others are using AI to optimize audio by analyzing a room’s background noise and voices.

TV manufacturers also hope new AI features will make screens more useful beyond entertainment — options include assisting people with workouts, providing telehealth tools, and becoming a hub for controlling other smart home devices. One end goal is to potentially replace other smart screens from companies like Amazon and Google by making the TV a central hub for everything in the home.

Some marketers who attended CES said they think new transparent TVs from LG and Samsung — also powered by new AI processors — offer compelling uses beyond showing viewers what’s on the wall behind their big screen. For example, the newly unveiled devices — which can operate like a normal TV or turn off to be clear as glass — might someday be used by retailers to give passersby new ways to window shop. 

AI is also enabling more options for commerce. Telly has added new AI-powered image recognition tools that show products to buy depending on what’s on the screen. In a demo for Digiday during CES, a televised basketball game showcased products worn by players that can then be bought directly through Telly. Advertisers will also be able to sponsor product ads to show alongside other unpaid personalized recommendations.

Telly isn’t the only TV startup thinking about AI and e-commerce. Another on the showroom floor at CES was Displace, which initially debuted a wireless TV at CES 2023. This year, the startup’s added a way to use hand gestures — which the TV recognizes with AI — to tell the TV to pause the show, analyze the screen for various products before Displace suggests similar products to buy.

Unlike Telly, Displace doesn’t plan to run sponsored product ads, but one of its TVs does have an NFC reader so people can tap their preferred method of payment to buy something on Amazon or another platform. (Displace is also using OpenAI’s technology along with various other sources.) Displace’s new features also hope to ultimately provide recipes and show where to buy ingredients based on a cooking show someone is watching in their kitchen.

“We are building a TV platform,” said Displace founder and CEO Balaji Krishnan. “This is exactly what happened with smartphones. Before smartphones, everything was SMS-based messaging. And even payments — PayPal started with email, but now everything is integrated into the phone because companies like Apple and Google created a platform … We are trying to create a contextual thing for the TV set.”

Of course, the AI used in new cameras and voice tools also creates new privacy concerns: How do they make sure their faces, conversations, credit card data and other information aren’t leaked by one of the LLMs that process them? Prueter also said Telly’s survey data is “highly confidential” and won’t be fed back into ChatGPT or other public AI models.

To offer a sense of privacy, Telly’s camera has a shutter in front and requires permission before taking a photo, but other sensors know when people are watching and how many are in the room. On Displace’s devices, the camera pops up and can get closed into the screen whenever they don’t want to use it.

Although ads are part of Telly’s pitch — people need to fill out a lengthy survey with more than 100 questions that helps to personalize features and advertising — that’s not the case for Displace. When asked about whether Displace plans to offer retargeting for advertisers, Krishnan said the company “will never sell data” because ads “aren’t our thing.”

“What we are enabling is the advertisers now can actually pay their ads in a way that is interactive and transactional,” Krishnan said. “Because they’re already playing ads….Like if you look at the Super Bowl ads, they’re millions of dollars worth of ads, but they’re not interactive.”