@mux/ai

Hey, Phil from Mux here.

We build video infrastructure

that any developer

can leverage to build

amazing video experiences

on their website.

I'm here with a quick demo of

how you can use one of our new

features, latest thumbnails,

to analyze and moderate the

content that's coming from

your live streams using ai.

Let's take a look.

We're gonna need three

things to build this out.

First of all, we're

gonna need a live stream.

We're gonna need an account

and some API keys for

OpenAI's, API, and then we're

just gonna need a little node

strip to glue it all together.

We've got our live

stream up and running

in OBS at the moment.

This is just showing Big Buck

Bunny, which is of course,

the best movie ever made.

But what if somebody started

streaming something that

wasn't Big Buck Bunny?

What if they streamed

a violent movie or

something even worse?

Let's hook up some content

moderation so that if or when

that happens, we can catch it.

Mux has always

had an image, API.

This lets you pull

thumbnails or storyboards

or animated GIFs from

videos or live streams.

We added a new feature a

couple of weeks ago, which

allows you to request

the latest thumbnail

from a live stream.

All you need to do is build

your image URL, the same way

you always would, but pass

a new query parameter called

Latest, and set it to true.

This works just like it

always has done, but by

adding the latest parameter,

this means that every 10

seconds this thumbnail will

get updated with the latest

thumbnail from the livestream.

This is great 'cause it means

you can pretty much just pass

this straight into any odd

moderation framework you have.

It doesn't matter whether

that's OpenAI's, it

doesn't matter if that's

hives or somebody else's.

Just having this constant

URL you can use to always

get that live image just

means that updating your

moderation flow is just as

simple as adding a new job

to your moderation queue.

So for this example,

we're gonna use OpenAI's

moderation model.

What's really cool about this

API is it's actually free.

There are rate limits.

So if you're building

something significant

in scale, you might want

to keep that in mind.

We're going to use OpenAI's

latest omni moderation model.

This can take and moderate

either text or images,

and obviously we're using

it in the image mode

'cause we're gonna pass

it this latest thumbnail

from our live stream.

Okay, so let's take

a look at the code.

This whole script is less

than 50 lines of Node.js.

First thing you need to

do is just import the

OpenAI SDK, and then set

up our API keys so we can

access the OpenAI client.

We've got a function, an

async function, which just.

You pass in a playback ID and

it moderates the live stream.

The first thing this function

does is construct the URL.

It passes in a playback ID,

obviously, and sets the latest

to true, and then it sets

the width to 640 pixels wide.

Obviously, we'll then

automatically pick a height

for it based on the aspect

ratio of the content.

640 is a pretty

good balance here.

It's probably more than you

actually need for moderation,

but if you are then using

this image for other

functions like summarization

or categorization, tagging,

that sort of thing.

You might want a little

bit higher image quality,

especially if you're using a

multimodal, uh, LVM for this.

Then we pretty much just

call the OpenAI client.

We tell it to use omni

moderation latest.

That's the model we

just talked about.

We pass it the image UL,

and that's kind of it.

This is asynchronous,

so we wait for this

response and then we

take a look at response.

There is a top level boolean.

That's whether this

content was flagged or not.

This is, set to true.

If any of the categories come

back over 80% confidence.

We're looking at

two categories here.

We're looking at sexual and

we're looking at violence.

So let's take a look and

see if anyone has put

anything sinister into our

copy of Big Buck Bunny.

Okay, so we've got our

moderation script running.

It's pretty quiet right now.

Every 15 seconds.

This is requesting the

latest frame of a live

stream that's updated every

10 seconds, so you always

know that you're gonna get

a new thumbnail this way.

We can actually also

turn up the frequency.

We refresh those images.

So if you need to be

slightly market, we

can help you do that.

There's not a whole lot

going on here, so we're

gonna just use a horror

movie for this and see

what happens in our script.

Let's take a look.

So now we've flipped over to

watching a horror movie on

our live stream, and you'll

see that we're starting to

get flags back for violence.

So this is quite a gory

horror movie, uh, and you'll

see that the violence value

is coming back at 0.8, 0.9.

What happens here is you get

a confidence value from zero

to one, where one is very high

confidence that the model is

seeing violence, and zero is

very low confidence value.

So we can see, for example,

the sexual confidence is very

low, which means the content

probably isn't sexual, but

yeah, it's quite violent.

It's a horror movie.

There are monsters

doing unspeakable

things to people, so.

We're correctly detecting that

this movie is a bit violent.

You can obviously see how

if you're building out

a user generated content

platform with a live

streaming functionality,

how this is really useful.

How can you use this to

catch streams that shouldn't

be on your platform.

What's really cool is this

same approach works for a

bunch of other functionality.

I recently wrote a blog post

about how you can also use

these same latest thumbnails.

With multimodal models

and prompts to do content

summarization or content

tagging all through

using the same latest

thumbnail function.

It's not just for moderation.

You can kind of string

all sorts of AI workflows

off the back of this.

As always, I'd love to

see what you build on

top of latest thumbnails.

You can use it to

build all sorts.

This is just one way of

leveraging that feature.

And if you want more

tips and tricks and cool

things built with video.

Don't forget to like and

subscribe to our YouTube

channel, but also go and

check out Mux.com where

you can play around with

all this infrastructure.

We've also got an amazing

new free plan for getting

started building with Mux,

so check that out too.