Cognitive Services Computer Vision API powers Teamstream video search

Teamstream and its clients have a problem understanding the content inside their video libraries—a problem not just for them, but for everyone who uses traditional methods of archiving and retrieving videos. A video is typically created by a team, indifferently tagged, and saved out to a file in some repository—a file server or a cloud-based shared service like YouTube. In both cases, the videos are stored in a folder structure which is organized by client and by project. This type of methodology is extremely limiting when searching for particular videos. What is the job number? Which client was that? Did we do similar videos? Is this the right version? The difficulty multiplies when you want to search for specific content within videos. If a video is not tagged or named correctly—which is often the case—it can be impossible to find what you’re looking for. Teamstream Productions was looking for a “fix” that offered deep searching capabilities within a set of videos, a fix that would be accessible through multiple channels so that if the team is on the go or giving a presentation, they can access this search through mobile or web.

Technology Services

Customer profile

Teamstream Productions is a content production and marketing firm that services some of the biggest consumer brands in the world. They work to understand each client’s business and communication goals, then use their expertise to create, publish, and promote content that services those goals across all channels, including videos, web and mobile content, social promotion, traditional print, events, and more.

Solution, Development, and Delivery

Solution

We believe Cognitive Services API is a great choice for us on this initiative for two reasons: First, it offers an already robust set of search parameters that can be utilized to sort videos. Second, its our understanding that the capabilities set is constantly expanding, which is a powerful differentiating factor as we offer this solution as an ongoing SaaS offering to clients—they’re signing on for a service that will only grow richer and better with time. – Keith Blanchard CEO Teamstream Productions

Microsoft Azure Search would sit on top of this data and return the results to a traditional web search but more importantly to a Bot Framework search that can be used via the Web Chat, text messaging and with Slack and Microsoft Teams.

Admin Architecture

There are many parts to this project but in essence it is broken down into 2 parts. The administrative tasks such as uploading the videos and then client side for searching the videos and the client side to search for the videos.

For the administration side is where most of the heavy lifting will take place.

  • ASP.NET MVC Web Application and Azure Web App Service to provide the tasks of uploading the videos to Azure Storage.
  • Azure Function with a Blob Trigger to start the video “slicing” and moving those images to another Azure Blob Storage.
  • A second Azure Function with a Blob Trigger will take each of the new images and call the* Cognitive Service Computer Vision API* to get the tags and descriptions for each of the frame images.
  • Azure MS SQL Database for storing all of the data.

Admin Development

ASP.NET MVC Web Application and Web App Service

We first created out Web App Service out on Azure for our ASP.NET Web Application which will be the main integration point for the Admin tasks as well as the Bot. Then we created our ASP.NET Web Application project in Visual Studio 2017.

Our first step was to build out the views and handle the form data. We wanted to use a lightweight and fast object mapper for .NET so we chose Dapper. This will extend our IDConnection interface and allow for quick and easy access to our DB. There is a NuGet libary that makes it easy to add by issuing the following command

We then used a Repository Pattern for our DB interaction. In order to help with the Object Relational Mapping we used

Then in our Admin controller constructor we can instantiate our repository and send the data from the form to our database.

Now that data is being stored we can update our UploadAsync method to include sending the video to Azure Blob Storage.

First we need to make sure we add the WindowsAzure.Storage package. We can then reference the libraries needed.

Here is the full UploadAsync method:

We now have the meta information from the user being saved into the DB along with the video moving over to Azure Blob Storage.

Admin ASP.NET MVC Web Application API

Eventually we will create Azure Functions that process these videos and images. We will need a way to send that data back to our app for storage in our Azure MS SQL Database. We will create an API endpoint that will allow those functions to send back data.

In order to add an API Controller to an existing project the following steps will be required:

WebApiConfig.cs file will need to be created in the App_Start directory to register your API Controller

In Global.asax.cs, the WebApiConfig needs to be registered with the GlobalConfiguration

Then a API controller can be created.

We will add a few more admin views but we are done with the Admin portion of the project.

Azure Functions – Setup

Once a video is stored in the video blob, we want to trigger a function to process the video. We also needed a way to determine the length of the video and a way to create images from frames or certain locations within the video. We are using FFMpeg to help with this. NReco has a C# .NET wrapper for FFMpeg as well as a C# .NET Wrapper for FFProbe. These wrappers will help with getting duration of the video along with grabbing a frame within the video.

We decided to use an Empty Web Project for publishing a .NET Class Library for our Azure Functions. This would allow us to pull in packages as needed for building out our full functions.

When starting off with an Empty Web Project for use with Azure Functions there are a few key files and command line tools that are needed.

First, install the Azure Functions Core Tools. Once installed and once you create your Empty Web Project perform the following steps:

  • Open the project properties. In the WEB tab choose START EXTERNAL PROGRAM
  • For the program path, enter the path to func.exe from the Azure Functions Core Tools. Location will be C:\Users\USERNAME\AppData\Local\Azure.Functions.Cli\1.0.0-beta.93\func.exe if you installed the tooling or installed via NPM C:\Users\USERNAME\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\func.exe
  • For COMMAND LINE ARGUMENTS: set host start
  • For WORKING DIRECTORY, specify the root of the web project on your machine

Now when you run your project Azure Functions command line tooling will take over and you can still debug your Azure Function but it will be connected to whatever triggers you have set.

Speaking of triggers we will set those up now.

Our structure of our Azure Functions project is the following:

  • root – this will host appsettings.json
  • Utils – class library for common routines
  • Models – any db and json models
  • Functions – the actual code for our function
  • VideoTrigger – functions.json definition file our the trigger and function
  • ImageTrigger – functions.json definition file our the trigger and function
  • App_Data – our FFMpeg exe files

Lets look at appsettings.json. There are a couple of ways you can build this file. By hand. Pull it down manually via Kudo on Azure or via Azure Function Command Line Settings. You only need to pull it down by hand if you already configured your Azure Functions in the portal. That is what we did.

We first went into the Portal and searched for Azure Function App and configured it as an App Service Plan. Once we had the Azure Function created we opened up a command prompt and issued the following commands:

  • func azure login
  • func azure account list
  • func azure account set
  • func azure functionapp list
  • func azure functionapp fetch-app-settings [name]

This then downloaded the appsettings.json (encrypted) and we moved this into our project root. This file contains variables for the storage accounts that functions will use as well as which ones will be triggered when new data is contributed to the account. Be cautious since this file contains secrets so you don’t want to check that into source control.

Azure Functions – Video Blob Trigger

Now that we have our project set up we need to create a definition file for our function. This will allow the Azure Function App to use a precompiled dll and the function entry point to start the job when it is triggered. We will create a file called function.json and add the following:

We added this to our VideoTrigger directory (and we will do this for our other function as well and change the entry point and pathing for the blob). Now lets build our function!

We need to add a package from NuGet Microsoft.Azure.WebJobs and any other packages that we will be needing (Storage, Threading and NReco)

Here is the video trigger.

The first step is to build out some paths, one for the actual video and the location of the tooling (FFMpeg.exe and FFProbe.exe). We then get the duration of the video. This is used to determine how many frames/images we will get per video. We arbitrarily chose these numbers but will fine tune later on. Then we loop through the video and get the thumbnail from the video at a given frame location. Once we have the image we then move that over to the other Blob storage for hosting the image.

Azure Functions – Image Blob Trigger

We will create another file called function.json and put that in the ImageTrigger directory and add the following:

The Image Trigger will now kick off when a new image is added to the blob. This function will call the Computer Vision API and get the results back in JSON and then update the MS SQL DB accordingly.

For each of the images in the Blob we make a call to the Computer Vision API and then the results are sent to API endpoint within our Admin tool application.

Client Architecture

The access to these results will be provided by a traditional search box as well as integration with the Bot Framework.

We chose to use the Bot Framework as the interface for the search for multiple reasons:

  1. The Embedded Web Chat can serve just as good as a search box but could also direct the user into a faceted search
  2. Using the Bot Channels, we can easily extend the search to Slack and Microsoft Teams where clients tend to work and communicate about projects.
  3. Along with the channels mentioned above, access the search on the go was a key requirements and having the ability to integrate with SMS easily was also an added benefit.

Client Development

Bot Framework

Videos are being stored, images are being created and data is being saved. We now need to retrieve it. The first step is to create a View to retrieve the data we want to be indexed by the Azure Search. Then we set up our Azure Search to index our Azure MS SQL Database.

 

Now we can work on the Bot. We did a similar course of action by adding Bot Framework components to an existing application.

First we install the Bot Builder Library

Then we add the keys value pairs to the Web.config (Remember, for local testing these values can be blank.)

Now we added a MessageController class for the Bot to communicate with

Now our project is setup for building the Bot.

We want the user to be able to search for videos using the Bot and its dialogs and prompts. We decided that we would use the Bot Search Dialogs.

The documentation was easy to follow to get up and running. A few tweaks where needed to the SearchHitStyle to accommodate the way we wanted our results to look.

The code above was to style the card collection.

We also had to modify the models, specifically the SearchHit to map to the results of our Azure Search.

Then the last modification was with the AzureSearchClient and using our search keys and index to return the proper results.

We created an IntroDialog that is invoked by the MessageController and will call the VideoSearchDialog to kick off the full dialog with the user to perform the search.

The dialog, search and mapper were all placed in a ContainerBuilder which was placed in the Global.asax.cs. A ContainerBuilder is an Autofac container based inversion pattern to help with dependency injection.

We were able to test locally using the Microsoft Bot Framework Emulator. The Bot Framework Emulator provides the ability to test your bot application locally as it would if it was being called by the Bot Framework Cloud Service.

The Bot Framework has a portal where we configured our Bot Application for publishing.

Our bot is now ready to go!

Insights & Next Steps

This project will save time because video is inherently difficult to manually search and to manually tag, and this reduces the amount of time for any client to spend on both ends. The reduced timeframe to find any particular dataset from a video library (e.g., “how many of our videos from the last five years are funny and feature cars”) makes new targeted sales initiatives possible, including being able to authoritatively answer questions about the contents of a video library within the context of an ongoing presentation.

We’ve completed the first phase of the project, with Juhi Saha’s team at Microsoft. At the start of their next fiscal year we’ll be applying for a continuation grant to take this to the next level and fully develop the mobile interface and finalize the back-end with whatever functionality we can include in the scope, and create a working mobile demo we can start showing to clients (using a small subset of their own video libraries). – Keith Blanchard CEO Teamstream Productions

Lessons Learned

There was a big time blocker with the Azure Functions and running an exe. We first tried adding the exe as a site extension however, Azure Functions are a site extension in itself. Reaching out to support for NReco provided a property that we can set for the path of the FFProbe and FFMpeg files. Once we did that it worked like a charm.

Another area that we needed to spend a bit of time on was with the DB. We originally planned for using Azure Cosmos DB as our data storage but there are limitations with the size of each document which we would quickly exceed with indexing each frame of a larger video.

We also started this project a few weeks back and since we begun the Microsoft Garage and Azure Media services team released product very similar to what we are accomplishing. The service is called Video Indexer. There is a REST API and a portal that you can upload your videos and gain insight from the frames within. Our build can be adapted to use this new service by adding the REST API calls within the first Azure Function Video Trigger. We would eliminate the need for the “slicing” of the video and sending it to a second storage container and issuing calls for each of the frames. The team will be looking into this for use in an upcoming release.

Resources

Leave a Comment

(0 Comments)

Your email address will not be published. Required fields are marked *