Cognitive Services Computer Vision API powers Teamstream Video Search

Teamstream and its clients have a problem understanding the content inside their video libraries—a problem not just for them, but for everyone who uses traditional methods of archiving and retrieving videos. A video is typically created by a team, indifferently tagged, and saved out to a file in some repository—a file server or a cloud-based shared service like YouTube. In both cases, the videos are stored in a folder structure which is organized by client and by project. This type of methodology is extremely limiting when searching for particular videos. What is the job number? Which client was that? Did we do similar videos? Is this the right version? The difficulty multiplies when you want to search for specific content within videos. If a video is not tagged or named correctly—which is often the case—it can be impossible to find what you’re looking for. Teamstream Productions was looking for a “fix” that offered deep searching capabilities within a set of videos, a fix that would be accessible through multiple channels so that if the team is on the go or giving a presentation, they can access this search through mobile or web.

Technology Services Used

 

Customer profile

logo

Teamstream Productions is a content production and marketing firm that services some of the biggest consumer brands in the world. They work to understand each client’s business and communication goals, then use their expertise to create, publish, and promote content that services those goals across all channels, including videos, web and mobile content, social promotion, traditional print, events, and more.

Solution, Development, and Delivery

Solution

We believe Cognitive Services API is a great choice for us on this initiative for two reasons: First, it offers an already robust set of search parameters that can be utilized to sort videos. Second, its our understanding that the capabilities set is constantly expanding, which is a powerful differentiating factor as we offer this solution as an ongoing SaaS offering to clients—they’re signing on for a service that will only grow richer and better with time. – Keith Blanchard CEO Teamstream Productions

Microsoft Azure Search would sit on top of this data and return the results to a traditional web search but more importantly to a Bot Framework search that can be used via the Web Chat, text messaging and with Slack and Microsoft Teams.

Admin Architecture

There are many parts to this project but in essence it is broken down into 2 parts. The administrative tasks such as uploading the videos and then client side for searching the videos and the client side to search for the videos.

arch-diagram-admin

For the administration side is where most of the heavy lifting will take place.

  • ASP.NET MVC Web Application and Azure Web App Service to provide the tasks of uploading the videos to Azure Storage.
  • Azure Function with a Blob Trigger to start the video “slicing” and moving those images to another Azure Blob Storage.
  • A second Azure Function with a Blob Trigger will take each of the new images and call the* Cognitive Service Computer Vision API* to get the tags and descriptions for each of the frame images.
  • Azure MS SQL Database for storing all of the data.

Admin Development

ASP.NET MVC Web Application and Web App Service

We first created out Web App Service out on Azure for our ASP.NET Web Application which will be the main integration point for the Admin tasks as well as the Bot. Then we created our ASP.NET Web Application project in Visual Studio 2017.

Our first step was to build out the views and handle the form data. We wanted to use a lightweight and fast object mapper for .NET so we chose Dapper. This will extend our IDConnection interface and allow for quick and easy access to our DB. There is a NuGet libary that makes it easy to add by issuing the following command

We then used a Repository Pattern for our DB interaction. In order to help with the Object Relational Mapping we used

Then in our Admin controller constructor we can instantiate our repository and send the data from the form to our database.

Now that data is being stored we can update our UploadAsync method to include sending the video to Azure Blob Storage.

First we need to make sure we add the WindowsAzure.Storage package. We can then reference the libraries needed.

Here is the full UploadAsync method:

We now have the meta information from the user being saved into the DB along with the video moving over to Azure Blob Storage.

Admin ASP.NET MVC Web Application API

Eventually we will create Azure Functions that process these videos and images. We will need a way to send that data back to our app for storage in our Azure MS SQL Database. We will create an API endpoint that will allow those functions to send back data.

In order to add an API Controller to an existing project the following steps will be required:

WebApiConfig.cs file will need to be created in the App_Start directory to register your API Controller

In Global.asax.cs, the WebApiConfig needs to be registered with the GlobalConfiguration

Then a API controller can be created.

We will add a few more admin views but we are done with the Admin portion of the project.

Azure Functions – Setup

Once a video is stored in the video blob, we want to trigger a function to process the video. We also needed a way to determine the length of the video and a way to create images from frames or certain locations within the video. We are using FFMpeg to help with this. NReco has a C# .NET wrapper for FFMpeg as well as a C# .NET Wrapper for FFProbe. These wrappers will help with getting duration of the video along with grabbing a frame within the video.

We decided to use an Empty Web Project for publishing a .NET Class Library for our Azure Functions. This would allow us to pull in packages as needed for building out our full functions.

When starting off with an Empty Web Project for use with Azure Functions there are a few key files and command line tools that are needed.

First, install the Azure Functions Core Tools. Once installed and once you create your Empty Web Project perform the following steps:

  • Open the project properties. In the WEB tab choose START EXTERNAL PROGRAM
  • For the program path, enter the path to func.exe from the Azure Functions Core Tools. Location will be C:\Users\USERNAME\AppData\Local\Azure.Functions.Cli\1.0.0-beta.93\func.exe if you installed the tooling or installed via NPM C:\Users\USERNAME\AppData\Roaming\npm\node_modules\azure-functions-core-tools\bin\func.exe
  • For COMMAND LINE ARGUMENTS: set host start
  • For WORKING DIRECTORY, specify the root of the web project on your machine

start-action

Now when you run your project Azure Functions command line tooling will take over and you can still debug your Azure Function but it will be connected to whatever triggers you have set.

Speaking of triggers we will set those up now.

Our structure of our Azure Functions project is the following:

  • root – this will host appsettings.json
  • Utils – class library for common routines
  • Models – any db and json models
  • Functions – the actual code for our function
  • VideoTrigger – functions.json definition file our the trigger and function
  • ImageTrigger – functions.json definition file our the trigger and function
  • App_Data – our FFMpeg exe files

Lets look at appsettings.json. There are a couple of ways you can build this file. By hand. Pull it down manually via Kudo on Azure or via Azure Function Command Line Settings. You only need to pull it down by hand if you already configured your Azure Functions in the portal. That is what we did.

We first went into the Portal and searched for Azure Function App and configured it as an App Service Plan. Once we had the Azure Function created we opened up a command prompt and issued the following commands:

  • func azure login
  • func azure account list
  • func azure account set
  • func azure functionapp list
  • func azure functionapp fetch-app-settings [name]

This then downloaded the appsettings.json (encrypted) and we moved this into our project root. This file contains variables for the storage accounts that functions will use as well as which ones will be triggered when new data is contributed to the account. Be cautious since this file contains secrets so you don’t want to check that into source control.

azf

Azure Functions – Video Blob Trigger

Now that we have our project set up we need to create a definition file for our function. This will allow the Azure Function App to use a precompiled dll and the function entry point to start the job when it is triggered. We will create a file called function.json and add the following:

We added this to our VideoTrigger directory (and we will do this for our other function as well and change the entry point and pathing for the blob). Now lets build our function!

We need to add a package from NuGet Microsoft.Azure.WebJobs and any other packages that we will be needing (Storage, Threading and NReco)

Here is the video trigger.

The first step is to build out some paths, one for the actual video and the location of the tooling (FFMpeg.exe and FFProbe.exe). We then get the duration of the video. This is used to determine how many frames/images we will get per video. We arbitrarily chose these numbers but will fine tune later on. Then we loop through the video and get the thumbnail from the video at a given frame location. Once we have the image we then move that over to the other Blob storage for hosting the image.

Azure Functions – Image Blob Trigger

We will create another file called function.json and put that in the ImageTrigger directory and add the following:

The Image Trigger will now kick off when a new image is added to the blob. This function will call the Computer Vision API and get the results back in JSON and then update the MS SQL DB accordingly.

For each of the images in the Blob we make a call to the Computer Vision API and then the results are sent to API endpoint within our Admin tool application.

Client Architecture

The access to these results will be provided by a traditional search box as well as integration with the Bot Framework.

arch-diagram-client

We chose to use the Bot Framework as the interface for the search for multiple reasons:

  1. The Embedded Web Chat can serve just as good as a search box but could also direct the user into a faceted search
  2. Using the Bot Channels, we can easily extend the search to Slack and Microsoft Teams where clients tend to work and communicate about projects.
  3. Along with the channels mentioned above, access the search on the go was a key requirements and having the ability to integrate with SMS easily was also an added benefit.

Client Development

Bot Framework

Videos are being stored, images are being created and data is being saved. We now need to retrieve it. The first step is to create a View to retrieve the data we want to be indexed by the Azure Search. Then we set up our Azure Search to index our Azure SQL Database.

search-1

search-2

Now we can work on the Bot. We did a similar course of action by adding Bot Framework components to an existing application.

First we install the Bot Builder Library

Then we add the keys value pairs to the Web.config (Remember, for local testing these values can be blank.)

Now we added a MessageController class for the Bot to communicate with

Now our project is setup for building the Bot.

We want the user to be able to search for videos using the Bot and its dialogs and prompts. We decided that we would use the Bot Search Dialogs.

The documentation was easy to follow to get up and running. A few tweaks where needed to the SearchHitStyle to accommodate the way we wanted our results to look.

The code above was to style the card collection.

card-collection

We also had to modify the models, specifically the SearchHit to map to the results of our Azure Search.

Then the last modification was with the AzureSearchClient and using our search keys and index to return the proper results.

We created an IntroDialog that is invoked by the MessageController and will call the VideoSearchDialog to kick off the full dialog with the user to perform the search.

The dialog, search and mapper were all placed in a ContainerBuilder which was placed in the Global.asax.cs. A ContainerBuilder is an Autofac container based inversion pattern to help with dependency injection.

We were able to test locally using the Microsoft Bot Framework Emulator. The Bot Framework Emulator provides the ability to test your bot application locally as it would if it was being called by the Bot Framework Cloud Service.

The Bot Framework has a portal where we configured our Bot Application for publishing.

Our bot is now ready to go!

bot

Insights & Next Steps

This project will save time because video is inherently difficult to manually search and to manually tag, and this reduces the amount of time for any client to spend on both ends. The reduced timeframe to find any particular dataset from a video library (e.g., “how many of our videos from the last five years are funny and feature cars”) makes new targeted sales initiatives possible, including being able to authoritatively answer questions about the contents of a video library within the context of an ongoing presentation.

We’ve completed the first phase of the project, with Juhi Saha’s team at Microsoft. At the start of their next fiscal year we’ll be applying for a continuation grant to take this to the next level and fully develop the mobile interface and finalize the back-end with whatever functionality we can include in the scope, and create a working mobile demo we can start showing to clients (using a small subset of their own video libraries). – Keith Blanchard CEO Teamstream Productions

Lessons Learned

There was a big time blocker with the Azure Functions and running an exe. We first tried adding the exe as a site extension however, Azure Functions are a site extension in itself. Reaching out to support for NReco provided a property that we can set for the path of the FFProbe and FFMpeg files. Once we did that it worked like a charm.

Another area that we needed to spend a bit of time on was with the DB. We originally planned for using Azure Cosmos DB as our data storage but there are limitations with the size of each document which we would quickly exceed with indexing each frame of a larger video.

We also started this project a few weeks back and since we begun the Microsoft Garage and Azure Media services team released product very similar to what we are accomplishing. The service is called Video Indexer. There is a REST API and a portal that you can upload your videos and gain insight from the frames within. Our build can be adapted to use this new service by adding the REST API calls within the first Azure Function Video Trigger. We would eliminate the need for the “slicing” of the video and sending it to a second storage container and issuing calls for each of the frames. The team will be looking into this for use in an upcoming release.

Resources

Remove The Spending Limit in Azure to use the Azure Marketplace

The Azure Marketplace is your online store for software applications, developer services and data, all pre-configured for Microsoft Azure. In order to access some of these services a credit card is required to be configured within the Azure portal.

BizSpark members have a luxury of free Azure credits but at times their compute charges may exceed the limits of what their plan is. A credit card is required as well.

Here is a quick video of how to add a payment method to your subscription.

PHOTO CREDIT: George Eastman House Collection

Activate Your DreamSpark Subscription to use Azure

What is DreamSpark?

DreamSpark is a Microsoft Program that supports technical education by providing access to Microsoft software for learning, teaching and research purposes.

DreamSpark is simple: it’s all about giving students Microsoft professional-level developer and designer tools at no cost so that students can chase their dreams and create the next big breakthrough in technology – or just get a head start on their career.

DreamSpark helps educators teach the latest technologies and experiment in research. Microsoft knows that to make learning more motivating, relevant, and engaging for today’s students requires a diverse set of resources. DreamSpark gives educators the resources to ensure their classrooms always have the latest technologies to challenge, motivate, and keep students engaged in new ways.

DreamSpark is also a subscription for Academic Institutions: it provides a cost-effective way to put Microsoft developer tools, platforms and servers in labs and classrooms, and on their students’ and faculty’s computers for learning and research purposes. It reduces lab costs and teaching budgets.

How do I get it?

As a Student: simply create an account, verify your student status and download software through this website at no cost. If your school/university has a subscription, you can also get access to more software titles.

Here are the steps:

  1. Head on over to DreamSpark.comDreamspark.com
  2. Click on Sign In or Get Azure Now. step-2
  3. Make sure you have a Microsoft Account first and sign in to create your DreamSpark account.step-3a
  4. Once logged in, you can now create your DreamSpark account profile. Click ‘Continue to Verify’ to get to the verification option screen.step-3b
  5. The verification option screen provides several methods to verify that you are a student. Using your student email address (john.doe@college.edu), your School DreamSpark account (My College Name), Student Identity Card, Verification Code or documentation you can supply.step-3c
  6. Once you enter your verification method, click verify.
  7. If you are verified you can click continue to get to the “congratulations” screen.step-3
  8. Click continue to access the DreamSpark main page. From here you will be able to get to the store and Download Software.step-4
  9. You will now be presented with all of the software and service options within DreamSpark. Click on Azure to activate your Azure Subscription.step-5
  10. You will now be prompted to start the Azure account activation process.step-6a
  11. Once you click on Register Now you will be in the Azure Account creation funnel. Fill out the appropriate information and make sure you are verified before you click sign up.step-7step-8
  12. The Azure account creation takes a couple of minutes. In the mean time you can view a video of Azure or sit and wait until you get the all done message.step-9step-10
  13. You can now start managing your service and access the Azure Portal.step-11
  14. You can jump right in and start creating Web Apps!step-14
  15. Unfortunately some features are not active with the Student DreamSpark subscription, such as Virtual Machines. But that is ok. There is a lot you still can do within Azure.step-13

 

Get Your Learning Hat Ready for Microsoft Events

Get ready to learn some awesome technology and services. Ranging from Machine Learning to Disaster Recovery, we have events lined up for everyone.

letsdevthisDevelopers, new to Visual Studio 2015 or Microsoft Azure? Sign up now for a free, hands-on day of expert-led demos and labs in a city near you: http://aka.ms/letsdevthistour

Boston 9/30
New York 10/15

technetIf disaster strikes, is your recovery plan ready? Sign up now for a free, hands-on Azure virtual lab in a city near you: http://aka.ms/TechNetTour

New York 10/8

ml

Microsoft Azure Machine Learning Bootcamp

Join area technologies in this hands-on bootcamp exploring the Microsoft Azure Machine Learning service, which leverages the perfect storm of the availability of massive, on-line data sets, low cost computation in the cloud, and leading edge predictive analytics algorithms.

The event will incorporate both lectures illustrating key concepts in machine learning and opportunities to apply these concepts using the free version of Microsoft Azure Machine Learning Studio:

Machine Learning Levelset – discussion of supervised vs. unsupervised machine learning models, regression vs. classification models, and popular algorithms for each

AzureML Overview – how to use AzureML and the Machine Learning Studio to get and save data, preprocess data,, and build and publish predictive models.

R Programming Fundamentals – learn how to use R with popular packages such as stats, caret, ggplot2, and the new AzureML package to produce models hosted on the AzureML platform

Have questions about Microsoft Azure Machine Learning Bootcamp? Contact Blue Metal Architects
Boston 10/24
azurecon
AzureCon 2015 Boston Area Viewing Party
AzureCon: You’re Invited to the Microsoft Northeast District Office in Cambridge, MA for an exclusive AzureCon viewing party!

Where’s the Party?: Microsoft Northeast District Office – Cambridge, MA
(255 Main St. Cambridge)
When: Tuesday, September 29, 2015
Party Starts: 11:30 AM EDT
Viewing Begins: 12:00 PM EDT
Party Concludes: 3:30 PM EDT

Lunch & snacks will be served – register now to reserve a seat!
Prizes and give-aways too!

AzureCon is the place to join us and be first to hear what’s next for Microsoft Azure, including additional features, powerful new capabilities and practical tips from pros who eat, drink and breathe Azure.

Ready to dig in? Good news: There’s more than enough Azure insights to go around – so while you can stream from anywhere, connect with your community and peers and watch with us.

More brains are better than one
Experience it first hand, and get the inside scoop on all things Microsoft Azure during this one-day live virtual event. Choose from more than 50 breakout videos lead by Azure engineers and community members that demonstrate how you can do even more with the Azure.

Hear it first from the minds behind Azure
Exclusive keynotes from Scott Guthrie, Jason Zander, Bill Staples, Scott Hanselman, Mark Russinovich and other Azure engineer gurus.

Hartford Area Microsoft Events

sign-me-up-2

I wanted to share these upcoming events in the Hartford and Danbury area.

Building Websites in the Cloud

Learn how to use Microsoft Azure, our global cloud solution, to power your websites

This workshop will show you how to quickly create web applications using popular frameworks such as WordPress, Joomla, NodeJS, PHP and more. You will learn how to create and deploy a website using Visual Studio Online. Other features such as cloud services will be discussed how to take your website to the next level.

Time: Monday, March 16, 2015 from 6:00pm to 8:00pm
Location: Microsoft Store, 211 Westfarms Mall, Farmington CT (located on the upper level between Center Court and Macy’s)

Time: Wednesday, April 1, 2015 from 6:00pm to 8:00pm
Location: Microsoft Store, Danbury Fair Mall, Danbury CT

btn-register

These workshops  are “hands-on” optional. You can attend and learn without participating in the labs. The labs will be provided at the end of the workshop. If you are particpating in the labs you will need the following:

 

Microsoft Cloud Dev Camp

Master the new Microsoft Azure features and services to build, deploy and move apps to the cloud

Roll up your sleeves and get ready to master the latest Microsoft Azure development tools and technologies. Join us at an expert-led Microsoft Cloud DevCamp and leave with code running in the cloud! DevCamps are fun, FREE events for developers, by developers. That means no fluff or filler – just valuable coding skills you can immediately put into action.
More information here.

Time: Thursday, March 26, 2015 from 9:00am to 5:00pm
Location: Microsoft’s Office, Hartford at 280 Trumbull Street, Hartford CT

btn-register

The Cloud Dev camp is intended to be “hands-on”. The following is required for attendance:

Note – you MUST bring a laptop to participate in this event. In addition:

 

Cloud DevCamps

sign-me-upJoin us for a FREE hands-on day of Microsoft Azure demos and labs in a city near you!

Roll up your sleeves and get ready to learn the latest Microsoft Azure dev tools and technologies. Join Microsoft experts at your local Microsoft Cloud DevCamp and leave with code running in the cloud!

From your first cup of coffee until we call it a day, you’ll learn how to use new Microsoft Azure features and services to build and move a variety of applications to the cloud – including websites, mobile apps, and enterprise-class applications.

Whether you use Linux, Java, Node.js, Python, Ruby or just about anything else, we speak your language. With Microsoft Azure, you can use practically any language, framework or tool to build or move existing applications to the cloud.

Don’t miss the live Cloud DevCamp in your local area. Save your seat today!

reg-today

We’ll also give away a $100 Microsoft Store gift card* to one lucky attendee at each event!

See All Events
Sessions run from 9:00am – 5:00pm.

Attendance is limited, so save your seat today.

Microsoft WebCamp – coming to a city near you!

sign-me-up

The new release of ASP.NET and Web Tools for Visual Studio 2013 puts the power to build cutting-edge websites right at your fingertips. Even better? Once you build a beautiful, interactive site, you can easily deploy and scale it with Microsoft Azure!

Join us for a free WebCamp and learn how to elevate your coding – in just one day. WebCamps are live, no-fluff events created for developers, by developers. You’ll learn from lively, informative experts who start with the basics and ramp up to advanced topics as the day goes on. Here are some of the topics we’ll cover in these hands-on sessions:

  • Introduction to ASP.NET and Visual Studio 2013 Web Tooling
  • Building web applications using the latest ASP.NET technologies
  • Building web front ends for both desktop and mobile using the latest web standards
  • API Services for both web and devices
  • Running, improving and maintaining a site in the real world
  • Real-time communications with SignalR

Don’t miss this free opportunity to sharpen your coding skills – regardless of which languages or platforms you use. We speak them all. We’ll also give away a $100 Microsoft Store gift card to one lucky attendee at each event! You must be present to win.*

reg-today

Select a date below to register online.

Attendance is limited, so save your seat today.

Win a trip to the 2015 Game Developers Conference!

Banner-08

Ready. Set. //GameOn.

Win an all access pass and free trip to the 2015 Game Developer’s Conference!

Submit a game that runs on a Windows Platform by February 9th. We will select three teams as winners and provide them all-access passes and a travel stipend to attend GDC 2015.*

Learn more and sign up at http://aka.ms/GameOnInfo

Why should I participate?

The Game Developer Conference is the world’s largest and longest-running professionals-only games conference, where 24,000 interactive games amateurs, professionals and luminaries exchange ideas and shape the future of the industry each year. It’s your opportunity to get immersed in the world of the professional gaming industry, network, learn about the latest technologies, and create job opportunities for when you graduate. And of course, have a lot of fun.

Get started

  1. Build a game using at least one Microsoft platform (yes, desktop games count!). You can also port an existing one to the Microsoft platforms. We have tools and resources to help you do that!
  2. Fill out the submission form to enter the contest
  3. Get users to like your entry on Facebook toincrease your chances of winning. How? Once you have submitted your game to the contest, you will receive an email providing instructions on how to find a Facebook post about your submission posted by the //GameOn Contest Facebook page.

Learn more and sign up at http://aka.ms/GameOnInfo

*Offer good only in the 50 United States & D.C.” to Students enrolled in higher educational institutions..