Strada worldwide (hereinafter referred to as Strada) is a digital media agency that produces high quality works of digital art. To differentiate from established media, Strada strategically adopts Microsoft HoloLens and challenges to business areas such as theme park, distribution and sales. These challenges are bringing tangible results. The purposes of this Hackfest were to learn technologies are able to be utilized for theme park and distribution businesses, and to apply technologies to new businesses.

The hack team included the following:

  • JunBeom Pyo - Strada, Head of R&D Center
  • YoungWook Kim - Microsoft, Sr. Technical Evangelist
  • HyeWon Ryu - Microsoft, Audience Evangelism Manager.

Customer profile

Strada logo

Strada moves toward brand entertainment and has global competitiveness of new media advertising, solution and entertainment consulting based on the humanities and new media. With partners and local offices in 8 countries, Strada is creating diverse new media campaigns and innovative services.

Strada image

Strada received awards in two categories of the world-famous IF Design Award in 2015 and has credentials collaborated with a variety of customers such as BMW Mini, Airbnb, Tourism of Brazil, Brazilian airport, Emputur, and ans Morengo Casino.

Pain point or problem area to be addressed

Strada was running project that builds a virtual reality theme park and wanted to use HoloLens as a tool not only for providing contents but also for collecting customer’s information. It needs a work that integrates various data of surrounding sensors. However, most developers in Strada had only experience in developing client side application using technologies such as Unity3D and Android. In addition, they didn’t have experience in IoT technologies neither. Because of this problem, Strada wanted to run HackFest using Microsoft’s technologies.

Solution, steps, and delivery

System Architecture

To solve problems in Strada, sever side development has to be minimized. Second, flexible solution that handle changeable scenarios is required. So approaches as follows are taken.

  • Make the best use of server-less architecture to minimize server-side development
  • Reuse exist resources as much as possible
  • Write readable code because some of developers in Strada are not familiar with C# language.

Analyzing customer’s emotion using Microsoft Cognitive Service

Strada wanted to collect more useful information besides simple data such as temperature and moisture, so Microsoft Cognitive Service was selected. Among many useful services provided Microsoft Cognitive Service, Emotion API was great choice as a solution. Emotion API analyzes 8 indexes of person’s emotion in a photo and provide a result in JSON format. An expected result with this API was collecting objective data easily by analyzing customer’s facial expression without survey.

Device Programming (UWP and Cognitive Service)

The selected solution to utilize HoloLens was building UWP app for taking pictures, analyzing person’s emotion in the picture, and writing code sends result to IoT Hub. To analyze person’s emotion, Microsoft Cognitivie Service (https://www.microsoft.com/cognitive-services) was used.

Building Camera app

First, build an app based on UWP for taking pictures. The UWP-based app can be deployed widely not only on Desktop or Windows Phone but also on IoT devices such as XBox, Rasberry Pi and on HoloLens. In the Visual Studio 2015, to create an empty project, navigate to Visual C# | Windows | Blank App(Universal Windows) Create Project

Add namespaces that are related camera feature to MainPage.xaml.cs.

using Windows.Media.Capture;
using Windows.Storage;
using Windows.Storage.Streams;
using Windows.Graphics.Imaging;
using Windows.UI.Xaml.Media.Imaging;

Insert the following code snippet that creates UI to MainPage.xaml

        <StackPanel>
            <Image x:Name="imageControl" Width="400" Height="400"/>
            <Button Name="btnTakeaPhoto" Content="Take a Photo" HorizontalAlignment="Center" Click="btnTakeaPhoto_Click"/>
            <TextBlock Name="txtStatus" Text="........"/>
        </StackPanel>

To add camera feature, you should use CameraCaptureUI.

CameraCaptureUI captureUI = new CameraCaptureUI();
captureUI.PhotoSettings.Format = CameraCaptureUIPhotoFormat.Jpeg;
captureUI.PhotoSettings.CroppedSizeInPixels = new Size(400, 400);
StorageFile photo = await captureUI.CaptureFileAsync(CameraCaptureUIMode.Photo);

if (photo == null)
{
    // User cancelled photo capture
    return;
}

After all this is done, you can take pictures.

Create Project

During creating project, you should face the following screen that requires to select UWP version. Leave those settings as default. Then, add a function that generates filenames for storing taken pictures.

private string CreateTimeToFileName()
{
    string fileName;
    fileName = DateTime.Now.ToString("yyyyMMddHHmmss") + ".jpg";

    return fileName;
}

Create bitmap source from file stream and connect it to image control. Now, you can see the result of taking pictures.

string fileName = CreateTimeToFileName();
StorageFolder destinationFolder = await ApplicationData.Current.LocalFolder.CreateFolderAsync("ProfilePhotoFolder", CreationCollisionOption.OpenIfExists);
            await photo.CopyAsync(destinationFolder, fileName, NameCollisionOption.ReplaceExisting);

IRandomAccessStream stream = await photo.OpenAsync(FileAccessMode.Read);
BitmapDecoder decoder = await BitmapDecoder.CreateAsync(stream);
SoftwareBitmap softwareBitmap = await decoder.GetSoftwareBitmapAsync();
SoftwareBitmap softwareBitmapBGR8 = SoftwareBitmap.Convert(softwareBitmap, BitmapPixelFormat.Bgra8, BitmapAlphaMode.Premultiplied);
SoftwareBitmapSource bitmapSource = new SoftwareBitmapSource();
await bitmapSource.SetBitmapAsync(softwareBitmapBGR8);
imageControl.Source = bitmapSource;

Connecting with Cognitive Service

You should check the API key before calling Microsoft Cognitive Service. Related information is provided on Cognitive Service website.

Cognitive Service

After checking the API key, assign it to a member variable.

private const string CognitiveApiKey = "{Cognitive Emotion API Key}" 

Cognitive Service consists entirely of Web APIs. Add the following namespace for calling Web APIs in UWP.

using Windows.Web.Http;

Now add following code that calls Cognitive Service and analyzes facial expressions to the latter part of btnTakeaPhoto_Click().

HttpStreamContent streamContent = new HttpStreamContent(stream);
streamContent.Headers.ContentType = new Windows.Web.Http.Headers.HttpMediaTypeHeaderValue("application/octet-stream");

Uri apiEndPoint = new Uri("https://api.projectoxford.ai/emotion/v1.0/recognize");
HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Post, apiEndPoint);
request.Content = streamContent;

// Do an asynchronous POST.            
HttpClient httpClient = new HttpClient();

httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", CognitiveApiKey);
HttpResponseMessage response = await httpClient.SendRequestAsync(request).AsTask();

// Read response
string responseContent = await response.Content.ReadAsStringAsync();

Connecting with IoT Hub

The Microsoft IoT Hub allows you to reliably collect and manage data from multiple devices. Project Strada used the Microsoft IoT Hub to collect not only data from HoloLens, but also IoT Data, which will be developed in the future, into a single architecture.   For more information about the Microsoft IoT Hub, please visit https://azure.microsoft.com/en-us/services/iot-hub/.

You can establish connection with IoT Hub using provided library. To install related packages, you should use NuGet. In Solution Explorer, right-click the project and select Manage NuGet Packages … as below. NuGet

In the NuGet Package manager, search for Microsoft.Azure.Devices.Client and install the package. NuGet

Add the following namespace for using IoT Hub at the top of the Progrma.cs file

using Microsoft.Azure.Devices.Client;

IoT Hub provides connection strings for establishing connection. SIgn in to the Azure Portal, select IoT Hub just created, find key icon and select it, then policy list will be appeared. Click Iothubowner policy and then copy and make note of the Connection string-primary key on the right side.

Azure IoT Hub Setting

Unregistered device is not allowed to connect with IoT Hub. Therefore registration is necessary using Device Explorer. Device Explorer is provided at https://github.com/Azure/azure-iot-sdks/tree/master/tools/DeviceExplorer

Device Explorer

After running Device Explorer, copy connection string of iothubowner on IoT Hub and paste it to IoT Hub Connection String field in Configuration tap. When the connection is established, select Management tap and click Create button. Type Device1 to Device ID field as below.

registration

Assign member variables in Program.cs file as below. Replace placeholder value with the IoT Hub connection string that you copied.

private const string DeviceConnectionString = "< Replace with the connection string of  IoT Hub!>";
private const string DeviceID = "Device1";

private static DeviceClient SensorDevice = null;

//Then, modify code of Main() as below.

static void Main(string[] args)
{
    SetTimer();

    SensorDevice = DeviceClient.CreateFromConnectionString(DeviceConnectionString, "Device1");
}

Add a function that sends actual data.

private async void SendMessage(string message)
{
    //string json = Newtonsoft.Json.JsonConvert.SerializeObject(Sensor.GetWetherData(DeviceID));
    Message eventMessage = new Message(Encoding.UTF8.GetBytes(message));
            await SensorDevice.SendEventAsync(eventMessage);
}

At last, add code that calls the function for sending data to the latter part of btnTakeaPhoto_Click().

SendMessage(responseContent);
await photo.DeleteAsync();

The emotion information entered through Microsoft HoloLens comes in JSON type.

[
  {
    "faceRectangle": {
      "left": 900,
      "top": 356,
      "width": 69,
      "height": 69
    },
    "scores": {
      "anger": 1.61575272e-7,
      "contempt": 0.000004247503,
      "disgust": 2.36979556e-8,
      "fear": 6.178071e-8,
      "happiness": 0.9946751,
      "neutral": 0.00531783467,
      "sadness": 0.00000216178728,
      "surprise": 4.332052e-7
    }
  }
]

Use Stream Analytics to analyze JSON with emotion information in real time. All emotion information is stored in Azure Storage. The 5-minute average is also stored in SQL Azure.

Hackfest

Hackfest

Stream Anaytics Query

SELECT
    *
INTO
    OUTPUT
FROM
    INPUT
    
SELECT
     DeviceID, AVG(anger)as Anger, 
     AVG(contempt) as Contempt, 
     AVG(disgust) as Disgust
     AVG(fear) as Fear
     AVG(happiness) as Happiness
     AVG(neutral) as Neutral
     AVG(disgust) as Disgust
     AVG(sadness) as Sadness
     AVG(surprise) as Surprise
INTO
    OUTPUTS
FROM
    INPUT
GROUP BY DeviceID, TumblingWindow(Minute, 1)

Conclusion

Hackfest with Strada was proceeded through three business meetings.

19 Oct 2016, Analyzing problems and designing architecture 26 OCt 2016 analyzing requisite technologies and doing prior studies about technologies used in implementation 8 Nov 2016, the day’s training and coding

Hackfest

The agreed architecture design

architecture
architecture
architecture

Through this Hackfest, Strada learned how to utilize Cognitive Service and and how to collect IoT data using IoT Hub together. These knowledge can be applied various IoT projects and moreover will be helpful to construct services with minimal effort. In addition, Microsoft Azure is considered as a solution more actively and server-less architectures are rated highly.