Nippon Systemware Co.,Ltd. has large amounts of data from devices. But we haven’t been able to take advantage of it. So we would like to gain an understanding of Azure IoT and analysis-related services and use technical verification to determine how they match up as NSW IoT solutions. Conditions of rooms (factories, conference rooms, etc.) that change depending on weather, temperature, humidity, and clothes greatly influence the employees working and performance and customer satisfaction. In this project, they propose to set the optimum temperature from the climate, temperature and the facial expression of employees working in and out of the room and to monitoring a productivity.

Key technologies

  • IoT Hub
  • Azure Functions
  • Machine Learning
  • Power BI
  • Raspberry Pi with Raspbian
  • Cognitive Services

Customer profile

Nippon Systemware Co.,Ltd. is an independent systems integrator in Japan. We focus on the IT solutions business and product solutions business. In our IT solutions business, we deploy operational and infrastructure solutions and service businesses, and in our product solutions business, we develop embedded applications, platforms, and deploy LSI/FPGA businesses. Over the last several years, we have used the unique characteristics of our businesses to focus on the IoT business.

People

Problem statement

The development of IoT requires many proofs and it is necessary to start up quickly at low cost, but development and operation based on on-premises and virtual machines takes more time and cost. In addition, there are a lot of opportunities for us in not only Japan but also overseas. Therefore, by using full of Microsoft Azure as PaaS, it will set up early service, reduce operating costs, expand our business globally. In addition, we make enhancements providing data analytics with Azure Data Analytics Services.

Solution and steps

Propose optimal temperature indoors from indoor humidity, face photograph (emotion, age, gender, etc), weather, maximum temperature, and lowest temperature data. The solution obtains information such as emotion, age, gender and etc. by sending various data to the IoT Hub from the device sensor and the external weather site API and calling the Face API and Emotion API of Cognitive Service. Azure Machine Learning judges whether the current indoor temperature is appropriate or not by using the acquired data as an argument. It is possible to improve employee’s work efficiency and customer’s satisfaction by adjusting the temperature based on the judgment result. Arch

Technical delivery

This application get the data of celsius, picture, pressure and humidity data from BME 280 sensor on Raspberry Pi(Raspberry Pi 2 Model B). After received it, the application push the data to Azure IoT Hub using A Java SDK for connecting devices to Microsoft Azure IoT services.
Device

Device Info

  • Hardware Raspberry Pi 2 Model B https://www.raspberrypi.org/products/raspberry-pi-2-model-b/

  • 900MHz quad-core ARM Cortex-A7 CPU
  • 1GB RAM
  • 32GB Storage (micro SD-Card)
  • Power is line

  • Operating System RASPBIAN JESSIE WITH PIXEL (Linux based on Debian) https://www.raspberrypi.org/downloads/raspbian/

  • Version:January 2017
  • Release date:2017-01-11
  • Kernel version:4.4

Now let’s take a deep dive into the solution technical details. First, we initialize each device sensor and camera connected to Raspberry Pi with Raspbian. The device is to capture customer faces from camera. We added IoT Hub to the front end to serve real-time data transmission from device to cloud. To connect the device and IoT Hub, we used Azure IoT device SDK for Java that contain that facilitate building devices and applications that connect to and are managed by Azure IoT Hub. It helps us to build applications that interact with our devices and manage device identities in the IoT Hub. To upload customer faces image files, actually we wanted to use “UploadToBlobAsync” method included in Azure IoT device SDK for C#. But we were not able to find it in Azure IoT device SDK for Java. That is why we decided to upload image files to BLOB Storage directly.

This is a Java sample code to take a photo and then post data to Azure BLOB Storage from the device.

public String takePhotoAndUploadStorage() throws InterruptedException, ExecutionException {
    ExecutorService newSingleThreadExecutor = Executors.newSingleThreadExecutor();
    Future<String> futureString = newSingleThreadExecutor.submit(() -> {
        String fileName = takePhotoAndGetFileName();
        Path upFileBytes = Paths.get(fileName);
        StorageService storage = new StorageService();
        Thread.sleep(800);
        storage.uploadFile(Files.readAllBytes(upFileBytes), upFileBytes.toString());
        Files.delete(upFileBytes);

        // Call Cognitive Services from took the photos
        String url = AZURE_STORAGE_BLOB_UPLOAD_URL + fileName;
        return url;
    });
    return futureString.get();
}


private String takePhotoAndGetFileName() {
    piCamera.setAWB(AWB.AUTO); // Change Automatic White Balance setting to automatic
    piCamera.setDRC(DRC.OFF); // Turn off Dynamic Range Compression
    piCamera.setISO(800);
    piCamera.setWidth(512);
    piCamera.setHeight(384);
    piCamera.setContrast(100); // Set maximum contrast
    piCamera.setSharpness(100); // Set maximum sharpness
    piCamera.setQuality(75); // Set maximum quality
    piCamera.setTimeout(1000); // Wait 1 second to take the image
    piCamera.turnOnPreview(); // Turn on image preview
    piCamera.setEncoding(Encoding.PNG); // Change encoding of images to PNG
    try {
        //Create new file name
        String currentDateTime = LocalDateTime.now().format(DateTimeFormatter.ofPattern("yyyy-MM-dd-HH-mm-ss-SSS"));
        UUID uuid = UUID.randomUUID();
        String fileName = currentDateTime + "_" + uuid.toString() + ".png";

        File takeStill = piCamera.takeStill(fileName, 512, 384);
        return takeStill.getAbsolutePath();
    } catch (IOException | InterruptedException ex) {
        LOGGER.log(Level.SEVERE, null, ex);
        return "";
    }
}  

It also configures the Azure Functions and Power BI Stream API on Azure separately. As a result, we can confirm the data on Power BI dashboard. At first, we were supposed to developing application with Stream analytics instead of Azure Functions. In this solution, one device push the data to Azure IoT hub per a minute which means we don’t need scalability like Stream Analytics. That is why we have decided to use Azure Functions after discussion. Azure Functions calls Cognitive Services API (Emotion API). Once the API is called, the API returns customer emotions that are needed in the application. If there are multiple people in a image, we average emotions for now. If there is no people in the image, we decided to ignore the data. And then we called Azure Machine learning API to predict optimum temperature from the climate. We made most appropriate learning model in advance. Finally, we created a Power BI dashboard. Because there is a Power BI Streaming API to serve real-time data transmission showing overall data gathered and processed by Azure Functions.

This is a C# sample code to post data to Power BI Stream API on Azure Functions.

// Send POST request to the push URL
WebRequest request = WebRequest.Create(realTimePushURL);
request.Method = "POST";
string postData = String.Format("[{ { \"currentTime\": \"{0}\", \"scoreLabel\":{1} } }]", currentTime, currentValue);
                    
// Prepare request for sending
byte[] byteArray = Encoding.UTF8.GetBytes(postData);
request.ContentLength = byteArray.Length;

// Get the request stream.
Stream dataStream = request.GetRequestStream();

// Write the data to the request stream.
dataStream.Write(byteArray, 0, byteArray.Length);

// Close the Stream object.
dataStream.Close();

// Get the response.
WebResponse response = request.GetResponse();

// Display the status.
Console.WriteLine(String.Format("Service response: {0}", ((HttpWebResponse)response).StatusCode));

// Get the stream containing content returned by the server.
dataStream = response.GetResponseStream();

// Open the stream using a StreamReader for easy access.
StreamReader reader = new StreamReader(dataStream);

// Read the content.
string responseFromServer = reader.ReadToEnd();

// Clean up the streams.
reader.Close();
dataStream.Close();
response.Close();

This is a C# sample code to call Emotion API on Azure Functions.

public static void Run(string myEventHubMessage, TraceWriter log)
{
    log.Info($"C# Event Hub trigger function processed a message: {myEventHubMessage}");

    string subscriptionKey = "[Cognitive Services subscriptionKey]";

    var list = ParseJson(myEventHubMessage);

    foreach (var input in list)
    {
        // call Emotion API
        UploadedPhotoStatus result = CallCognitiveAPI(subscriptionKey, input.Url).Result;
    }
}

// Deserialize  myEventHubMessage
public static List<SensorModel> ParseJson(string jsondata)
{
    var list = Newtonsoft.Json.JsonConvert.DeserializeObject<List<SensorModel>>(jsondata);

    return list;
}

public static async Task<UploadedPhotoStatus> CallCognitiveAPI(string subscriptionKey, string bloburi)
{
    CloudBlockBlob myBlob = new CloudBlockBlob(new Uri(bloburi));
    var emotionSC = new EmotionServiceClient(subscriptionKey);
    UploadedPhotoStatus uploadedStatus = new UploadedPhotoStatus();
    Emotion[] emotionsResult = await GetEmotionsResult(myBlob, emotionSC);

    try
    {
        double angerTotal = 0;
        double contemptTotal = 0;
        double disgustTotal = 0;
        double fearTotal = 0;
        double happinessTotal = 0;
        double neutralTotal = 0;
        double sadnessTotal = 0;
        double supriseTotal = 0;

        int numOfPerson = emotionsResult.Length;

        if (emotionsResult.Count() > 0)
        {
            foreach (var em in emotionsResult)
            {
                angerTotal += em.Scores.Anger;
                contemptTotal += em.Scores.Contempt;
                disgustTotal += em.Scores.Disgust;
                fearTotal += em.Scores.Fear;
                happinessTotal += em.Scores.Happiness;
                neutralTotal += em.Scores.Neutral;
                sadnessTotal += em.Scores.Sadness;
                supriseTotal += em.Scores.Surprise;
            }

            uploadedStatus.Anger = angerTotal / numOfPerson;
            uploadedStatus.Contempt = contemptTotal / numOfPerson;
            uploadedStatus.Disgust = disgustTotal / numOfPerson;
            uploadedStatus.Fear = fearTotal / numOfPerson;
            uploadedStatus.Happiness = happinessTotal / numOfPerson;
            uploadedStatus.Neutral = neutralTotal / numOfPerson;
            uploadedStatus.Sadness = sadnessTotal / numOfPerson;
            uploadedStatus.Suprise = supriseTotal / numOfPerson;
        }

        return uploadedStatus;

    }
    catch (Exception e)
    {
        return uploadedStatus;
    }
}

// Call Emotion API
public static async Task<Emotion[]> GetEmotionsResult(CloudBlockBlob myBlob, EmotionServiceClient emotionSC)
{
    Emotion[] emotionsResult = null;

    using (var memoryStream = new MemoryStream())
    {
        await myBlob.DownloadToStreamAsync(memoryStream);
        memoryStream.Seek(0, SeekOrigin.Begin);
        emotionsResult = await emotionSC.RecognizeAsync(memoryStream);
    }
    return emotionsResult;
}

public class UploadedPhotoStatus
{
    public int NumOfPerson { get; set; }
    public double Disgust { get; set; }
    public double Anger { get; set; }
    public double Contempt { get; set; }
    public double Fear { get; set; }
    public double Happiness { get; set; }
    public double Neutral { get; set; }
    public double Sadness { get; set; }
    public double Suprise { get; set; }
}

[JsonObject("sensor")]
public class SensorModel
{
    public string Date { get; set; }
    public double Temperature { get; set; }
    public double Humidity { get; set; }
    public double Max { get; set; }
    public double Min { get; set; }
    public string Url { get; set; }
}

In terms of security, we used IoT Hub access token. In the future, we will use X.509 certificates. https://docs.microsoft.com/en-us/azure/iot-hub/iot-hub-devguide-security

At first, we can not create our own Azure Functions that integrate with IoT Hub. This article was very useful at that time. A Step-by-Step Guide to Creating Functions Within Azure IoT Hub
http://www.10thmagnitude.com/tech-blog/step-step-guide-creating-functions-within-azures-iot-hub/

Conclusion

This Propose optimal temperature indoors which can notify optimal temperature using Machine Learning worked well in spite of our first solution on Microsoft Azure. We believe it can set up early service, reduce operating costs, expand our business globally. We suppose Azure IoT Hub is very useful to develop IoT applications easily and agilely. In terms on-premises, there is Azure IoT Gate SDK to consolidate much of data produced by assembly line sensors into concise information about the production cycle, even without an internet connection. It should be very useful for Japanese manufacturing scenarios. We would like to try next time. We will keep moving forward leveraging Azure Platform not only this solution but also another solutions. Particularly with scalable solutions, we leverage Azure Stream Analytics.

Additional resources

  • GitHub repos
    https://github.com/daisukei/EventHubTriggerCallEmotionAPI
    https://github.com/daisukei/PBIStreamFuncApp