Sharp Corporation concept new solution using their large screen devices “Big Pad”, which assists and accrelates business meetings. Using Cognitive Services Speech Recognition API, they challanged to capture meetings analyzing what was talked.

  • Key technologies used Cognitive Services

  • Core Team: Names, roles and Twitter handles Tadashi Motoyama - Project Owner, Sharp Corporation Daisuke Yamashita - Software Engeneer, Sharp Corporation Akihiro Kumata - Software Engeneer,Sharp Corporation Yuuki Iwamoto - Software Engeneer,Sharp Corporation Ayako Omori - DX Technology Evangelist, Microsoft Yoshio Terada - DX Technology Evangelist, Microsoft

Customer profile

  • Company name and URL Sharp Corporation http://www.sharp-world.com/corporate/info/ci/index.html

  • Company description Sharp Corporation is one of leading manufactures and a Japanese multinational corporation that designs and manufactures electronic products. Sharp Corporation was the tenth largest of television manufacture in the world before as leading company, however, Sharp Corporation has been an integral part of Taiwan-based Foxconn Group in 2016 to expand their mobile phone business.

  • Company location Osaka, Japan (HQ)

  • What are their product/service offerings? Sharp has signage products (Android OS devices) which can show static and dynamic contents. They would like to collect customers’ information from their behavior in retail stores which their products used as advertising, and develop solution to analyze those data to gain marketing insights and/or new business opportunity.

Problem statement

Sharp has been exploring solutions to measure ad effectiveness using their signage products. They decided to start to collect ad viewers’ data from photo/video which can be taken build-in or connected camera in 1 to few seconds and to detect #of viewers (accumulated and unique) and attributes which can be analyzed by time and ad contents.

Solution and steps

Using Cognitive Services Face API and Emotion API, now Sharp can detect signage viewers (defining unique or repeater) and their emotions in score. Now they can collect data to analyze viewers’ attributes (age, gender) with positive/negative reaction by time. Also, they store ad contents data shown with time and device data, so now viewer analytics by contents available, such as marketing A/B test.

System Architecture

Technical delivery

  • Cognitive Services APIs Cognitive Services Face API & Emotion API

  • Additional Technology integration: Azure IoT Hub Azure Blob & Table Storage Azure Functions Power BI Desktop

  • SDKs used, languages, etc. Cognitive Services Face API & Emotion API SDKs Azure IoT SDK for Java https://github.com/Azure/azure-iot-sdk-java

  • Learnings from the Microsoft team and the customer team

  1. Limitations & Issues
    • Need Azure IoT SDK for Java to post data(photo) from Android devices direct. (Tunneled by windows PC during Hackfest)
    • Face API cannot detect same person (and add different Face ID) in last frame photo (taken in few second before), which may show different side of face. Face API be better to detect as same person even in such case and add same Face ID without using Find Similar query.
    • Face API fails (40%, returns 400 response) 1 call per second with S1 Plan, calling from Azure Functions (in Japan East)
  2. Better processing architecture ideas
    • For reporting, use SQL Database to configure and store report-demand data.
    • Use video and detect face by Azure Media Analytics Face Detector before call Face API

Conclusion

Signage and its’ add effectiveness can be visible by capturing and analyzing watching user faces data. Cognitive Services Face API and Emotion API are very powerful since those quantify human attibutes and emotion at a time, and should be more useful if those share Face ID and conbine those data with no work. The solution this project challgenged is analyze ad effectiveness and generate just-in-time reports. Futher development, it will be conbined to signage ad controll system that reflects vewers’ results.