Microsoft DX TE team leverages Cognitive Services helping Techsun to empower their Magic Mirror with the ability to chat and show the age, gender, emotion of the user.

Summary

Techsun needs to build a “Magic Mirror” putting in store for a more personal experience, it will detect people’s age, gender and emotion, and do natural language interaction, when VIP customers come it will automatically show their name by face recognition, like “Welcome MR/MS xx , you looks happy today…”. This solution adopts Microsoft Cognitive Services to develop all the required services of the Magic Mirror, connecting to LUIS to empower the Magic Mirror with conversation ability, and leveraging Face API and Emotion API in cognitive services to give the mirror ability to describe people’s basic profile.

Magic Mirror final demo:

Cognitive Services Magic Mirror

Key technologies

Customer profile

Techsun is an important CRM service provider in China. It has been a Microsoft Dynamics CRM Partner since 2006, and has been certified as a Dynamics CRM Gold Certified Partner since 2009. In the cloud transformation process of Microsoft, Techsun actively played a part by taking the lead in launching Dynamics CRM on Azure solutions in China, and won 2015 Microsoft China Dynamics Pioneer Partner Award. This year, they expended their business to smart devices, and built a new product line named “Magic Mirror”. It will show the basic data including weather info, time, date, and also some additional functions like recognizing the age, gender, and emotion status of the people in front of mirror and also the conversation between user and mirror. Their target customer will be shopping malls, supermarkets and other public area like airports, in which case the mirror will have additional function like showing the map to navigate people.

Add : Room 1205,Shantou Science and Technology Museum, 12 Haibin Road, Shantou City,Guangdong Province, China. 515031

For more: http://www.techsun.com/

Techsun Profile`

Solution overview

The Techsun Magic Mirror solution architecture can be represented as follows.

High Level Flow

The basic user flow of the app is shown in the following sequence:

  • Intelligent Recognition: Magic Mirror Camera-> Magic Mirror Local Application -> Face API & Emotion API -> Local Application -> Magic Mirror Screen
  • Intelligent Conversation: Magic Mirror Screen QR Code -> WeChat Backend -> LUIS API -> WeChat Backend -> Magic Mirror Local Application -> Magic Mirror Screen

Cognitive Services architecture diagram

Intelligence Recognition:

The Magic Mirror front end will show the people’s profile by calling the Face API and Emotion API from Azure Cognitive Services. This process will be automatically triggered by Face Detection on the frontend application. So when a customer shows in front of the mirror, the application will automatically recognize by using Open CV, then the camera is triggered to capture the image of the customer. Then application will send to Cognitive Services for intelligent recognition. After Recognizing, the Mirror will give the basic info of the people by showing text like” You looks like a 24 years old young man, with a happy face. We have some perfume for your type, would you like to try?” It will be configured case by case based on the scenario.

Intelligent Conversation:

The conversation part will be held in WeChat backend, the magic mirror will show a QR code, user scan the QR code then talk to the mirror through WeChat. The mirror backend will deal with the conversation by calling LUIS in Azure Cognitive Services. In this way, user can talk to the mirror by typing in WeChat showing below. The WeChat backend will transfer the conversation to Magic Mirror local application. Then the mirror will show the conversation on the screen. The reason why we use the WeChat to talk to the mirror based on the Techsun’s need on the customer info collection. When a user scan the QR code in wechat, this people’s image and the WeChat profile will be connected and store in the database, so when next time the same person appears, the Shopping Mall system will automatically detect the person by shows welcome info with his/her name on Mirror. Thus to show the VIP experience.

Technical delivery

Prerequisites

Build intelligent recognizing flow:

Set up a Cognitive Services account

  1. Before you can begin, you need to create a new Face API and Emotion API Cognitive Services account in your Azure portal.
  2. Sign in to the Azure portal.
  3. At the bottom of your services list on the left of your screen, click More services.
  4. Search for Cognitive Services and select it.
  5. Click the large + sign to add a new service, selecting Face API from the API type list.
  6. After that is created, you can navigate to the new service, and select Keys to see your keys.

Cognitive Services Face API

Import Face API SDK to your code (Use Android SDK in this case) Before you can begin, you must set up a Cognitive Services account.

  1. Download Face API SDK and Import to your code file
import com.microsoft.projectoxford.face.FaceServiceClient;
import com.microsoft.projectoxford.face.contract.Face;

  1. New FaceServiceClient Instance to get the age, gender, headPose, smile, facialHair info of the Face detected from the frontend.
protected Face[] doInBackground(InputStream... params) {
        // Get an instance of face service client to detect faces in image.
        FaceServiceClient faceServiceClient = FaceServiceClientInstance.getFaceServiceClient();
        try {
            publishProgress("Detecting...");

            // Start detection.
            return faceServiceClient.detect(
                    params[0],  /* Input stream of image to detect */
                    true,       /* Whether to return face ID */
                    true,       /* Whether to return face landmarks */
                        /* Which face attributes to analyze, currently we support:
                           age,gender,headPose,smile,facialHair */
                    new FaceServiceClient.FaceAttributeType[]{
                            FaceServiceClient.FaceAttributeType.Age,
                            FaceServiceClient.FaceAttributeType.Gender,
                            FaceServiceClient.FaceAttributeType.Glasses,
                            FaceServiceClient.FaceAttributeType.Smile,
                            FaceServiceClient.FaceAttributeType.HeadPose
                    });
        } catch (Exception e) {
            mSucceed = false;
            publishProgress(e.getMessage());
            return null;
        }
    }
  1. In order to connect to the Face database, the face info needed to be stored for the future using. So the View ID, gender, age of the people will be stored.
@Override
    protected void onPostExecute(Face[] result) {
        recognizeHelper.step();

        final int fibalViewId = viewId;

        if (result == null || result.length < 1) {
            AppHelper.mHandler.post(new Runnable() {
                @Override
                public void run() {
                    recognizeHelper.step();
                    AppHelper.activity.DeleteImageView(fibalViewId);
                }
            });
        } else {
            for (Face r : result) {
                final Face rTmp = r;
                AppHelper.mHandler.post(new Runnable() {
                    @Override
                    public void run() {
                        AppHelper.activity.setFaceByTag(viewId, new RecognizeTextView.Face(rTmp.faceAttributes.gender, String.valueOf(rTmp.faceAttributes.age)));
                    }
                });

            }
        }
        AppHelper.FaceAnalysing = false;
    }
}

Build intelligent Conversation flow:

Set up a Cognitive Services account (Similar to the Face API Key create process)

  1. Before you can begin, you need to create a new Face API and Emotion API Cognitive Services account in your Azure portal.
  2. Sign in to the Azure portal.
  3. At the bottom of your services list on the left of your screen, click More services.
  4. Search for Cognitive Services and select it.
  5. Click the large + sign to add a new service, selecting LUIS API from the API type list.
  6. After that is created, you can navigate to the new service, and select Keys to see your keys.

Cognitive Services LUIS API

Connect to LUIS to do the language understanding (Use Android SDK and LUIS API in this case)

Before you can begin, you must set up a Cognitive Services account.

  1. Build the variable to store the endpoint of LUIS with key embedded
    private String endPoint = "https://westus.api.cognitive.microsoft.com/luis/v2.0/apps/LUISIdhere?subscription-key=putyourkeyhere&verbose=true&timezoneOffset=0.0&q=";
  1. Use LuisModel that have been defined to parses the JSON data,then here to use the model to get the result of the LUIS analyze result
public LUISModel Query(String keyword){
        LUISModel luisModel = null;
        try
        {
            String jsonData = Get(keyword);
            if(jsonData!=null){
                luisModel = new LUISModel(new JSONObject(jsonData));
            }
        }
        catch (Exception e){
            Log.e("err", e.getMessage());
        }
        return luisModel;
    }

3.1 LuisModel Definition, with the query , topscoringIntent, intents and entities, variables to parses the corresponding data

private String query;
private TopScoringIntent topScoringIntent;
private ArrayList<Intent> intents;
private ArrayList<Entity> entities;

3.2 Intent Definition including intent name and score shows the possibilities of the relevant intent.

class Intent {
    private String intent;

    public Intent(String intent, double score) {
        this.intent = intent;
        this.score = score;
    }

    public String getIntent() {
        return this.intent;
    }

    public void setIntent(String intent) {
        this.intent = intent;
    }

    private double score;

    public double getScore() {
        return this.score;
    }

    public void setScore(double score) {
        this.score = score;
    }
}

3.4 Entity Definition including entity name, start and end index and score shows the possibilities of the relevant entity.

class Entity {
    public Entity(String entity, String type, int startIndex, int endIndex, double score) {
        this.entity = entity;
        this.type = type;
        this.startIndex = startIndex;
        this.endIndex = endIndex;
        this.score = score;
    }

    private String entity;

    public String getEntity() {
        return this.entity;
    }

    public void setEntity(String entity) {
        this.entity = entity;
    }

    private String type;

    public String getType() {
        return this.type;
    }

    public void setType(String type) {
        this.type = type;
    }

    private int startIndex;

    public int getStartIndex() {
        return this.startIndex;
    }

    public void setStartIndex(int startIndex) {
        this.startIndex = startIndex;
    }

    private int endIndex;

    public int getEndIndex() {
        return this.endIndex;
    }

    public void setEndIndex(int endIndex) {
        this.endIndex = endIndex;
    }

    private double score;

    public double getScore() {
        return this.score;
    }

    public void setScore(double score) {
        this.score = score;
    }
}

3.3 Use LuisModel to parse the Json file, this is the whole file for processing all the topscoringIntent, Intents and Entities. We put this into LuisModel.java. For those who want to do the similar thing can just copy these code to your project.

import org.json.JSONArray;
import org.json.JSONObject;
import java.util.ArrayList;

public class LUISModel {
    private String query;

    public String getQuery() {
        return this.query;
    }

    public void setQuery(String query) {
        this.query = query;
    }

    private TopScoringIntent topScoringIntent;

    public TopScoringIntent getTopScoringIntent() {
        return this.topScoringIntent;
    }

    public void setTopScoringIntent(TopScoringIntent topScoringIntent) {
        this.topScoringIntent = topScoringIntent;
    }

    private ArrayList<Intent> intents;

    public ArrayList<Intent> getIntents() {
        return this.intents;
    }

    public void setIntents(ArrayList<Intent> intents) {
        this.intents = intents;
    }

    private ArrayList<Entity> entities;

    public ArrayList<Entity> getEntities() {
        return this.entities;
    }

    public void setEntities(ArrayList<Entity> entities) {
        this.entities = entities;
    }

    public LUISModel(JSONObject jsonObject) {

        try {
            // query
            this.query = jsonObject.getString("query");

            // topScoringIntent
            this.topScoringIntent = new TopScoringIntent(
                    jsonObject.getJSONObject("topScoringIntent").getString("intent"),
                    jsonObject.getJSONObject("topScoringIntent").getDouble("score")
            );

            // intents
            ArrayList<Intent> intents = new ArrayList<Intent>();
            JSONArray jsonIntent = jsonObject.getJSONArray("intents");
            for (int i = 0; i < jsonIntent.length(); i++) {
                JSONObject tmp = (JSONObject) jsonIntent.get(i);
                intents.add(
                        new Intent(
                                tmp.getString("intent"),
                                tmp.getDouble("score")
                        )
                );
            }
            this.intents = intents;

            // entities
            ArrayList<Entity> entities = new ArrayList<Entity>();
            JSONArray jsonEntity = jsonObject.getJSONArray("entities");
            for (int i = 0; i < jsonEntity.length(); i++) {
                JSONObject tmp = (JSONObject) jsonEntity.get(i);
                entities.add(
                        new Entity(
                                tmp.getString("entity"),
                                tmp.getString("type"),
                                tmp.getInt("startIndex"),
                                tmp.getInt("endIndex"),
                                tmp.getDouble("score")
                        )
                );
            }
            this.entities = entities;

        } catch (Exception e) {
        }
    }
}

Conclusion

The scenario of magic mirror is very attractive and useful at certain places. One of their customer in beauty and cosmetics industry has decided to deploy one mirror to their real shop, when customers go to the shop, try some products and look at the mirror, the conversation and information will be displayed. People who wants to talk with the mirror can scan the QR code on the screen and chat with the mirror through wechat, mirror will be able to talk with the user with relevant insights about the user like their emotion. At the same time, the mirror will send the user infomation to the shop CRM system, when next time the customer shows up, the mirror will able to recognize the user.

We delivered code which has been shared on github and can be reused on the integration of LUIS API and Face API to the embedded Android system, for those who want to build similar senario to allow users chatting with the mirror by natural language and empower the mirror with the function of showing users’ age, gender and emotion, can leverage the open sourse code.

Opportunity going forward: They have tried to the integration of bot framework to the embedded Android system. Allow users to asking weather, hi to the mirror and will have more topic added in the future. Also the Bing Speech API for the voice communication function in the future.

Additional resources

Code file including LUIS, Face Recognition, Speech API are available on Github for reusing : https://github.com/andrewdyhhub/CognitiveServices/tree/master/Android%20SampleCode

Team

  • Xiaosong Zheng –Dev lead, Techsun
  • Samuel Tan - Engineer, Techsun
  • Yan Zhang - Audience Evangelism Manager, Microsoft China
  • Qixiao Wang – Principle Technical Evangelist, Microsoft China
  • Leon Liang – Senior Technical Evangelist, Microsoft China
  • Yuheng Ding – Technical Evangelist, Microsoft China