Longfor Properties is one of Microsoft China customers and builds an intelligent virtual assitant on IT helpdesk for internal employees, implemented “Agenda Checking” and “Problem Claiming” functions, built a connection between support team with business team, leveraging the Microsoft bot framework integrating with longfor app linked through Direct Line3.0 and Microsoft Language Understanding API in Cognitive services.

Technologies Used

Core team

Special thanks to the Longfor team, the Microsoft China DX Technical Evangelist team, and the Microsoft Audience Evangelism team. Following are the participants in this project:

  • Chenglin Lu – Product Manager frosm Longfor Properties
  • Fa Jing – Engineer Lead of Longfor Properties
  • Changteng Feng – Engineer of Longfor Properties
  • Yan Zhang - Microsoft Audience Evangelism Manager
  • Leon Liang – Microsoft DX Senior Technical Evangelist
  • Yuheng Ding – Microsoft DX Technical Evangelist

Customer profile

Longfor Properties, founded in Chongqing in 1993, has built a nation-wide presence in China. As a premier developer focusing on quality products and services, Longfor Properties is currently engaged in property development, investment and management.
Longfor Properties Co. Ltd. (stock code: 960) was listed on the Hong Kong Stock Exchange in 2009. The Group now has approximately 17,000 employees and presence in 26 cities in Yangtze River Delta, wetern China, the Pan Bohai Rim, South China and Central China. It has completed and delivered over 160 projects with a total floor area of more than 54 million sq.m., and has a total land bank of approximately 39 million sq.m. The Group’s turnover amounted to over RMB88.14 billion yuan in 2016 and has built a leading position in the industry in terms of operating scale and overall strength.

To learn more about Longfor Properties, visit http://longfor.com/en/

customer profile

Problem statement

Because the size of the company keeps growing, employees always find troubles to get the direct support from the right person. Currently, it always needs a series of Emails to link to the right person, especially from the support departments. Based on this situation, they decided to develop a virtual assistant integrated to their current application named “Longxin”, which is for internal employees. The assistant need to understand natural language and be able to connect IT system and database at backend.

Solution and steps

Bot framework makes it possible to develop a virtual assistant. At the same time, with help of LUIS in cognitive services, it makes the assistant has the ability to do natural language interaction. So this project will focus on the connection of Bot with their existing IT services. At the same time, the application will use its existing user profiles to do the authentication.

Architecture Diagram

  • When user ask a question to the chat client which is integrated with Longfor App, the app will tranfer the query to bot framewrok connector Direct Line 3.0. During the data sending, the speech to text function will help do speech recognition on the client side and give bot connector the text content.
  • Bot Framewrok will handle the query from the bot connector and feedabck to the connector.
  • The feedback of bot is based on the language usderstanding result from LUIS in Cognitive Services, connected to Longfor IT System which will provide the IT data to bot to give a concrete result to user, like when user ask for agenda infomation, the agenda info will be provided

Technical delivery

Prerequisite steps

Build Bot to host conversation interaction

  • Register to bot official website

Bot Register

  • Create chat bot in Node.JS, provide app id and app secret to connector
var connector = new builder.ChatConnector({
//bot的id和密码
	appId: process.env.MICROSOFT_APP_ID,
	appPassword: process.env.MICROSOFT_APP_PASSWORD
});
var bot = new builder.UniversalBot(connector);
server.post('/api/messages', connector.listen());
  • Link to luis,untill luis get true, rest is not needed.
var model = 'https://westus.api.cognitive.Microsoft.com/luis/v2.0/apps/yourid?subscription-key=yourkey&verbose=true';
var recognizer = new builder.LuisRecognizer(model);
var dialog = new builder.IntentDialog({
	recognizers: [recognizer]
});
bot.dialog('/', dialog);
dialog.matches('意图', [
	function(session, args, next) {
		var card = new builder.HeroCard(session).title("善待你sheng");
		card.text("Your bots - wherever your users are talking.")
		card.images([
			builder.CardImage.create(session, "http://www.longfor.com/img/public/logo.png")
		]).buttons([{
				type: "url",
				value: "https://www.baidu.com/",
				title: "测试"
				image: "http://docs.botframework.com/images/demo_bot_image.png"
			},
			builder.CardAction.openUrl(session, "https://en.wikipedia.org/wiki/Space_Needle", "Wikipedia"),
			builder.CardAction.imBack(session, "select:100", "Select")
		]).tap(builder.CardAction.openUrl(session, "https://en.wikipedia.org/wiki/Space_Needle"));
		var msg = new builder.Message(session).attachments([card]);
		session.send(msg);
	}

]);

Connect to Longfor App through Direct Line 3.0

  • Constant and variable declaration
    var constant = {
    //	url:"http://114.251.168.251:8080"
      url : "http://worko.longfor.com:59650",
      key : "你的directline 密钥",
      directlineUrl : "https://directline.botframework.com"
    }
    //--------------------变量声明----------------
    //会话信息-创建会话后产生
    var conversations = {
    "conversationId": "string",
    "token": "string",
    "expires_in": 0,
    "streamUrl": "string",
    "eTag": "string"
    }
    
  • Decalare webSocket and activity, use direct line to connect
var ws = null;
//活动信息
var activity = {
	"type": "message",
	"channelId": "directline",
	"from": {
		"id": "token",//用户的id
		"name": "lucl" //当前会话的用户名
	},
	"text": ""
}
  • Build conversation connection from Client side ```js

var botCli= { //——————–初始化bot客户端———————– init : function(){ //1、单点登陆 // common.login(); // activity.from.name=loginUser.data.usercode; var para=common.getUrlParams(); //2、创建会话 this.createConversations(); //3、建立webSocket连接 ws = new WebSocket(conversations.streamUrl); }, //——————–重置客户端输入框高度———————– resetHeight:function(){ var obj=$(“#sendMsg”)[0]; if(obj.scrollHeight>34 && obj.scrollHeight<100){ $(obj).parent().height(obj.scrollHeight); $(obj).parent().parent().parent().height(14+obj.scrollHeight); } if($(obj).val()==””){ $(obj).parent().height(34); $(obj).parent().parent().parent().height(48); } console.log(obj.scrollHeight) },

- Start new conversation 
```js
    createConversations : function(){
		//开始新的会话 
		$.ajax({
				url: constant.directlineUrl + "/v3/directline/conversations",
				beforeSend: function(xhrObj) {
					// Request headers
					xhrObj.setRequestHeader("Accept",
						"application/json");
					xhrObj.setRequestHeader("Content-Type",
						"application/json; charset = utf-8");
					xhrObj
						.setRequestHeader("Authorization",
							"Bearer "+constant.key);
				},
				type: "POST",
				async : false,
			}).done(function(data) {
				//会话信息赋值
				conversations=data;
				console.log(conversations);
				//把会话id存入localStorage
				localStorage.conversationsId=conversations.conversationId;
			}).fail(function(e) {
				console.log(e);
			});
	},
  • Send Messages to bot
      //--------------------发送文件给bot-----------------------
      upload : function(){
          $.ajax(
                  {
                      url : constant.directlineUrl+"/v3/directline/conversations/"+params
                            +"/stream?t="+streamurl,
                      beforeSend : function(xhrObj) {
                          // Request headers
                          xhrObj.setRequestHeader("Accept",
                                  "application/json");
                          xhrObj.setRequestHeader("Content-Type",
                                  "application/json; charset = utf-8");
                          xhrObj
                                  .setRequestHeader("Authorization",
                                          "Bearer 8MNhtddtwrk.cwA.iHY.lblyc-cTEXIrWNx7pEKFxEtgk6jNY22SJ6pln5srN-I");
                      },
                      type : "GET",
                      data :{
                          "type": "message",
                          "from": {
                              "id": "user1"
                          },
                          "text": "明天有没有时间"
                      },
                  }).done(function(data) {
              console.log(data);
          }).fail(function(e) {
              console.log(e);
          }); 
      },
    
  • Get conversation history activity
      //--------------------获取历史会话活动-----------------------
      getActivities : function(){
          $.ajax({
                  url: constant.directlineUrl+"/v3/directline/conversations/"+localStorage.conversationsId+"/activities",
                  beforeSend: function(xhrObj) {
                      // Request headers
                      xhrObj.setRequestHeader("Accept",
                          "application/json");
                      xhrObj.setRequestHeader("Content-Type",
                          "application/json; charset = utf-8");
                      xhrObj
                      .setRequestHeader("Authorization",
                          "Bearer "+constant.key);
                  },
                  type: "GET"
              }).done(function(data) {
                  console.log(data);
                  conversations=data;
              }).fail(function(e) {
                  console.log(e);
              });
      },
      //--------------------发送文本给bot-----------------------
      postActivities : function(){
          activity.text=$("#sendMsg").val();
          htmlCreater.cbotHtmlMine(activity.text);
          $("#sendMsg").val("");
          //重置输入框高度
          this.resetHeight();
          console.log(activity);
          console.log(localStorage.conversationsId+"---------");
          $.ajax({
                  url: constant.directlineUrl+"/v3/directline/conversations/"+localStorage.conversationsId+"/activities",
                  beforeSend: function(xhrObj) {
                      // Request headers
                      xhrObj.setRequestHeader("Accept",
                          "application/json");
                      xhrObj.setRequestHeader("Content-Type",
                          "application/json;charset=UTF-8");
                      xhrObj
                      .setRequestHeader("Authorization",
                          "Bearer "+constant.key);
                  },
                  type: "POST",
                  data : JSON.stringify(activity)
              }).done(function(data) {
                  console.log(data);
              }).fail(function(e) {
                  console.log(e);
              });
      }
    }
    
  • Listen and callback methods for the connectipn between websocket and bot ```js ws.onopen = function(e){//监听连接是否打开 console.log(“Connection open…”); htmlCreater.cbotTip(new Date().Format(“MM-dd hh:ss”)+’ 创建链接成功’); htmlCreater.cbotHtmlOther(“您好,我是龙湖小智,有什么事请吩咐。”); }

ws.onmessage = function(e) {//监听是否有消息接收 if(typeof e.data === “string”){ console.log(“String message received”); if(e.data!=””){ var jsond=JSON.parse(e.data); console.log(jsond); // if(jsond.activities[0].from.name!=loginUser.data.usercode){ // htmlCreater.cbotHtmlOther(jsond.activities[0].text); // } }else{ console.log(e.data) } } else { console.log(“Other message received”, e, e.data); } }; ws.onerror = function(e){//监听是否异常产生 console.log(‘websocked error’); } ws.onclose = function(e) {//监听链接是否关闭 console.log(“Connection closed”, e); };

* Build html page to show the conversation
```js

var htmlCreater = {
	//--------------------bot方会话html-----------------------
	cbotHtmlOther : function(data){
		var html ='';
		html += '<li class="msg-content msg-user-other page-wrap">';
				html += '<div class="left">';
					html += '<img src="img/bot.png" />';
				html += '</div>';
				html += '<div class="right">';
					html += '<div class="content">';
						html += data;
							html += '<span class="icon-backwardfill"></span>';
				html += '</div></div></li>';
				
				$("#msg-list").append(html);
	},
	//--------------------发送者方会话html-----------------------
	cbotHtmlMine : function(data){
		var html ='';
		html += '<li class="msg-content msg-user-mine page-wrap wrap-end">';
				html += '<div class="left">';
					html += '<div class="content">';
							html += data;
							html += '<span class="icon-forwardfill"></span>';
						html += '</div>';
					html += '</div>';
					html += '<div class="right">';
						html += '<img src="http://worko.longfor.com:59650/workofile/head/'+loginUser.data.headIcon+'.jpg" onerror="this.src=\'img/nohead.png\'" />';
					html += '</div>';
				html += '</li>';
		$("#msg-list").append(html);
	},
	//--------------------时间信息-----------------------
	cbotTip : function(data){
		$("#msg-list").append('<li class="msg-time"><span>'+data+'</span></li>')
	}
}

Conclusion

Microsoft China DX team together with Longfor engineer team integrated bot framework with ”Longxin” application by Node.js and connect to Skype too, implemented virtual assistant named “龙小湖”,with the basic functions, built a natural language interaction interface to the IT system, implemented the “Agenda Function” and “Problem Claim”, building the connection between support team with business team for “Problem Claiming”, improved the Direct line 3.0 performance by using the active callback mode instead of web socket and share it with other parthers who are doing the integration with DL3.0.

  • Opportunity going forward:

Though Azure Intelligent cloud, customer has started to use bot and cognitive services, especially they have tried different API in cognitive service. In the next step, they shows great interest in the other APIs in Cognitive services. Also this gives us an opportunity for the further engagement of Azure Services.

Additional resources

During the hackfest we summarized a document in Chinese on the hands on of LUIS and Bot framework, which helps local customers to have a quick start on build their on bot and also our customer shared the connection through direct line 3.0 to 3rd party apps in Node.js. link: https://github.com/andrewdyhhub/BotFramework

For the final video please visit: https://youtu.be/wzDFBi39s_Q