Travelers want to have an unique expeience in trip, but hard to find local food, sightseeing spot and more. With normal app they can find famous food and spot, but can’t find local information easily, so they ask local people, and it is somtimes hard by language barrier. Navitime want to solve the problem and have implemented chatbot in Kamakura NAVITIME Travel App. Users can find famous information from the app. In addition to that users can communicate with bot for finding local food and intersting spot. You can see more detail with this video.

Technologies Used:

Teams:

  • NAVITIME Travel Project Members
  • Ikuo Odanaka : Developer Manager, NAVITIME JAPAN
  • Shinichi Tanabe : Senior Software Development Engineer, NAVITIME JAPAN
  • Makoto Yoshihama : Software Development Engineer, NAVITIME JAPAN
  • Naoya Sasaki : Software Development Engineer, NAVITIME JAPAN
  • Ayako Omori : Technical Evangelist, Microsoft Japan
  • Hiroyuki Watanabe: Technical Evangelist, Microsoft Japan
  • Naoki Sato : Senior Technical Evangelist, Microsoft Japan
  • Daiyu Hatakeyama : Principal Software Developer, Microsoft Japan
  • Masayuki Ota : Technical Evangelist, Microsoft Japan

Customer profile

Company Logo

NAVITIME JAPAN is the leading provider of navigation technology and services. It is one of the biggest navigation business company in Japan. They offer mainly B2B and B2C navigation applications. In a B2B area, they offer navigation app for business person, consulting business for transportation and geo advertisement. In a B2C area, they offer navigation app for walk, train, bus, car, bicycle and started travel business, it is called NAVITIME Travel.

Problem statement

Travelers want to know unique things in each trip, but rarely know local food name, nearest sightseeing spot and more. Now they can’t find good information easily about it with normal apps, so they ask local people, but it is sometimes hard by language barrier. For example, many travelers know Sushi as Japanese food but don’t know great local food such as Shirasu-don, Motsu-yaki and more. Navitime want to solve this problem with chatbot communication and cognitive services by Tokyo Olympics in 2020 because many foreign trippers will come Japan.

Solution and steps

Architecture

Architecture We used the Microsoft Bot Framework to implement a chatbot and we also utilize Direct Line API channel to communicate with the chatbot from our iOS app (Android and Windows can call it). For extracting intent and entity from user input, we used LUIS. Because some users may do typo, we support correcting it with Bing Specll check API then pass texts to LUIS. After extracting intent and entity we call Azure Search with them for fetching information about local food and sightseeing spot stored in DocumetDB. Users can freely input the text and it is somtimes pass over our expectation, so our implmented LUIS app sometimes can’t get Entity. For supporting this we also use Text Analytics API to get key phrase. If LUIS can’t get Entity, we pass key phrase to Azure Search.

Users can also communicate with bot by their photo. For supporting this senario we implemented image recognition with Bing Image Search and Custom Vision API.

On the other hand, Ops and continuous improvement is important for developing bot. For managing bot version, we used API Management. For storing user’s input log, we call Azure Functions and save data in the Cosmos DB. We also utilize Azure Search, Azure Storage and Power BI for monitoring search terms. We can get users’ demand from this text logs and brush up bot again and again.

User experience by app and chatbot combination

We’ve implemented bot in iOS app because we thought chatbot can extend app’s user experience. We will show notification message (green window) in app and invite users to chatbot communication in 2 situation.

  1. User start to use the app for the first time. For letting starters know chatbot, app say “Welcome! Feel free to ask me for travel tips and spot information!”.
  2. App detect that user see several recommended articles in app for long time. It means users can’t find good information for them, so app notify “Not intersted in our recommended articles? You can also…” for starting chatbot communication to help users.

Notification

Users can tap chat button (4th button) in the menu list and can start communication with chatbot. Users can input with free text in Enlgish and Japanese like “I want to eat soba” then bot reply “Do you want …?” and recommend restaurant. App also show menu list and user can select next action.

Communication

If user tap [See More Results]button, user can see another recommendation. User can tap [Search another one]button for inputting different keyword. When user tap [Map]button, users can see map and user can also see how to go there if user tap [Go to the spot (Get Direction)]. Because NAVITIME has own routing technology, bot redirect user to their own routing web app for helping user.

Navigation

Travelers tend to upload their photo to SNS such as Instagram and Facebook. Users find it and want to go place or eat food, but they don’t know where it is and how to get it. Our bot will support this senario. Users can sent photo to our bot and it will help to recognize images and tell users how to get it. ImageRecognition.png

Technical delivery

This section will describe how to implement similar bot with Bot Framework, Cognitive Services and Azure.

Prerequisite steps

Develop bot with Bot Framework

After the prerequisite steps, we started to implement own bot from multi dialog sample code. Because we want to release as iOS app, we used Direct Line API channnel.

Extract intent and entity with LUIS and Bing Spell Check API

For extracting intent and entity from user’s messages we used LUIS. Sometimes users do typo and LUIS can’t extract well, we also use Bing Spell Check API to detect typo and correct it automatically before inputing messages to LUIS with steps below.

  1. Create Bing Spell Check API from Azure Portal.
  2. In a LUIS menu, click [My keys].
  3. Click [External Keys] and [Add a new key] to add Bing Spell Check API key to LUIS.
  4. Go to your each LUIS app menu and click [Publish App] in lefe meun.
  5. In the bottom, you can click [Add Key Association] button and can bind Bing Spell Check API to LUIS.
  6. Trun on [Enable bing spell checker]. It will change Endpoint url. If you call this changed url, you can call LUIS with Bing Spell Check API.

Save NoSQL data in Cosmos DB

We used fully managed NoSQL database service Cosmos DB to store NAVITIME’s location information, which is formatted as JSON. We also use it for storing communication log between users and bot.

You can start with this tutorial for initializing Cosmos DB and make code. You need to import data for testing and production by this step.

Azure Seach Implementation in Bot

Although LUIS can get intents well, users sometimes input just word not sentenses, so we are using Azure Seach too. With the architecture below, we can handle sentences and words and reply information correctly now.

For using Azure Search, we wrote C# code like below.

[Serializable]
public class AzureSearchService
{
    private static readonly string QueryString = $"https://{WebConfigurationManager.AppSettings["SearchName"]}.search.windows.net/indexes/{WebConfigurationManager.AppSettings["IndexName"]}/docs?api-key={WebConfigurationManager.AppSettings["SearchKey"]}&api-version=2015-02-28&";

    public async Task<SearchResult> SearchByName(string name)
    {
        using (var httpClient = new HttpClient())
        {
            string nameQuey = $"{QueryString}search={name}";
            string response = await httpClient.GetStringAsync(nameQuey);
            return JsonConvert.DeserializeObject<SearchResult>(response);
        }
    }

    public async Task<FacetResult> FetchFacets()
    {
        using (var httpClient = new HttpClient())
        {
            string facetQuey = $"{QueryString}facet=Era";
            string response = await httpClient.GetStringAsync(facetQuey);
            return JsonConvert.DeserializeObject<FacetResult>(response);
        }
    }

    public async Task<SearchResult> SearchByEra(string era)
    {
        using (var httpClient = new HttpClient())
        {
            string nameQuey = $"{QueryString}$filter=Era eq '{era}'";
            string response = await httpClient.GetStringAsync(nameQuey);
            return JsonConvert.DeserializeObject<SearchResult>(response);
        }
    }
}

You can use it from Bot Framework with the code below. If you want to see sample project, please see this GitHub repo. This project describes how to setup Azure Search and Cosmos DB and how to call Azure Search by bot.

public virtual async Task MessageRecievedAsync(IDialogContext context, IAwaitable<IMessageActivity> result)
{
    var message = await result;
    try
    {
        SearchResult searchResult = await searchService.SearchByName(message.Text);
        if(searchResult.value.Length != 0)
        {
            CardUtil.showHeroCard(message, searchResult);
        }
        else{
            await context.PostAsync($"No musicians by the name {message.Text} found");
        }
    }
    catch(Exception e)
    {
        Debug.WriteLine($"Error when searching for musician: {e.Message}");
    }
    context.Done<object>(null);
}

Recognize image with Bing Image Search and Custom Vision Service

We can use Bing Image Seach to find similar images and bestRepresentativeQuery. e.g When you send photo to Bing Image Search, it replies similar image urls and “Tsuruoka Hachimangu Temple, Kamakura” as bestRepresentativeQuery. Bing has huge image knowledge and we can use it to recognize famous places and foods with the code below.

public async Task<ImageResult> GetSimilarImagesAsync(string url)
{
    using (var httpClient = new HttpClient()){
        httpClient.DefaultRequestHeaders.Add("Ocp-Apim-Subscription-Key", ApiKey);
        string apiUrl = BingApiUrl + $"&imgUrl={HttpUtility.UrlEncode(url)}";

        var text = await httpClient.GetStringAsync(apiUrl);
        var response = JsonConvert.DeserializeObject<BingImageResponse>(text);

        ImageResult result = new ImageResult();
        if(response.bestRepresentativeQuery.displayText != null)
        {
            result.suggestedText = response.bestRepresentativeQuery.displayText;
        }

    result.similarImages = response
    ?.visuallySimilarImages
    ?.Select(i => new SimilarImage{
        HostPageDisplayUrl = i.hostPageDisplayUrl,
        HostPageUrl = i.hostPageUrl,
        Name = i.name,
        ThumbnailUrl = i.thumbnailUrl,
        WebSearchUrl = i.webSearchUrl
    }).ToList();

    return result;
    }
}

You can find sample project in this repo and learn about how to post image to bot and how to recognize image by Bing Image Seach and get bestRepresentativeQuery. You can see more detail in this document.

Although Bing image search help us to find famous places and foods, it’s difficult to recognize local places and foods. Therefore we decided to use Custom Vision Service. We started to build, test and use API with this document.

Because it is very important thing to brush up bot logic with user’s real input, we want to see user’s input and search logs easily.

For saving user’s messege, we call Azure Functions and store data in Cosmos DB. Azure Functions nativelly support Cosmos DB bind and can save data with small codes like this.

#r "Newtonsoft.Json"

using System;
using System.Net;
using Newtonsoft.Json;

public static HttpResponseMessage Run(HttpRequestMessage req, TraceWriter log, out string outputDocument)
{
    log.Info($"Webhook was triggered!");
    string jsonContent = req.Content.ReadAsStringAsync().Result;
    //Store Data in Cosmos DB
    outputDocument = jsonContent;

    dynamic data = JsonConvert.DeserializeObject(jsonContent);

    //Return HTTP Response (BadRequest or OK)
    if (data.first == null || data.last == null) {
        return req.CreateResponse(HttpStatusCode.BadRequest, new {
            error = "Please pass first/last properties in the input object"
        });
    }

    return req.CreateResponse(HttpStatusCode.OK, new {
        greeting = $"Hello {data.first} {data.last}!"
    });
}

PowerBI can see the data in the Cosmos DB with this tutorial, and can analyze search logs with this steps.

Response performance improvement

We deployed bot in West US region to minimize response time. Because Bot State Service is in West US region, and Direrct Line API endpoints are in Eastern Asia, Eurpose, and North America, and LUIS is in West US region.

At first we deployed our bot in Japan East region and it make slow response becuase when bot call LUIS and Bot State Services, the traffic will round trip over the pacific ocean again and again. If you feel slow response, you can improve the performance by moving your bot to West US region. Architecture

Conclusion

Travelers want to have an unique expeience in trip, but hard to find local food, sightseeing spot and more with normal app. It’s also difficult to find it by communicating with local people because of language barrier. Navitime want to solve this problem and have implemented chatbot in Kamakura NAVITIME Travel App. Users can find famous and local information by app and bot combination.

It’s difficult to make NLP logics in multi-languages, but LUIS solve this problem. LUIS is really good for getting intents in the messege. By Ikuo Odanaka, Developer Manager, NAVITIME JAPAN.

It’s very cool data store. We can save NoSQL data in it, and can fetch data with SQL like query, so it is very easy to use. Read/Write speed is very quick, integration with Azure Seach is seameless. By Shinichi Tanabe, Senior Software Developer, NAVITIME JAPAN.

It is the most smooth project for me. PaaS such as Cosmos DB and Azure Search can help us to make new thing rapiddly. By Shinichi Tanabe, Senior Software Developer, NAVITIME JAPAN.

Additional resources