Lüp have a small team that are responsible for development and operations of their event management systems. They interact with over 2.5 million people across 200+ events every year throughout the world. Most data collection is performed during an event when event organisers and exhibitors interact with attendees resulting in large load peaks.

While the collection of customer data had already been automated ensuring that the systems used to do this could scale on the existing legacy IaaS cloud platform was requiring significant operational effort and expense. By moving these workloads to a PaaS solution, the cost of running unnecessary infrastructure to deal with peak loads has been significantly reduced as has the operational effort required via the use of auto scaling.

The use of Azure functions provides a serverless method to scale components of the larger system that are suitable to be called via an event based model. In this solution the user creation process has been moved to a function to allow it to be triggered in a stateless manner from multiple systems. This allows for user creation to scale independently from the other App Services to further assist to manage the variable load in a cost effective manner.

The system was an immediate success with over 30,000 users being created by the system in the first week of its operation alone with no operational issues. This kind of fast success and the removal of the operational burden has allowed the Lüp team to focus on more strategic areas of its business. They are confident that Azure can support their planned future growth and are now looking at more advanced workloads including cognitive services and stream analytics.

Project Team

  • Jeremy Cade - Senior Software Engineer, Lüp
  • Chris Briggs, Software Engineer, Lüp,
  • David Burela - DX Technical Evangelist, Microsoft Australia

Customer Profile

Lüp is a full service event information strategy company whose power lies in our ability to fuse intelligence, technology and deep events expertise to create our integrated suite of products.

These products are designed and built by our team to work in synergy at the deepest levels. Together, they enable a seamless experience for visitors, exhibitors and organizers alike. Lüp enrich the event experiences for over 1.5 million people every year by partnering with event organisers and exhibitors at more than 200 events across Australasia.

Formed in 2011 with a vision to empower people to influence the outcome of their event experience, the Lüp team aspires to invigorate the events market around the world. We tailor our offering to various event groups, ensuring that we deliver the best service and product available in the industry.

Problem statement

Lüp operates with a small, multi discipline engineering team consisting of 2 engineers and 3 developers. This team is responsible for all of the infrastructure and services offered to Lüp’s customers. This includes the maintenance and day to day operations of all of our Infrastructure as a Service (IaaS) virtual machines.

While a large majority of daily tasks had been automated, the number of hours required to maintain and manage scaling events was starting to add up. A key concern with the current IaaS model was the number virtual machines and associated costs, that are required to effectively maintain standby services in order to cope with usage spikes or system failures. This has been further exacerbated by Lüp’s rapid Year over Year (YoY) growth which has only placed more demand on our services & infrastructure.

It was decided that any new engineering work would need to target a Platform as a Service (PaaS) solution in order to reduce the management burden on the engineering team.

Discussions with Lüp Management and Customers helped the engineering team to derive the following requirements:

  1. Auto-Scaling is essential.
  2. A minimum of 2 geographic regions (Data persistence).
  3. .NET Core should be a first class citizen. All greenfield work is .NET Core (where applicable).
  4. Deployment model needed to be compatible with existing build / deploy pipelines and infrastructure.
  5. Application Performance Metrics (APM) should be easily observable.
  6. Serverless computing should be considered (where applicable).
  7. Cost minimization, both in hours and finances.
  8. Staging and Productions environments should be identical.
  9. Integration with Lüp’s existing services via RESTful APIs or Message Passing (e.g. Message Queues);

After considering a number of providers Microsoft Azure was chosen, and a proof of concept project conceived which has since been promoted to a production solution.

Solution

The Lüp engineering team had identified the need to replace our existing authentication services with modern Authentication as a Service (AaaS) and User Management services. These services need to integrate with existing legacy applications whilst being highly-available.

Technical Architecture

The solution developed has three separate main interaction points with other systems. The first of these is via Azure Storage Queues. This was selected to facilitate integration with the existing Lüp applications with minimal change. It also ensures scalability as many messages can be written to the queue in a short time during peak load ensuring that no items are lost.

Azure functions are then used to read the items placed in the queue and then execute the necessary activities e.g. a create user message triggers the create user function. The use of a function for each of these activities allows it to scale independently based on the number of messages requiring action hence providing additional flexibility.

Two separate App Services are then used, one for the AaaS elements of the solution and the second for the User Management elements providing both logical separation of functions but also independent scaling and reporting mechanisms for each without the need to maintain servers.

To ensure that the application was functioning in a consistent manner Application Insights was included for each of the App Services. The effort required to include this element was almost zero however the wealth of data that is provides has helped to assist the Lüp team with visibility of how each of the services are running and to enable a more proactive management approach.

For storage and access of information from the multiple App Services SQL Azure was used. This was configured with Geographic redundancy to meet the stated data persistency requirements. The effort to configure this initially was dramatically reduced when compared with the previous approach of running SQL on a virtual machine as is the ongoing management effort.

Finally a user search layer was added via Azure Search, this separates the search element that can be load intensive into a separate scalable item. It also provides additional capabilities above the previous basic search engine that can be utilised at a later stage if required including natural language processing.

The below diagram shows the high level architecture, for the sake of brevity some elements have been simplified , specifically this diagram is focussed on User Creation quene and function, other queues are utilised to perform different actions in the system.

Architectural Overview Figure: Technical Architecture

The separation of the multiple elements into different App Services and Functions allows each item to interpedently scale when needed, previously scaling of these elements had become very time consuming due to the start time of adding a new virtual machine to run the solution. The use of serverless elements has also dramatically reduced the maintenance needs by removing the requirement to patch and maintain servers.

The core AaaS piece is a ASP.NET Core application derived from Identity Server 4, which provided a number or OpenId and oAuth2 authentication services out of the box. Additional APIs endpoints were added to facilitate User Management and common user activities such as account recovery and user information modification. Each of the API Endpoints provided by the AaaS application is secured to a finite set of clients and/or users. Account data and user information is persisted to a SQL Azure database setup with geographic redundancy in place and Transparent Data Encryption (Encryption at Rest). The AaaS application also provides the end-user UI for authenticating with Lüp services.

Lüp Login Figure: Lüp Login

The User Management UI is an ASP.NET Core application that consumes secured API Endpoints provided by the AaaS application as well as a User Search service provided by Azure Search.

Deployment Model

Lüp has an existing battle tested continuous deployment pipeline consisting of Jenkins CI and Octopus Deploy. Under Lüp’s IaaS model, any successful builds (triggered by commit) would also trigger the the provisioning of new infrastructure for that build. Octopus deploy would be notified upon completion of the infrastructure provisioning step to trigger a deployment to the newly provisioned infrastructure. Once completed Octopus Deploy would run the appropriate scripts to move the newly provisioned infrastructure and deployed application into the staging environment. A further step would be needed to promote these builds into production.

This model while powerful, meant that Lüp’s engineering team needed to be aware that large numbers of virtual machines could be in staging or production environments at any given time.

Contrasted to this the Azure App Service deployment model was an order of magnitude simpler, due in part to deployment Slots. Each of the applications (AaaS and User Management) were configured with a “Staging” deployment slot. Any successful builds were published directly to the Staging Slot. Promotion to “Production” was as simple as executing the appropriate Azure Powershell commandlets.

Deployment Model Figure: Deployment Model for Lüp Applications in Azure App Services

Octopus Deploy was able to accomplish the both the deployment and promotion to Production with a minimal amount of effort.

Azure Function

As an example of one of the items created the code for the user creation Azure function is included below:

public static async Task Run(UserToCreateDTO createUserQueueItem, ICollector<UserToCreateDTO> userWithAlreadyTakenEmailAddressQueue, TraceWriter log)
{
	var clientId = ConfigurationManager.AppSettings["clientId"];
	var clietnSecret = ConfigurationManager.AppSettings["clientSecret"];
	var endpoint = ConfigurationManager.AppSettings["endpoint"];
	var disco = await DiscoveryClient.GetAsync(endpoint);

	// request token
	var tokenClient = new TokenClient(disco.TokenEndpoint, $"{clientId}", $"{clientSecret}");
	var tokenResponse = await tokenClient.RequestClientCredentialsAsync("api");

	if (tokenResponse.IsError)
		{
			log.Info(tokenResponse.Error);
			throw new Exception($"Unable to get a auth token due to {tokenResponse.Error}");
		}

	var client = new HttpClient();
	client.SetBearerToken(tokenResponse.AccessToken);

	var contents = new StringContent(JsonConvert.SerializeObject(createUserQueueItem), Encoding.UTF8, "application/json");

	var response = await client.PostAsync($"{endpoint}/api/users", contents);
	var createUserResponse = new APIResponse<UserToCreateDTO>(response);

	if (!createUserResponse.IsOk())
	{
			if(createUserResponse.Errors.ContainsKey("DuplicateUserName"))
			{
				userWithAlreadyTakenEmailAddressQueue.Add(createUserQueueItem);
			}
			else
			{
				var errorString = string.Empty;

				if(createUserResponse.Errors != null)
				{
					string.Join(";", createUserResponse.Errors);
				}

				throw new Exception($"Api threw an error when creating user: {errorString}");
			}
	}

	var doesUserExistResponse = await client.GetAsync($"{endpoint}/api/users/exists/{createUserQueueItem.Email}");
	if (!doesUserExistResponse.IsSuccessStatusCode)
	{
		throw new Exception($"User was not created.");
	}
}

Utilising an Azure function for this component allows it to be called via an event raised from multiple systems and provides a mechanism for this function to scale separately to the larger applications.

It also ensures that this common activity is not duplicated into multiple separate applications and hence is performed in the same manner regardless of where it is triggered from.

Handling Scaling Events

Under the IaaS model, the time it took to adjust to scaling events was anything from 15 minutes to 45 minutes, depending on the application / services involved. It also required a large amount of configuration as code in order to ensure that services came online with the correct environmental configurations. This amounted to a significant engineering investment in both time and virtual machine resources.

Responding to Scaling Events under the App Service model was as simple as setting a Scale Out policy in the App Service Environment for the two applications. This has allowed both applications to scale based on CPU load when needed.

Scale Out Figure: Azure App Services Scale Out

Reduction in Costs

In comparison to the IaaS model Lüp has seen considerable cost reduction in both engineering time and financial costs in relation to the services.

Conclusion and next steps

The initial production system has been successful in Production with over 30,000 users created in the first week of operation, with zero downtime or scaling issues. The App Service component was able to successfully elastically scale with peak demand without any interaction from the engineering team. Cost both in engineering hours and financial cost has been greatly reduced when compared to a comparable IaaS solution.

Jeremy Cade - Senior Software Engineer, Lüp described some of the outcomes seen:

“Scaling was a massive concern on our legacy system that we no longer need to worry about. It’s an instant win,”

“Azure has proven itself very quickly. Its services are available in two discreet data centres in Sydney and Melbourne, giving us the geographic redundancy and automatic back-ups that were needed to assure our customers of continuity of service,”

The rapid success of this project and the reduction in operational effort for the technical team has allowed Lüp to now confidently plan for higher future rapid growth by leveraging the Azure platform. The team can also spend more time focussing on strategic areas for future growth and is reviewing the possibilities to leverage additional areas of the Azure platform such as cognitive services and stream analytics.