Redback are exploring ways to integrate Blockchain technologies into their energy products and services. However, at this early stage Blockchain technology and frameworks are an extremely new field without many “best practices” defined yet. Most notably there has been very little guidance around implementing DevOps around Blockchain projects, with everything being done manually on developer’s machines.

We engaged Redback to implement an initial DevOps process that would allow developers on the team to modify their smart contracts, have a Continuous Integration (CI) build confirm there are no build errors, provision an in memory test Blockchain testnet to run the integration tests, and publish the test results back into Visual Studio Team Services (VSTS).

This was ground breaking work, with the learnings being published in blog posts, added to official documentation pages and reused internally by other teams within the Microsoft ‘Blockchain Technical Working Group’.

VSTS Dashboard showing unit tests passing

Customer profile

Redback is an Australian company located in Brisbane, Queensland. They are focused on the development of advanced, low cost solar solutions for residential and commercial users. Their products include Smart Hybrid Solar Inverter System, which has intelligent technology that gives you the power to store, monitor and manage your home’s solar energy all in a compact, elegantly designed unit mounted either inside or outside your home.

They have been featured in the media as an innovative energy solutions company, and have seen it partner with energy giant Energy Australia http://redbacktech.com/category/media/

redback company logo http://redbacktech.com/

photo of the group

Microsoft hack team participants

  • David Burela Melbourne, Australia Senior Technical Evangelist, DX Australia @DavidBurela
  • Paul Bouwer Brisbane, Australia Senior SDE, Partner Catalyst team @pbouwer
  • Shawn Cicoria NY Metro, USA Senior SDE, Partner Catalyst team @cicorias
  • Eric Maino Redmond, USA Senior SDE, Partner Catalyst team @ericmaino
  • Ali Hajimirza Redmond, USA SDE, Partner Catalyst team @Ali92HM
  • Tyler Gibson Redmond, USA Senior SDE, Partner Catalyst team

Problem statement

Blockchain based developments are complicated. To further exacerbate this, the current state of the tools and frameworks are in the very early stages of development. This means that there is not much in the way of guidance or best practices available, and requires us to come up with unique solutions to all traditional development practices.

Redback were engaged with Microsoft DX TED around integrating Blockchain technology into their products and services. Early on it was discovered that having multiple developers working on the same smart contract code, could result in the typical range of issues prone to modern software development:

  • Merge conflicts.
  • Not all solution files being checked in.
  • New commits breaking changes in previously working code.
  • Not running tests frequently, especially before/after committing code.
  • Building on a developer’s local machine and deploying to production.

It was decided that putting a basic DevOps process around the critical inner portion would greatly increase the team’s confidence in the current state of the code, and accelerate future development work.

Solution, steps, and delivery

Solutions, steps and delivery It was quite a journey to get from a manual single developer experience, to a workflow and toolchain which scaled up to allow a DevOps flow for a team of developers. We looked at a number of tools and frameworks out in the community, and tried to see which had enough mindshare to allow for easier support. We worked as a team to look through what the challenges were, and how best to overcome the inefficiencies.

photo of the group during initial discussions

Blockchain development done manually

Writing Blockchain “smart contracts” is done in a language called “Solidity” https://solidity.readthedocs.io. The DX TED team had previously used a number of manual tools on a developer’s machine to develop smart contract code in Solidity, in order to compile, deploy, etc.

initial value stream map

Our value stream mapping exercise looked at what it would the impact would be if Redback to adopt this development style. It exposed that using the tools as is would be very wasteful. There were a lot of manual steps throughout the process, with many chances for defects to arise from misconfigured tools, incorrect Blockchain network connection settings and manual copy & pasting. The toolchain was optimised for a single developer and did not scale well with a team of developers. It also did not have a way to automate processes like having a Continuous Integration build to check all committed code. The VSM exercise identified that there were potential improvements around automating builds and automating the testing. These would remove the time intensive manual steps, as well as ensuring that there is a repeatable process for building smart contract. Automating this would also give direct benefits by giving an indication of the health of the current checked in code.

photo of the group mapping out workflows

Improving Blockchain development with DevOps

In order to improve the development experience for the team, it was decided to automate and improve the core check in process. At a minimum, a CI build was needed, and a way to automatically run unit/integration tests on Blockchain code.

The hardest problems we encountered during the engagement overcome were:

  • Finding a Blockchain toolchain that could be used on developer machines and on the VSTS build agent.
  • Having a reliable way to get global npm packages to be executable from the VSTS build agent.
  • Getting a temporary Blockchain environment provisioned and teared down for each execution of the unit/integration tests.
  • Outputting the test results into a format that VSTS would accept.

Using Open Source Blockchain tools

After evaluating a few different Blockchain tool offerings, an open source framework “Truffle” was selected to assist with improving the workflow http://TruffleFramework.com. Truffle allows a developer on their local machine to compile, run unit tests, and deploy to the artefacts to a Blockchain network. This tight “inner loop” integration rapidly improves general developer productivity, and cut down on some manual steps. While running Truffle on a single developer’s machine helped improve their individual workflow, by itself it didn’t scale to automate the team’s DevOps flow. I then focused on finding a way to get Truffle running on a build agent which would allow VSTS could be to orchestrate a DevOps process around it.

Configuring a build agent

To support the VSTS build process, a Windows Virtual Machine was provisioned in Azure and configured with the required build tools and VSTS build agent. As all the Blockchain development tools require Node and npm to function, the first step was to install and configure the Windows versions of these tools. Next the supporting build tools such as Visual C++ compilers and Python are installed automatically via the Windows build tools npm package. Once the prerequisite frameworks are installed, Truffle, TestRPC, and MochaJS extension package get installed as global npm tools. The work we did to find a repeatable way to install the prerequisites on a Windows desktop machine & Windows based build server resulted in a set of tutorials which were put onto the official Truffle documentation site.

Configuring Truffle to output Junit format

Truffle utilises the Mocha test framework to run the smart contract unit tests. Normally the test results are displayed on the developer’s machine when run locally, but we wanted to have the tests run on the build agent, and the results uploaded into VSTS. Mocha can be configured to output the results in a variety of file formats, but none of the defaults worked well with VSTS. The Mocha Junit reporter was able to output the results into a format that VSTS understood and could import.

running Truffle test VSTS Dashboard showing unit tests passing

Hosting a Blockchain testnet for test execution

When Truffle executes the tests on the VSTS build agent, it needs a Blockchain network to deploy the smart contracts to, so that the tests can be executed in the environment. Doing it on a real Blockchain network would take an extraordinarily long time, as each operation would need to wait for the block to be mined, before the next operation could take place. With each block taking 12 seconds, and each test requiring multiple operations, it could take tens of minutes to execute a test suite. TestRPC is an in memory Blockchain environment which allows for instant mining, perfect for development and test execution.

In order to take advantage of it on the build server, we needed to find a way to start a fresh environment, allow test execution to happen, and then tear down the instance after test completion. A series of PowerShell scripts were created to automate this process, allowing VSTS to orchestrate the test run. These scripts are now available on the official Truffle documentation site, for reuse by others.

photo of the training the group to use Truffle

Automated DevOps solution

The final DevOps solution that was deployed involved using a mixture of Visual Studio Team Services, combined with a number of Open Source Blockchain tools and frameworks.

diagram of the workflow implemented in VSTS

  • Developer checks in new code
  • VSTS build is triggered
  • Build agent pulls down the source code
  • Truffle is used to compile the smart contracts
  • Powershell script spins up a temporary in memory Blockchain by using the TestRPC tool
  • Truffle is used to run the unit tests. Results are output into a JUnit.xml file
  • Powershell scripts tear down the TestRPC blockchain.
  • Unit test results are uploaded to VSTS

A follow up value stream mapping exercise found that the introduction of better tooling, combined with the automation by VSTS drastically reduced the process time. The removal of manual steps helped to speed things up and eliminated the defects that arose from manually moving from one tool to another.

VSM after DevOps

Deploying to the Blockchain is still a manual step, however now that the tooling has been improved, it is now easier to take the artefacts and run Truffle to run the same migration script to deploy to a UAT or production blockchain on Azure Blockchain as a Service.

Conclusion

photo of the group reviewing the work

Redback is doing some great work with Blockchain development to explore ways to improve their product offerings. It was great to be involved in helping them use a combination of Visual Studio Team Services and open source Blockchain tooling to automate their team’s development. The learnings from this engagement were turned into new training materials that were published in blog posts, added to official documentation pages, and reused internally by other teams at Microsoft.

We introduced new DevOps practices around Blockchain development by removing manual steps and using VSTS to implement the practices of continuous integration & automated testing. The introduction of DevOps via VSTS and having a visible dashboard showing the health of the build and unit tests is giving the team at Redback the confidence to develop their code quickly with a faster feedback loopp, over the previous manual way of doing Blockchain development.

Future work

There was only limited time to implement the basics of a working VSTS system. In the future there is additional functionality that could be integrated in:

  1. Automate Release Management: Currently the builds artefacts need to be taken and then published to a Blockchain via the “truffle migrate” command. Automating this via VSTS Release Management to push to a Azure Blockchain as a Service testnet or production network is the next logical step.

  2. Use a Linux build server or Windows Subsystem for Linux: VSTS supports both Windows & Linux based build agents. The blockchain development tools are npm based and will work on both platforms. Offering a build solution for teams that prefer Linux over Windows would be useful.

  3. Upgradable smart contracts: Smart contract logic is non-mutable once published. However through the use of address variables within a smart contract, it is possible to point internal functions at other contracts, which can then be swapped out to perform an upgrade. Integrating this with the future Release Management work above is a natural tie in.

Additional resources

Here are links to additional documentation walkthroughs, and tools that were used throughout this project.

Published walkthroughs

  • Details of steps and instructions on David Burela’s blog https://davidburela.wordpress.com/2016/12/23/ethereum-devops-with-truffle-testrpc-visual-studio-team-services/
  • Syndicated on the official Truffle documentation http://truffleframework.com/tutorials/ethereum-devops-truffle-testrpc-vsts

Tools

  • VSTS https://www.visualstudio.com/team-services/
  • Truffle (Ethereum smart contract framework) http://truffleframework.com/
  • Ethereum TestRPC (In-memory Blockchain environment) https://github.com/ethereumjs/testrpc/
  • MochaJS (testing framework) http://mochajs.org/

Source code

To make it easy for others to reproduce our work, the VSTS steps were implemented as a small number of discreet PowerShell scripts. This helped keep each section small and easily debugable.

The PowerShell scripts assume that you have npm as well as the Truffle and TestRPC packages installed on the build machine.

photo of the group reviewing the work

System version information

#assists with debugging and ensuring all tools are installed 
#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path
npm config set prefix $env:npm_path    #only needs to be set once, will update for user
#DEBUG
#$env:path
#npm list -g –depth=0
#Display system information
Write-Host “System version information”
Write-Host -nonewline    “node version: ” ; node -v
Write-Host -nonewline    “npm version: “; npm -v
Write-Host -nonewline    “npm prefix: “;  npm prefix -g
Write-Host -nonewline    “truffle: ” ;    truffle version

Config transform & previous test cleanup

# remove old test results
rm .\junitresults.xml -ea SilentlyContinue 

# Modify the Truffle test runner to use the JUnit reporter
Rename-Item .\truffle.js .\truffle_temp.js
cat .\truffle_temp.js | % { $_ -replace ‘reporter: “spec”‘, ‘reporter: “mocha-junit-reporter”‘ } | Out-File -Encoding ASCII .\truffle.js
rm .\truffle_temp.js

Truffle build

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path#Truffle build
truffle compile

Launch TestRPC

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path# launch the process
echo “launching TestRPC”
$testrpcProcess = Start-Process testrpc -passthru
# persist the PID to disk and display in logs
$testrpcProcess.Id | Export-CliXml testrpcPID.xml
cat testrpcPID.xml

Run Truffle tests

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path# Run the tests
truffle test

Shutdown TestRPC

#Setting environment paths
$ENV:Path = $ENV:Path + “;” + $env:npm_path# retrieve the PID and kill the entire processs tree
cat testrpcPID.xml
$testrpcPID = Import-CliXml testrpcPID.xml
taskkill /pid $testrpcPID /F /T