Moving to continuous integration doesn’t happen over night. You start with the pain points. I guess a very common one is the analysis of NUnit test results. Comparing XML files to see trends and getting answers to the question: “When did the test fail for the first time” is a very frustrating business. Remember: Software developers are not in the business of reading XML files.
This blog post is one of two posts that give step by step instructions for importing NUnit test result files into TeamCity. This one covers the “bulk” import of a large number of artifacts. Part 2 demonstrates how to use PowerShell and the .NET FileSystemWatcher object to import test results in real time via TeamCity and a NAnt script.
Figure 1: Test results analysis pain level reduced from a 10 to a 0 using TeamCity
The 10,000 foot overview
I assume there is a list of folders which are named after a build number. Each of the folders contains a NUnit TestResults.xml file. The files are located in the folder C:\TestResultsImporter\ToImport\v0.1.
A helper script called CopyNewBuildSimulator.ps1 will copy each of the original build artifact folders to a folder that is being watched by the BuildTestResultsWatcher.ps1 script.
When a new TestResults.xml get created the file watcher script will trigger the ProcessXML.ps1 script. This script will then call the StartTeamCityBuildWithCurl.ps1 script. The last script will start a TeamCity build that uses a NAnt build task to locate and import the TestResults.xml file using TeamCity Service Messages.
Script file listings
The main script files that are involved in this workflow chain are listed here:
$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = "C:\TestResultsImporter\PublishedTestResults\v0.1"
$watcher.IncludeSubdirectories = $true
$watcher.EnableRaisingEvents = $true
$watcher.Filter = "TestResults.xml"
function Process-TestResultsFile([string] $BuildFolder)
$TestResultsFilePath = Join-Path -Path $watcher.Path -ChildPath $BuildFolder
$BuildNumber = Split-Path $BuildFolder
Write-Host "Found new test results xml"
. "C:\TestResultsImporter\Scripts\Common\ProcessXML.ps1" `
-BeforeXMLFilePath $TestResultsFilePath `
-AfterXMLFilePath "C:\TestResultsImporter\Pickup\TestResults.xml" `
Write-Host "Error converting XML"
. "C:\TestResultsImporter\Scripts\Common\StartTeamCityBuildWithCurl.ps1" -BuildNumber $BuildNumber
Write-Host "Error calling curl script"
$ProcessTestResultFileJob = Register-ObjectEvent -InputObject $watcher -EventName "Created" -SourceIdentifier "TestResultsFileCreatedEventIdentifier"
$CreatedEvent = Wait-Event -SourceIdentifier "TestResultsFileCreatedEventIdentifier"
$BuildFolder = $CreatedEvent.SourceEventArgs.Name
Remove-Event -SourceIdentifier "TestResultsFileCreatedEventIdentifier"
Process-TestResultsFile -BuildFolder $BuildFolder
Unregister-Event -SourceIdentifier "TestResultsFileCreatedEventIdentifier"
$NewBuildFolder = Get-ChildItem -Path "C:\TestResultsImporter\ToImport\v0.1"
$NewBuildFolder | Sort-Object $_.Name
$FolderToWatch = "C:\TestResultsImporter\PublishedTestResults\v0.1"
Get-ChildItem -Path $FolderToWatch | Remove-Item -Force -Recurse
$NewBuildFolder | ForEach-Object `
Copy-Item -Path $_.FullName -Destination $FolderToWatch -Recurse
param([string] $BeforeXMLFilePath, [string] $AfterXMLFilePath, [string] $XMLTemplatePath)
function Load-XML([string] $Path)
$xml = [xml](get-content -Path $Path -Force)
$FUTestResultXML = Load-XML -Path $BeforeXMLFilePath
#Do some processing, if required.
Set-StrictMode -Version "Latest"
$DebugPreference = "Continue"
$URL = "`"http://localhost:8887/httpAuth/action.html?add2Queue=bt3&env.name=ExternalBuildID&env.value=$BuildNumber`""
Write-Debug "Starting teamcity via cURL and teamcity REST API"
$result = start-process -filePath "C:\TestResultsImporter\Tools\curl\curl" `
-ArgumentList "--user", "autoadmin:LuckyYou", $URL `
-RedirectStandardError "error.txt" `
-RedirectStandardOutput "output.txt" `
<?xml version="1.0" ?>
<project name="TestResultsImporter" default="build" xmlns="http://nant.sf.net/release/0.90/nant.xsd">
<target name="build" depends="send.nunit.report.import.message.to.teamcity"></target>
<target name="send.nunit.report.import.message.to.teamcity" description="Sends message via standard output to teamcity to trigger the import of a NUnit test result report">
<echo message="##teamcity[importData type='nunit' path='C:\TestResultsImporter\Pickup\TestResults.xml' parseOutOfDate='true']"></echo>
Step By Step setup guide
The first important setup is to extract the zip file that is available as download and copy its content into the C:\ root folder of your PC.
The ToImport folder contains the NUnit test results XML files for each particular build.
Figure 2: Test results file to import
The PublishedTestResults folder is being watched by the FileSystemWatcher script. The CopyNewBuildSimulator.ps1 script is copying build folders from the ToImport folder into this folder every 1 minute.
Figure 3: Copy target and folder being watched by the FileSystemWatcher script
You need to create a very simple build configuration to import a NUnit test result XML files.
Figure 4: Create a new TeamCity project and a new build configuration
Figure 5: This build configuration doesn’t need to be connected to a source control system
Figure 6: Add a NAnt build runner and select the NAnt executable in the tools folder of the download and hook it up to the *.build file
Figure 7: Add a configuration parameter that is going to store the original build number of the imported XML test result
This is this simple. In the next section I am going to show how the bulk import script does it’s job.
Running the import
First start the StartFileWatcher.bat file to initiate the FileSystemWatcher. Then double click the StartCopySchedule.bat to start copying the old test result artifacts into the folder that is being watched. The schedule script is going to wait 1 minute between each copy transaction.
Figure 8: File copy scheduler and File system watcher script are kicked off
Figure 9: The copy scheduler is copying the old artifact folders one-by-one into the target folder
Figure 10: The file system watcher script is detecting new folder being created and is triggering a TeamCity build using cURL and the RESTAPI
Figure 11: TeamCity is running a NUnit test result import only build
Figure 12: The cURL request passed in the value for the ExternalBuildID parameter
Figure 13: The bulk import is resulting in some interesting statistics
Figure 14: Pretty soon the test result imports provide enough data to be able to analyze tests and do forecasts
Figure 15: Teamcity lets you drill down into each individual test and its history
Figure 16: Drill down data for one particular test includes the duration of the test and success and failures
Watch the following YouTube video to see the import magic in action.
Watch the scripts in action
The scripts and sample data can be downloaded here: C_TestResultsImporter.zip
The next blog post about importing NUnit test results into Teamcity is demonstrating an even tighter integration. In this second variation the FileSystemWatcher PowerShell script is running as part of the NAnt script. This way the TeamCity test execution build will take as long as the actual NUnit tests take.