Introduction

Moving to continuous integration doesn’t happen over night. You start with the pain points. I guess a very common one is the analysis of NUnit test results. Comparing XML files to see trends and getting answers to the question: “When did the test fail for the first time” is a very frustrating business. Remember: Software developers are not in the business of reading XML files.

This blog post is one of two posts that give step by step instructions for importing NUnit test result files into TeamCity. This one covers the “bulk” import of a large number of artifacts. Part 2 demonstrates how to use PowerShell and the .NET FileSystemWatcher object to import test results in real time via TeamCity and a NAnt script.

Figure 1: Test results analysis pain level reduced from a 10 to a 0 using TeamCity

The 10,000 foot overview

I assume there is a list of folders which are named after a build number.  Each of the folders contains a NUnit TestResults.xml file. The files are located in the folder C:\TestResultsImporter\ToImport\v0.1.

A helper script called CopyNewBuildSimulator.ps1 will copy each of the original build artifact folders to a folder that is being watched by the BuildTestResultsWatcher.ps1 script.

When a new TestResults.xml get created the file watcher script will trigger the ProcessXML.ps1 script. This script will then call the StartTeamCityBuildWithCurl.ps1 script. The last script will start a TeamCity build that uses a NAnt build task to locate and import the TestResults.xml file using TeamCity Service Messages.

Script file listings

The main script files that are involved in this workflow chain are listed here:

BuildTestResultsWatcher.ps1

$watcher = New-Object System.IO.FileSystemWatcher
$watcher.Path = "C:\TestResultsImporter\PublishedTestResults\v0.1"
$watcher.IncludeSubdirectories = $true
$watcher.EnableRaisingEvents = $true
$watcher.Filter = "TestResults.xml"


function Process-TestResultsFile([string] $BuildFolder)
{
  Write-Host $BuildFolder
  $TestResultsFilePath = Join-Path -Path $watcher.Path -ChildPath $BuildFolder
  
  $BuildNumber = Split-Path $BuildFolder
  Write-Host $BuildNumber
  
  Write-Host "Found new test results xml"
  try
  {
    . "C:\TestResultsImporter\Scripts\Common\ProcessXML.ps1" `
     -BeforeXMLFilePath $TestResultsFilePath `
     -AfterXMLFilePath "C:\TestResultsImporter\Pickup\TestResults.xml" `
  }
  catch
  {
    Write-Host "Error converting XML"
  }
       
  try
  {
    . "C:\TestResultsImporter\Scripts\Common\StartTeamCityBuildWithCurl.ps1" -BuildNumber $BuildNumber
  }
  catch
  {
    Write-Host "Error calling curl script"
  }
}

$ProcessTestResultFileJob = Register-ObjectEvent -InputObject $watcher -EventName "Created" -SourceIdentifier "TestResultsFileCreatedEventIdentifier"

while($true)
{
    $CreatedEvent = Wait-Event -SourceIdentifier "TestResultsFileCreatedEventIdentifier"
    $BuildFolder = $CreatedEvent.SourceEventArgs.Name
    Remove-Event -SourceIdentifier "TestResultsFileCreatedEventIdentifier"
    [System.Threading.Thread]::Sleep(1000)
    Process-TestResultsFile -BuildFolder $BuildFolder
}
Unregister-Event -SourceIdentifier "TestResultsFileCreatedEventIdentifier"

CopyNewBuildSimulator.ps1

$NewBuildFolder = Get-ChildItem -Path "C:\TestResultsImporter\ToImport\v0.1"

$NewBuildFolder | Sort-Object $_.Name

$FolderToWatch = "C:\TestResultsImporter\PublishedTestResults\v0.1"

Get-ChildItem -Path $FolderToWatch | Remove-Item -Force -Recurse


$NewBuildFolder | ForEach-Object `
{
    $_.Name
    [System.Threading.Thread]::Sleep(60000)
    Copy-Item -Path $_.FullName -Destination $FolderToWatch -Recurse
}

ProcessXML.ps1

param([string] $BeforeXMLFilePath, [string] $AfterXMLFilePath, [string] $XMLTemplatePath)

function Load-XML([string] $Path)
{
    $xml = [xml](get-content -Path $Path -Force) 
    return $xml
}


$FUTestResultXML = Load-XML -Path $BeforeXMLFilePath

#Do some processing, if required.

$FUTestResultXML.Save($AfterXMLFilePath)

StartTeamCityBuildWithCurl.ps1

Param([string] $BuildNumber="50")

Set-StrictMode -Version "Latest"
$DebugPreference = "Continue"


$URL = "`"http://localhost:8887/httpAuth/action.html?add2Queue=bt3&env.name=ExternalBuildID&env.value=$BuildNumber`""
Write-Debug $URL
Write-Debug "Starting teamcity via cURL and teamcity REST API"
$result = start-process -filePath "C:\TestResultsImporter\Tools\curl\curl" `
                        -ArgumentList "--user", "autoadmin:LuckyYou", $URL `
                        -NoNewWindow `
                        -RedirectStandardError "error.txt" `
                        -RedirectStandardOutput "output.txt" `
                        -wait `
                        -passthru
$result.HasExited
$result.ExitCode

ImportTestResultsBulk.build

<?xml version="1.0" ?>
<project name="TestResultsImporter" default="build" xmlns="http://nant.sf.net/release/0.90/nant.xsd">
  <target name="build" depends="send.nunit.report.import.message.to.teamcity"></target>
  <target name="send.nunit.report.import.message.to.teamcity" description="Sends message via standard output to teamcity to trigger the import of a NUnit test result report">
    <echo message="##teamcity[importData type='nunit' path='C:\TestResultsImporter\Pickup\TestResults.xml' parseOutOfDate='true']"></echo>  
  </target>
</project>

Step By Step setup guide

Folder structure

The first important setup is to extract the zip file that is available as download and copy its content into the C:\ root folder of your PC.

The ToImport folder contains the NUnit test results XML files for each particular build.

Figure 2: Test results file to import

The PublishedTestResults folder is being watched by the FileSystemWatcher script. The CopyNewBuildSimulator.ps1 script is copying build folders from the ToImport folder into this folder every 1 minute.

Figure 3: Copy target and folder being watched by the FileSystemWatcher script

TeamCity Configuration

You need to create a very simple build configuration to import a NUnit test result XML files.

Figure 4: Create a new TeamCity project and a new build configuration

Figure 5: This build configuration doesn’t need to be connected to a source control system

Figure 6: Add a NAnt build runner and select the NAnt executable in the tools folder of the download and hook it up to the *.build file

Figure 7: Add a configuration parameter that is going to store the original build number of the imported XML test result

This is this simple. In the next section I am going to show how the bulk import script does it’s job.

Running the import

Pictures

First start the StartFileWatcher.bat file to initiate the FileSystemWatcher. Then double click the StartCopySchedule.bat to start copying the old test result artifacts into the folder that is being watched. The schedule script is going to wait 1 minute between each copy transaction.

Figure 8: File copy scheduler and File system watcher script are kicked off

Figure 9: The copy scheduler is copying the old artifact folders one-by-one into the target folder

Figure 10: The file system watcher script is detecting new folder being created  and is triggering a TeamCity build using cURL and the RESTAPI

Figure 11: TeamCity is running a NUnit test result import only build

Figure 12: The cURL request passed in the value for the ExternalBuildID parameter

Figure 13: The bulk import is resulting in some interesting statistics

Figure 14: Pretty soon the test result imports provide enough data to be able to analyze tests and do forecasts

Figure 15: Teamcity lets you drill down into each individual test and its history

Figure 16: Drill down data for one particular test includes the duration of the test and success and failures

Video

Watch the following YouTube video to see the import magic in action.

Watch the scripts in action

Download

The scripts and sample data can be downloaded here: C_TestResultsImporter.zip

Ausblick

The next blog post about importing NUnit test results into Teamcity is demonstrating an even tighter integration. In this second variation the FileSystemWatcher PowerShell script is running as part of the NAnt script. This way the TeamCity test execution build will take as long as the actual NUnit tests take.