Tips and Tricks

NUnit 3

dotnet test and VS Test .runsettings configuration

Certain NUnit Test Adapter settings are configurable using a .runsettings file. The following options are available:

Key Type Options Default
InternalTraceLevel enum Off, Error, Warning, Info, Verbose, Debug Nothing => Off
NumberOfTestWorkers int nr of workers -1
ShadowCopyFiles bool True, False False
Verbosity int -1, 0-5 . -1 means quiet mode 0
UseVsKeepEngineRunning bool True, False False
BasePath string path ?
PrivateBinPath string directory1;directory2;etc ?
RandomSeed int seed integer random
DefaultTimeout int timeout in mS, 0 means infinite 0
DefaultTestNamePattern string Pattern for display name {m}{a}
WorkDirectory string specify directory Test assembly location
TestOutputXml string specify directory Test Result Xml output folder
OutputXmlFolderMode enum UseResultDirectory,RelativeToResultDirectory,RelativeToWorkFolder,AsSpecified RelativeToWorkFolder
DumpXmlTestDiscovery bool Enable dumping of NUnit discovery response xml false
DumpXmlTestResults bool Enable dumping of NUnit execution response xml false
PreFilter bool Enable pre-filtering to increase performance for Visual Studio testing false
ShowInternalProperties bool Turn on showing internal NUnit properties in Test Explorer false
Where string NUnit Filter expression
UseParentFQNForParametrizedTests bool Enable parent as FQN for parametrized tests false
UseNUnitIdforTestCaseId bool Uses NUnit test id as VSTest Testcase Id, instead of FullyQualifiedName false
ConsoleOut int Sends standard console output to the output window 1
UseTestNameInConsoleOutput bool Adds name of test as a prefix in the output window for console output true
StopOnError bool Stops on first error false
SkipNonTestAssemblies bool Adapter supports NonTestAssemblyAttribute true
MapWarningTo enum Map Assert.Warn to either Passed, Failed or Skipped Skipped
DisplayName enum Set what a DisplayName is, options: Name, FullName or FullNameSep Name
FullnameSeparator string FullNameSep separator ':'
DiscoveryMethod enum How execution discovery is done, options are Legacy or Current Current
AssemblySelectLimit int Number of tests accepted before filters are turned off 2000
NewOutputXmlFileForEachRun bool Creates a new file for each test run false
IncludeStackTraceForSuites bool Includes stack trace for failures in suites, like exceptions in OneTimeSetup true
ExplicitMode enum Changes handling of explicit tests, options are Strict or Relaxed Strict
SkipExecutionWhenNoTests bool Skip execution if no tests are found false
AllowParallelWithDebugger bool Allow parallel execution when debugger is attached false
ThrowOnEachFailureUnderDebugger bool false

Visual Studio templates for runsettings

You can install item templates for runsettings in Visual Studio (applies to version 2017, 2019 and upwards) which includes the NUnit settings mentioned here. Note that there are available separate installs for earlier Visual Studio versions, links to these can be found in the above.

Example implementation

See https://github.com/nunit/nunit3-vs-adapter/blob/8a9b8a38b7f808a4a78598542ddaf557950c6790/demo/demo.runsettings

NUnit .runsettings implementation

https://github.com/nunit/nunit3-vs-adapter/blob/master/src/NUnitTestAdapter/AdapterSettings.cs#L143

Details

DefaultTestNamePattern

The default format used to provide test names, expressed as an NUnit Template Based Test Name Pattern.

WorkDirectory

Our WorkDirectory is the place where output files are intended to be saved for the run, whether created by NUnit or by the user, who can access the work directory using TestContext. It's different from TestDirectory, which is the directory containing the test assembly. For a run with multiple assemblies, there could be multiple TestDirectories, but only one WorkDirectory. User sets work directory to tell NUnit where to put stuff like the XML or any text output. User may also access it in the code and save other things there. Think of it as the directory for saving stuff.

TestOutputXml

If this is specified, the adapter will generate NUnit Test Result Xml data in the folder specified here. If not specified, no such output will be generated.
The folder can be

  1. An absolute path
  2. A relative path, which is then relative to either WorkDirectory, or if this is not specified, relative to the current directory, as defined by .net runtime.

(From version 3.12)

OutputXmlFolderMode

This setting sets which folder the TestOutputXml will be going to. The default is the RelativeToWorkFolder (see WorkDirectory) above. The option UseResultDirectory will put the results in the same directory as the Trx files, the overall specified test result directory. The last option is the RelativeToResultDirectory, which is normally some path below the result directory. The last option is AsSpecified, which should be used when TestOutputXml is an absolute path.

(From version 4.3.0)

The last option AsSpecified is set automatically when TestOutputXml is an absolute path.

(From version 4.3.1)

InternalTraceLevel

This setting is a diagnostic setting forwarded to NUnit, and not used by the adapter itself. For further information see the NUnit Tracelevel documentation

NumberOfTestWorkers

This setting is sent to NUnit to determine how parallelization should be performed.
Note in particular that NUnit can either run directly or for parallel runs use queue of threads. Set to 0, it will run directly, set to 1 it will use a queue with a single thread.

ShadowCopyFiles

This setting is sent to NUnit to enable/disable shadow-copying.

Verbosity

This controls the outputs from the adapter to the Visual Studio Output/Tests window. A higher number includes the information from the lower numbers. It has the following actual levels:

-1 : Quiet mode. Only shows errors and warnings.

0 : Default, normal information verbosity

1-3: Some more information from setting are output (in particular regarding parallelization)

4: Outputs the values from the runsettings it has discovered.

5: Outputs all debug statements in the adapter

UseVsKeepEngineRunning

This setting is used by the adapter to signal to the VSTest.Execution engine to keep running after the tests have finished running. This can speed up execution of subsequent test runs, as the execution engine already is loaded, but running the risks of either holding onto test assemblies and having some tests not properly cleaned out. The settings is the same as using the Visual Studio Test/Test Settings/Keep Test Execution Engine running.

DumpXmlTestDiscovery and DumpXmlTestResults

These settings are used to dump the output from NUnit, as it is received by the adapter, before any processing in the adapter is done, to disk. It is part of the diagnostics tools for the adapter. You can find the files under your current output folder, in a sub-folder named Dump. (Note: This is not the same as the TestResults folder, this data is not test results, but diagnostics dumps)

PreFilter

A pre-filter will improve performance when testing a selection of tests from the Test Explorer. It is default off, because there are issues in certain cases, see below. If you don't have any of the cases below, you can turn PreFilter on.

  • Your code contains a SetupFixture #649
  • Your code uses a TestCaseSource and uses SetName to set the display name instead of SetArgDisplayNames #650
  • You are using a version of NUnit lower than 3.11 #648

If you just need to add this, you can add a runsettings file (any filename, extension .runsettings) containing:

<RunSettings>
   <NUnit>
       <PreFilter>true</PreFilter>
   </NUnit>
</RunSettings>

(From version 3.15.1)

ShowInternalProperties

The NUnit internal properties have been "over-populating" in the Test Explorer. These are default filtered out, although you may still see these when you have Source Based Discovery (SBD) turned on (which is the default in VS). Once you have run test execution, they will be gone. We expect this part of the issue (SBD) to be fixed in VS. If you still want to see them, set this property to true.

Where

A NUnit Test Selection Language filter can be added to the runsettings file. The details are described in this blog post

Using the runsettings should be like:

<RunSettings>
    <NUnit>
        <Where>cat == SomeCategory or method == SomeMethodName or namespace == My.Name.Space or name == 'TestMethod(5)'</Where>
    </NUnit>
</RunSettings>

Note that the Where statement does not work for the Visual Studio Test Explorer, as it would generate a conflict with the test list the adapter receives. It is intended for use with command line tools, dotnet test or vstest.console.

(From version 3.16.0)

UseParentFQNForParametrizedTests

Setting this may give more stable results when you have complex data driven/parametrized tests. However, when this is set selecting a single test within such a group, means that all tests in that group is executed.

Note that this often has to be set together with UseNUnitIdforTestCaseId

(From version 3.16.1)

UseNUnitIdforTestCaseId

The default setting is false, causes the VSTest Testcase ID to be based on the NUnit fullname property, which is nearly equal to a FullyQualifiedName. The fullname is also set into the Testcase FullyQualifiedName property.

By setting this property true, it shifts to using the NUnit id as the basis for the testcase id. This may in certain cases give more stable results, and are more correct.

However, it has been seen to also have adverse effects, so use with caution.

(From version 3.16.1)

ConsoleOut

When set to 1 or 2 (1 is default), will send Console standard output to the Visual Studio Output/Test window, and also with dotnet test, it will appear in the output.

Note

The ConsoleOut default changed from 2 to 1 in v4.6.0

Disable this by setting it to 0, which is also the default for version earlier than 3.17.0.

The value 1 sends it out as Information, while 2 sends it out as Warning.

Using dotnet test with ConsoleOut=1 (Information) (dotnet test -- NUnit.ConsoleOut=1), nothing will appear in the output.

Setting ConsoleOut=2 (Warning) the message will be shown.

Adding the logger dotnet test -l "console;verbosity=detailed" -- NUnit.Consoleout=1 the message will be shown.

See Issue 343 for more information and discussion

In earlier versions you had to use -v n, but that is no longer required. In order to silence it in dotnet test you have to do:

dotnet test -- NUnit.ConsoleOut=0

(Note the space after --. )

(From version 3.17.0, Modified in 4.2.0)

UseTestNameInConsoleOutput

When set to true, the default, the name of the test is added as a prefix in front of each console line output. This only applies to the output in the output window.

(From version 4.0.0)

StopOnError

When enabled (set to true), the tests will stop after the first test failed. Useful when working with a system with many tests running for a long time, where a few fails. It can then speed up by stopping at the first one.

See Issue 675 for more information and discussion

(From version 3.17.0)

MapWarningTo

Assert Warnings will default map to Skipped, but you can set this to any other state, using MapWarningTo. The options are: Passed, Failed or Skipped.

(From version 3.17.0)

SkipNonTestAssemblies

If the attribute NonTestAssembly is added in an assembly, it will be skipped from further testing. If RTD is enabled in Visual Studio, the tests will be displayed, but running will skip them.
See explanation for the NonTestAssembly Attribute, and Issue explanation here.

(From version 3.17.0) (Default changed to true from 4.0.0)

DisplayName

The default for Test Explorer DisplayName is to use the Name of the test, which normally is the method name. Using DisplayName you can change between Name, FullName or FullNameSep. The last one will then use the FullNameSeparator,which defaults to ':'. See Issue 640 for explanations of use and sample code here.

(From version 3.17.0)

FullNameSeparator

The separator character used when DisplayName is FullNameSep. It is default ':', but can be changed to anything.

(From version 3.17.0)

DiscoveryMethod

The 4.0 version of the NUnit3TestAdapter has a rewritten discovery mechanism and also other redesigns/refactoring done. This setting let you switch back to the old form of discovery, using the setting Legacy. The default value is Current.

The Current setting enables the Explicit feature back. It also performs better (approx 30% faster for large test sets). It might affect certain special cases, so therefore you can switch back.

(From version 4.0.0)

AssemblySelectLimit

If you run from the IDE (Visual Studio) the adapter will receive a list of tests to process. This is heavy when the number of tests are large. If the number of tests exceeds this limit, the list will be skipped and all tests in the assembly will be run (except those tests that are Explicit or Ignored).

This might have an adverse effect if you select a category and you have more than 2000 tests, the category setting will be ignored. In that case, just increase this limit to higher than your number of tests.

You might also receive a list from the command line, and in that case it will also be skipped the same way. Here the category will be honored since the category filter will be converted to a NUnit filter.

(From version 4.0.0)

NewOutputXmlFileForEachRun

Default behavior is to produce one test output file which is being overwritten for each run Setting this to true, the adapter produces a new test output file for each run, numbering them sequentially. Default is false.

The background is the following scenario, as described by netcorefactory in Issue 800:

"Running test in azure devops one can choose to rerun failed ( flaky ) tests. Mostly when running (selenium) e2e tests this becomes important. The .xml results file is currently overwritten each retry run. Other test coverage tooling dependent on this file receives only latest run results. Better to allow the possibility to deliver a results file per run."

(From version 4.0.0)

IncludeStackTraceForSuites

Exceptions outside test cases are reported with its stack trace included in the message. One example here is exceptions in OneTimeSetUp and OneTimeTearDown. They belong to the outer suite, and their exceptions are reported as part of the message. This option includes the stack trace in that message. If this becomes too much information, this option turns the stack trace reporting off.

(From version 4.0.0)

ExplicitMode

This setting can be either Strict or Relaxed. The default is Strict, which means that Explicit tests can only be run with other Explicit tests, and not mixed with non-Explicit tests. The Relaxed mode is the original NUnit mode, where if you select a category, a class or a set of tests, both explicit and non-explicit tests will be run. From Visual Studio Test Explorer there are no longer (since VS2019) any way of separating between a Run-All and run selected tests, so Relaxed mode doesn't work properly. It may or may not work for command line tests, dependent upon how your tests are set up and run.

(From version 4.2.0)

SkipExecutionWhenNoTests

If set, this setting will skip execution for an assembly if no tests are found during the pre-execution discovery phase. It will give you a small performance boost, but if you skip the execution, this assembly will not generate any log files. The default is false.

(From version 4.2.0)

AllowParallelWithDebugger

If set, this setting will allow parallel execution of tests even if the debugger is attached. The default is false.

(From version 4.5.0)

ThrowOnEachFailureUnderDebugger

If set, this setting will surface an assertion exception for each failure in an Assert.Multiple block if the debugger is attached. The default is false.
See docs for more info.

(From version 4.6, require NUnit from 4.2)

Some further information on directories

(From comment on issue 575 by Charlie )

NUnit also supports TestContext.TestDirectory, which is the directory where the current test assembly is located. Note that if you have several test assemblies in different directories, the value will be different when each one of them accesses it. Note also that there is no way you can set the TestDirectory because it's always where the assembly is located.

The BasePath is a .NET thing. It's the base directory where assemblies are searched for. You can also have subdirectories listed in the PrivateBinPath. NUnit takes care of all this automatically now, so the old console options are no longer supported. For finding things you want to read at runtime, the TestDirectory and the BasePath will usually be the same thing.

Registry Settings

Note

As of the 3.0 final release, the registry settings are no longer recognized. Instead, use settings in the .runsettings file.

NUnit 2.x

NUnit 2.X does not support runsettings.

Registry Settings for NUnit 2.x

Certain settings in the registry affect how the adapter runs. All these settings are added by using RegEdit under the key HKCU\Software\nunit.org\VSAdapter.

ShadowCopy

By default NUnit no longer uses shadowcopy. If this causes an issue for you shadowcopy can be enabled by setting the DWORD value UseShadowCopy to 1.

KeepEngineRunning

By default the NUnit adapter will "Kill" the Visual Studio Test Execution engine after each run. Visual Studio 2013 and later has a new setting under its top menu, Test | Test Settings | Keep Test Execution Engine Running. The adapter normally ignores this setting.

In some cases it can be useful to have the engine running, e.g. during debugging of the adapter itself. You can then set the adapter to follow the VS setting by setting the DWORD value UseVsKeepEngineRunning to 1.

Verbosity for NUnit 2.x

Normally the adapter reports exceptions using a short format, consisting of the message only. You can change it to report a verbose format that includes the stack trace, by setting a the DWORD value Verbosity to 1.