Search Results for

    Show / Hide Table of Contents

    Tips and Tricks

    NUnit 3

    dotnet test and VS Test .runsettings configuration

    Certain NUnit Test Adapter settings are configurable using a .runsettings file. The following options are available:

    Key Type Options Default
    InternalTraceLevel enum Off, Error, Warning, Info, Verbose, Debug Nothing => Off
    NumberOfTestWorkers int nr of workers -1
    ShadowCopyFiles bool True, False False
    Verbosity int -1, 0-5 . -1 means quiet mode 0
    UseVsKeepEngineRunning bool True, False False
    BasePath string path ?
    PrivateBinPath string directory1;directory2;etc ?
    RandomSeed int seed integer random
    DefaultTimeout int timeout in mS, 0 means infinite 0
    DefaultTestNamePattern string Pattern for display name {m}{a}
    WorkDirectory string specify directory Test assembly location
    TestOutputXml string specify directory Test Result Xml output folder
    OutputXmlFolderMode enum UseResultDirectory,RelativeToResultDirectory,RelativeToWorkFolder,AsSpecified RelativeToWorkFolder
    DumpXmlTestDiscovery bool Enable dumping of NUnit discovery response xml false
    DumpXmlTestResults bool Enable dumping of NUnit execution response xml false
    PreFilter bool Enable pre-filtering to increase performance for Visual Studio testing false
    ShowInternalProperties bool Turn on showing internal NUnit properties in Test Explorer false
    Where string NUnit Filter expression
    UseParentFQNForParametrizedTests bool Enable parent as FQN for parametrized tests false
    UseNUnitIdforTestCaseId bool Uses NUnit test id as VSTest Testcase Id, instead of FullyQualifiedName false
    ConsoleOut int Sends standard console output to the output window 1
    UseTestNameInConsoleOutput bool Adds name of test as a prefix in the output window for console output true
    StopOnError bool Stops on first error false
    SkipNonTestAssemblies bool Adapter supports NonTestAssemblyAttribute true
    MapWarningTo enum Map Assert.Warn to either Passed, Failed or Skipped Skipped
    DisplayName enum Set what a DisplayName is, options: Name, FullName or FullNameSep Name
    FullnameSeparator string FullNameSep separator ':'
    DiscoveryMethod enum How execution discovery is done, options are Legacy or Current Current
    AssemblySelectLimit int Number of tests accepted before filters are turned off 2000
    NewOutputXmlFileForEachRun bool Creates a new file for each test run false
    IncludeStackTraceForSuites bool Includes stack trace for failures in suites, like exceptions in OneTimeSetup and OneTimeTearDown true
    IncludeStackTrace bool Includes stack trace for all failures true
    ExplicitMode enum Changes handling of explicit tests, options are Strict or Relaxed Strict
    SkipExecutionWhenNoTests bool Skip execution if no tests are found false
    AllowParallelWithDebugger bool Allow parallel execution when debugger is attached false
    ThrowOnEachFailureUnderDebugger bool false

    Visual Studio templates for runsettings

    You can install item templates for runsettings in Visual Studio (applies to version 2017, 2019 and upwards) which includes the NUnit settings mentioned here. Note that there are available separate installs for earlier Visual Studio versions, links to these can be found in the above.

    Example implementation

    See https://github.com/nunit/nunit3-vs-adapter/blob/8a9b8a38b7f808a4a78598542ddaf557950c6790/demo/demo.runsettings

    NUnit .runsettings implementation

    https://github.com/nunit/nunit3-vs-adapter/blob/master/src/NUnitTestAdapter/AdapterSettings.cs#L143


    Details

    DefaultTestNamePattern

    The default format used to provide test names, expressed as an NUnit Template Based Test Name Pattern.

    WorkDirectory

    Our WorkDirectory is the place where output files are intended to be saved for the run, whether created by NUnit or by the user, who can access the work directory using TestContext. It's different from TestDirectory, which is the directory containing the test assembly. For a run with multiple assemblies, there could be multiple TestDirectories, but only one WorkDirectory. User sets work directory to tell NUnit where to put stuff like the XML or any text output. User may also access it in the code and save other things there. Think of it as the directory for saving stuff.

    TestOutputXml

    If this is specified, the adapter will generate NUnit Test Result Xml data in the folder specified here. If not specified, no such output will be generated.
    The folder can be

    1. An absolute path
    2. A relative path, which is then relative to either WorkDirectory, or if this is not specified, relative to the current directory, as defined by .net runtime.

    (From version 3.12)

    OutputXmlFolderMode

    This setting sets which folder the TestOutputXml will be going to. The default is the RelativeToWorkFolder (see WorkDirectory) above. The option UseResultDirectory will put the results in the same directory as the Trx files, the overall specified test result directory. The last option is the RelativeToResultDirectory, which is normally some path below the result directory. The last option is AsSpecified, which should be used when TestOutputXml is an absolute path.

    (From version 4.3.0)

    The last option AsSpecified is set automatically when TestOutputXml is an absolute path.

    (From version 4.3.1)

    InternalTraceLevel

    This setting is a diagnostic setting forwarded to NUnit, and not used by the adapter itself. For further information see the NUnit Tracelevel documentation

    NumberOfTestWorkers

    This setting is sent to NUnit to determine how parallelization should be performed.
    Note in particular that NUnit can either run directly or for parallel runs use queue of threads. Set to 0, it will run directly, set to 1 it will use a queue with a single thread.

    ShadowCopyFiles

    This setting is sent to NUnit to enable/disable shadow-copying.

    Verbosity

    This controls the outputs from the adapter to the Visual Studio Output/Tests window. A higher number includes the information from the lower numbers. It has the following actual levels:

    -1 : Quiet mode. Only shows errors and warnings.

    0 : Default, normal information verbosity

    1-3: Some more information from setting are output (in particular regarding parallelization)

    4: Outputs the values from the runsettings it has discovered.

    5: Outputs all debug statements in the adapter

    UseVsKeepEngineRunning

    This setting is used by the adapter to signal to the VSTest.Execution engine to keep running after the tests have finished running. This can speed up execution of subsequent test runs, as the execution engine already is loaded, but running the risks of either holding onto test assemblies and having some tests not properly cleaned out. The settings is the same as using the Visual Studio Test/Test Settings/Keep Test Execution Engine running.

    DumpXmlTestDiscovery and DumpXmlTestResults

    These settings are used to dump the output from NUnit, as it is received by the adapter, before any processing in the adapter is done, to disk. It is part of the diagnostics tools for the adapter. You can find the files under your current output folder, in a sub-folder named Dump. (Note: This is not the same as the TestResults folder, this data is not test results, but diagnostics dumps)

    PreFilter

    A pre-filter will improve performance when testing a selection of tests from the Test Explorer. It is default off, because there are issues in certain cases, see below. If you don't have any of the cases below, you can turn PreFilter on.

    • Your code contains a SetupFixture #649
    • Your code uses a TestCaseSource and uses SetName to set the display name instead of SetArgDisplayNames #650
    • You are using a version of NUnit lower than 3.11 #648

    If you just need to add this, you can add a runsettings file (any filename, extension .runsettings) containing:

    <RunSettings>
       <NUnit>
           <PreFilter>true</PreFilter>
       </NUnit>
    </RunSettings>
    

    (From version 3.15.1)

    ShowInternalProperties

    The NUnit internal properties have been "over-populating" in the Test Explorer. These are default filtered out, although you may still see these when you have Source Based Discovery (SBD) turned on (which is the default in VS). Once you have run test execution, they will be gone. We expect this part of the issue (SBD) to be fixed in VS. If you still want to see them, set this property to true.

    Where

    A NUnit Test Selection Language filter can be added to the runsettings file. The details are described in this blog post

    Using the runsettings should be like:

    <RunSettings>
        <NUnit>
            <Where>cat == SomeCategory or method == SomeMethodName or namespace == My.Name.Space or name == 'TestMethod(5)'</Where>
        </NUnit>
    </RunSettings>
    

    Note that the Where statement does not work for the Visual Studio Test Explorer, as it would generate a conflict with the test list the adapter receives. It is intended for use with command line tools, dotnet test or vstest.console.

    (From version 3.16.0)

    UseParentFQNForParametrizedTests

    Setting this may give more stable results when you have complex data driven/parametrized tests. However, when this is set selecting a single test within such a group, means that all tests in that group is executed.

    Note that this often has to be set together with UseNUnitIdforTestCaseId

    (From version 3.16.1)

    UseNUnitIdforTestCaseId

    The default setting is false, causes the VSTest Testcase ID to be based on the NUnit fullname property, which is nearly equal to a FullyQualifiedName. The fullname is also set into the Testcase FullyQualifiedName property.

    By setting this property true, it shifts to using the NUnit id as the basis for the testcase id. This may in certain cases give more stable results, and are more correct.

    However, it has been seen to also have adverse effects, so use with caution.

    (From version 3.16.1)

    ConsoleOut

    When set to 1 or 2 (1 is default), will send Console standard output to the Visual Studio Output/Test window, and also with dotnet test, it will appear in the output.

    Note

    The ConsoleOut default changed from 2 to 1 in v4.6.0

    Disable this by setting it to 0, which is also the default for version earlier than 3.17.0.

    The value 1 sends it out as Information, while 2 sends it out as Warning.

    Using dotnet test with ConsoleOut=1 (Information) (dotnet test -- NUnit.ConsoleOut=1), nothing will appear in the output.

    Setting ConsoleOut=2 (Warning) the message will be shown.

    Adding the logger dotnet test -l "console;verbosity=detailed" -- NUnit.Consoleout=1 the message will be shown.

    See Issue 343 for more information and discussion

    In earlier versions you had to use -v n, but that is no longer required. In order to silence it in dotnet test you have to do:

    dotnet test -- NUnit.ConsoleOut=0
    

    (Note the space after --. )

    (From version 3.17.0, Modified in 4.2.0)

    UseTestNameInConsoleOutput

    When set to true, the default, the name of the test is added as a prefix in front of each console line output. This only applies to the output in the output window.

    (From version 4.0.0)

    StopOnError

    When enabled (set to true), the tests will stop after the first test failed. Useful when working with a system with many tests running for a long time, where a few fails. It can then speed up by stopping at the first one.

    See Issue 675 for more information and discussion

    (From version 3.17.0)

    MapWarningTo

    Assert Warnings will default map to Skipped, but you can set this to any other state, using MapWarningTo. The options are: Passed, Failed or Skipped.

    (From version 3.17.0)

    SkipNonTestAssemblies

    If the attribute NonTestAssembly is added in an assembly, it will be skipped from further testing. If RTD is enabled in Visual Studio, the tests will be displayed, but running will skip them.
    See explanation for the NonTestAssembly Attribute, and Issue explanation here.

    (From version 3.17.0) (Default changed to true from 4.0.0)

    DisplayName

    The default for Test Explorer DisplayName is to use the Name of the test, which normally is the method name. Using DisplayName you can change between Name, FullName or FullNameSep. The last one will then use the FullNameSeparator,which defaults to ':'. See Issue 640 for explanations of use and sample code here.

    (From version 3.17.0)

    FullNameSeparator

    The separator character used when DisplayName is FullNameSep. It is default ':', but can be changed to anything.

    (From version 3.17.0)

    DiscoveryMethod

    The 4.0 version of the NUnit3TestAdapter has a rewritten discovery mechanism and also other redesigns/refactoring done. This setting let you switch back to the old form of discovery, using the setting Legacy. The default value is Current.

    The Current setting enables the Explicit feature back. It also performs better (approx 30% faster for large test sets). It might affect certain special cases, so therefore you can switch back.

    (From version 4.0.0)

    AssemblySelectLimit

    If you run from the IDE (Visual Studio) the adapter will receive a list of tests to process. This is heavy when the number of tests are large. If the number of tests exceeds this limit, the list will be skipped and all tests in the assembly will be run (except those tests that are Explicit or Ignored).

    This might have an adverse effect if you select a category and you have more than 2000 tests, the category setting will be ignored. In that case, just increase this limit to higher than your number of tests.

    You might also receive a list from the command line, and in that case it will also be skipped the same way. Here the category will be honored since the category filter will be converted to a NUnit filter.

    (From version 4.0.0)

    NewOutputXmlFileForEachRun

    Default behavior is to produce one test output file which is being overwritten for each run Setting this to true, the adapter produces a new test output file for each run, numbering them sequentially. Default is false.

    The background is the following scenario, as described by netcorefactory in Issue 800:

    "Running test in azure devops one can choose to rerun failed ( flaky ) tests. Mostly when running (selenium) e2e tests this becomes important. The .xml results file is currently overwritten each retry run. Other test coverage tooling dependent on this file receives only latest run results. Better to allow the possibility to deliver a results file per run."

    (From version 4.0.0)

    IncludeStackTraceForSuites

    Exceptions outside test cases are reported with its stack trace included in the message. One example here is exceptions in OneTimeSetUp and OneTimeTearDown. They belong to the outer suite, and their exceptions are reported as part of the message. This option includes the stack trace in that message. If this becomes too much information, this option turns the stack trace reporting off. Default is true.

    (From version 4.0.0)

    IncludeStackTrace

    This is default true. If turned off (false) it will stop outputting any stacktraces. This is in cases where you just want to reduce the total output and don't need the stacktrace information.

    (From version 5.1.0)

    ExplicitMode

    This setting can be either Strict or Relaxed. The default is Strict, which means that Explicit tests can only be run with other Explicit tests, and not mixed with non-Explicit tests. The Relaxed mode is the original NUnit mode, where if you select a category, a class or a set of tests, both explicit and non-explicit tests will be run. From Visual Studio Test Explorer there are no longer (since VS2019) any way of separating between a Run-All and run selected tests, so Relaxed mode doesn't work properly. It may or may not work for command line tests, dependent upon how your tests are set up and run.

    (From version 4.2.0)

    SkipExecutionWhenNoTests

    If set, this setting will skip execution for an assembly if no tests are found during the pre-execution discovery phase. It will give you a small performance boost, but if you skip the execution, this assembly will not generate any log files. The default is false.

    (From version 4.2.0)

    AllowParallelWithDebugger

    If set, this setting will allow parallel execution of tests even if the debugger is attached. The default is false.

    (From version 4.5.0)

    ThrowOnEachFailureUnderDebugger

    If set, this setting will surface an assertion exception for each failure in an Assert.Multiple block if the debugger is attached. The default is false.
    See docs for more info.

    (From version 4.6, require NUnit from 4.2)


    Some further information on directories

    (From comment on issue 575 by Charlie )

    NUnit also supports TestContext.TestDirectory, which is the directory where the current test assembly is located. Note that if you have several test assemblies in different directories, the value will be different when each one of them accesses it. Note also that there is no way you can set the TestDirectory because it's always where the assembly is located.

    The BasePath is a .NET thing. It's the base directory where assemblies are searched for. You can also have subdirectories listed in the PrivateBinPath. NUnit takes care of all this automatically now, so the old console options are no longer supported. For finding things you want to read at runtime, the TestDirectory and the BasePath will usually be the same thing.

    Registry Settings

    Note

    As of the 3.0 final release, the registry settings are no longer recognized. Instead, use settings in the .runsettings file.


    NUnit 2.x

    NUnit 2.X does not support runsettings.

    Registry Settings for NUnit 2.x

    Certain settings in the registry affect how the adapter runs. All these settings are added by using RegEdit under the key HKCU\Software\nunit.org\VSAdapter.

    ShadowCopy

    By default NUnit no longer uses shadowcopy. If this causes an issue for you shadowcopy can be enabled by setting the DWORD value UseShadowCopy to 1.

    KeepEngineRunning

    By default the NUnit adapter will "Kill" the Visual Studio Test Execution engine after each run. Visual Studio 2013 and later has a new setting under its top menu, Test | Test Settings | Keep Test Execution Engine Running. The adapter normally ignores this setting.

    In some cases it can be useful to have the engine running, e.g. during debugging of the adapter itself. You can then set the adapter to follow the VS setting by setting the DWORD value UseVsKeepEngineRunning to 1.

    Verbosity for NUnit 2.x

    Normally the adapter reports exceptions using a short format, consisting of the message only. You can change it to report a verbose format that includes the stack trace, by setting a the DWORD value Verbosity to 1.

    • Edit this page
    In this article
    Back to top Generated by DocFX | Copyright (c) 2018- The NUnit Project - Licensed under CC BY-NC-SA 4.0