Problems with .NET unit testing

 

JetBrains Rider automatically detects unit tests and provides a unit test runner to run and debug them. We are continuously improving unit testing with JetBrains Rider, but there are still some areas which are not supported.

So what is currently supported and what is not? 

  Supported since 2017.2
Unit testing Frameworks

xUnit

NUnit

Classic MSTest (Windows-only)

MSTest v2 (.NET Core projects - cross-platform, .NET Framework projects - Windows-only)

Runtime

.NET Framework

.NET Core

Mono (4.8+ recommended)

If you experience any issues with a supported unit testing framework on a supported runtime, please create a request and describe the problem in details, as it will help to sort it out timely. The following information is essential to determine the cause of the issue:

Environment

  1. OS version, for example, Windows 10 or Mac OS X 10.8
  2. JetBrains Rider version: Help -> About
  3. Unit testing framework version, for example, xUnit 2.2.0
  4. MSBuild version: Settings -> Build, Execution, Deployment -> Toolset and Build (find screenshot below)
  5. .NET Core version in case you have .NET Core project:
    • Check a path to dotnet executable via Settings -> Build, Execution, Deployment -> Toolset and Build (find screenshot below)
    • Go to the directory with dotnet executable via terminal
    • Run "dotnet --version" command

Log Files

  1. Build your solution and then get 'backend.log' file: Help -> Show Log in Explorer/Finder/File Manager
  2. Run Unit Tests if you can see them in unit test explorer 
    • Get launch log files: Launch settings -> Show Last Launch Log (find screenshot below)
    • Get unit test process log files as described here
  3. Zip these three log files and attach the archive to the request

 

Additional information

  1. Sample solution where the issue reproduces is very much appreciated
  2. Can you see your unit tests in unit test explorer?
  3. Are there any steps to reproduce the issue?
  4. Has it worked in the previous version of JetBrains Rider?

 Thank you for helping us make JetBrains Rider better!

27 comments
Comment actions Permalink

The marketing page for Rider quotes NUnit test support. Is this only for .Net Framework and not .Net Core projects? There is no template for a .Net core Nunit test project: 

For now I am using NunitLite to debug and run tests.

Edited by Alex Taroghion
0
Comment actions Permalink

@Alex
NUnit dotnetcore is not working in the current release.

However it is already fixed in Re# and will get to Rider 2017.2.

https://youtrack.jetbrains.com/issue/RSRP-464912#comment=27-2312068

0
Comment actions Permalink

To add to this, in Rider 2018.1.3 the settings within the unit test runner settings (see screenshot) are not respected in the unit test runner itself. The "Automatic" option was still selected while the platform architecture was set to x64.

1
Comment actions Permalink

Kevin,

Thanks for your feedback! Do I understand correctly that you've changed Platform Type in the NuGet tool window (screenshot 2), but it was not changed accordingly in Unit Testing settings (screenshot 1)?

0
Comment actions Permalink

Not quite, the second window was the unit test settings. Before taking the screenshot it looks like I'd already fixed the setting, but the unit test tools (window?) didn't use the setting I'd set in the Settings window for the architecture. I took the screenshot after manually setting the arch to x64 and verifying it worked.

0
Comment actions Permalink

I seem to be unable to debug unit tests in

JetBrains Rider 2018.2.3
Build #RD-182.4231.496, built on September 13, 2018
Licensed to Gary Woodfine
Subscription is active until November 5, 2018
JRE: 1.8.0_152-release-1248-b8 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Linux 4.15.0-36-generic

The debugging of Unit Tests was working this morning but it seems to just randomly stopped working.  I have shut down and restarted rider,  rebooted machine. bu the problem keeps recurring.  I am actually able to Run the tests, using Rider and Dotnet Test.  but when I try debug,

I Get Exception below.

I have tried to CLEAN , RESTORE and Rebuild several times.  I created a blank file api.tests.xunit.runner.json

 

System.IO.FileNotFoundException: Could not find file '/home/gary/code/rea-development/backend/candidate/tests/bin/Debug/netcoreapp2.1/api.tests.xunit.runner.json'.
File name: '/home/gary/code/rea-development/backend/candidate/tests/bin/Debug/netcoreapp2.1/api.tests.xunit.runner.json'
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Interop.CheckIo(Error error, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.IO.File.OpenRead(String path)
at Xunit.ConfigReader_Json.File_OpenRead(String path) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Configuration\ConfigReader_Json.cs:line 172
at Xunit.ConfigReader_Json.LoadFile(String configFileName) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Configuration\ConfigReader_Json.cs:line 141
Edited by gary woodfine
1
Comment actions Permalink

Unable to edit the above comment 

But I am using .net core 2.1 

I noticed the debug output had some additional info.  I am going to attempt to recreate the unit project and see if the problem recurs

I tried creating a Fresh Unit Test project Using dotnet new xunit and wrote a simple unit test and the problem recurs.

 

Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.MemoryMappedFiles.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll was not found or failed to read
Edited by gary woodfine
0
Comment actions Permalink

How about removing 

/usr/share/dotnet/shared/Microsoft.NETCore.App 
And letting Rider or dotnet restore to restore it again?
0
Comment actions Permalink

Not sure where exactly where you want me to do this?

What I have tried is basically

sudo apt remove --purge dotnet*

Then I completely reinstalled dotnet using

sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod bionic main" > /etc/apt/sources.list.d/dotnetdev.list'

sudo apt-get update
sudo apt-get install dotnet-sdk-2.1 

Howeever the problem remains

In the Rider Settings --> Build ... -> Toolset and Build

and reset all the options to custom then  dotnet restore  and try test run debug i still get the error

 

 

 

Edited by gary woodfine
0
Comment actions Permalink

I meant deleting folder in the file-system:

/usr/share/dotnet/shared/Microsoft.NETCore.App 
1
Comment actions Permalink

Ok Cool.

After some more investigating yesterday, and trying out other projects which I have Unit Tests in. It is only one project which has the issue.

This leads me to believe that there must be something about this particular project's configuration which may be a little irregular. I'll do some more digging this morning and get back to you.

 

0
Comment actions Permalink

 

Interesting!  I have just tried a dotnet build on the solution.

I found a couple of warnings that point to the Unit Test Project. 

Do you know where the build log is stored? so I can view it?   I can't seem to find it

/usr/share/dotnet/sdk/2.1.403/Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "Microsoft.Extensions.DependencyModel" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/gary/code/rea-development/backend/candidate/tests/api.tests/api.tests.csproj]
api.tests -> /home/gary/code/rea-development/backend/candidate/tests/api.tests/bin/Debug/netcoreapp2.1/api.tests.dll

Build succeeded.

/usr/share/dotnet/sdk/2.1.403/Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "Microsoft.Extensions.DependencyModel" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/gary/code/rea-development/backend/candidate/tests/api.tests/api.tests.csproj]
1 Warning(s)
0 Error(s)

 

Edited by gary woodfine
0
Comment actions Permalink

It should be "Open log file" in the Build Toolwindow.


1
Comment actions Permalink

OK Cool!   Honestly didn't think of that!!   Thats a cool feature I was blissfully unaware of!  Thanks for letting me know

0
Comment actions Permalink

@gary woodfine did you ever figure this out? I'm running into the exact same issue: can't debug an xunit test.

0
Comment actions Permalink

I'm afraid I haven't as yet!

I still have the problem, I have a suspicion that it has something to do with referencing the API project, which is developed in .net core 2.1. I am able to debug other unit test projects within the same solution, and other solutions.   The only anomoly I can identify is that the particular project I have an issue with references a .net core API 2.1 project!

I'm not sure if this is correct, it's the only conclusion I can jump too.  I am still waiting for feedback

What project type are you having an issue with?

Hope this helps

0
Comment actions Permalink

Unfortunately it seems, that Rider does not fully support non-deterministic xUnit tests, like ones based on AutoFixture. It seems that it's a problem with pre-enumeration.

I've built a small example at https://github.com/drivenet/rider-xunit-discovery-problem

Load solution, restore packages, build the project.

Now, run all tests -- everything seems OK.

Now, try to run the "DetectedProperly" test only -- it works.

Then -- try to run the "NotDetectedProperly" test only -- it either becomes "Unknown, test nor run" or "Inconclusive".

 

Should I file a bug somewhere?

1
Comment actions Permalink

Xm,

Please file an issue on YouTrack: https://youtrack.jetbrains.com/issues/rider

Thank you!

1
Comment actions Permalink

Encountered the same problem as Gary. On newly created xUnit test project I started getting those File not found exceptions when I was trying to debug a test (any test in the project)

Creating a new xUnit project (in the same solution) didn't help. 

I went to the Run -> Stop on exception configuration and re-added the System.Exception. Pop up appeared saying that the exception type is already added. This was expected behaviour.

What was unexpected that after this I was able to F5 thru the FileNotFound exception and after that a break point was hit in the test method...

0
Comment actions Permalink

gary woodfineLukas Belovic

The issue was that the debugger didn't respect the "Just my code" setting in exception breakpoints. The workaround for Rider 2018.2.3 is either disable exception breakpoints (Run -> View Breakpoints) or step through all FileNotFound exceptions. In Rider 2018.3, we've fixed "Just my code" setting. If it is turned on, Rider won't stop on the exceptions thrown by xUnit.

Hope this helps!

0
Comment actions Permalink

I've run into an issue after upgrading. It appears as if the "Run <n> number of assemblies in parallel" option is not being honored. I have some Nunit test assemblies that cannot be run in parallel so I have that setting set to 1. In previous versions, my tests worked out fine. Since upgrading I'm getting massive test failures that seem to indicate that some of these assemblies are either not being unloaded properly. I would like to find out if there is a work around or if anyone has seen this issue.

1
Comment actions Permalink

I'm having this issue on 2019.1.1 on Mac OS X with .NET Core 2.2. I can confirm that it's stopping on intermediate code. To be fair, I told it to, but it's really confusing during startup. Especially because (as discussed elsewhere), it doesn't show the exception.

0
Comment actions Permalink

Unit tests work fine for me in NUnit 3.12.0 however any time I try to run tests with coverage, I get "Exit code 134" on all my tests and they are all marked as inconclusive.

Am I just doing something wrong here or should I submit a ticket?

0
Comment actions Permalink

Hi @John,

It seems that this is a Catalina OSX + Rider 2019.2 case. If so, please try Rider.2019.2.3 or Rider 2019.3EAP and let me know about the results. This issue is known and fixed: https://youtrack.jetbrains.com/issue/RIDER-33547.
If it does not help, we would be grateful if you file an issue on youtrack with collected logs.

 

Sofia.

0
Comment actions Permalink

We are unable to run tests without rebuilding all dependencies for dotnet core projects. Our integration test project depends on a huge number of other related projects and every time someone tries to run tests they have to wait for about 1 minute for the build to finish, despite the fact that no code has been changed. 

0

Please sign in to leave a comment.

Have more questions?

Submit a request