Problems with .NET unit testing

 

As of version 2018.1, JetBrains Rider automatically detects unit tests and provides a unit test runner to run and debug them. We are continuously improving unit testing with JetBrains Rider, but there are still some areas which are not supported.

So what is currently supported and what is not? 

  Supported since 2017.2
Unit testing Frameworks

xUnit

NUnit

Classic MSTest (Windows only)

MSTest v2

Runtime

.NET Framework

.NET Core

Mono (4.8+ recommended)

If you experience any issues with a supported unit testing framework on a supported runtime, please create a request and describe the problem in details, as it will help to sort it out timely. The following information is essential to determine the cause of the issue:

Environment

  1. OS version, for example, Windows 10 or Mac OS X 10.8
  2. JetBrains Rider version: Help -> About
  3. Unit testing framework version, for example, xUnit 2.2.0
  4. MSBuild version: Settings -> Build, Execution, Deployment -> Toolset and Build (find screenshot below)
  5. .NET Core version in case you have .NET Core project:
    • Check a path to dotnet executable via Settings -> Build, Execution, Deployment -> Toolset and Build (find screenshot below)
    • Go to the directory with dotnet executable via terminal
    • Run "dotnet --version" command

Log Files

  1. Build your solution and then get 'backend.log' file: Help -> Show Log in Explorer/Finder/File Manager
  2. Run Unit Tests if you can see them in unit test explorer 
    • Get launch log files: Launch settings -> Show Last Launch Log (find screenshot below)
    • Get unit test process log files as described here
  3. Zip these three log files and attach the archive to the request

 

Additional information

  1. Sample solution where the issue reproduces is very much appreciated
  2. Can you see your unit tests in unit test explorer?
  3. Are there any steps to reproduce the issue?
  4. Has it worked in the previous version of JetBrains Rider?

 Thank you for helping us make JetBrains Rider better!

20 comments

The marketing page for Rider quotes NUnit test support. Is this only for .Net Framework and not .Net Core projects? There is no template for a .Net core Nunit test project: 

For now I am using NunitLite to debug and run tests.

Edited by Alex Taroghion
0

@Alex
NUnit dotnetcore is not working in the current release.

However it is already fixed in Re# and will get to Rider 2017.2.

https://youtrack.jetbrains.com/issue/RSRP-464912#comment=27-2312068

0

To add to this, in Rider 2018.1.3 the settings within the unit test runner settings (see screenshot) are not respected in the unit test runner itself. The "Automatic" option was still selected while the platform architecture was set to x64.

0

Kevin,

Thanks for your feedback! Do I understand correctly that you've changed Platform Type in the NuGet tool window (screenshot 2), but it was not changed accordingly in Unit Testing settings (screenshot 1)?

0

Not quite, the second window was the unit test settings. Before taking the screenshot it looks like I'd already fixed the setting, but the unit test tools (window?) didn't use the setting I'd set in the Settings window for the architecture. I took the screenshot after manually setting the arch to x64 and verifying it worked.

0

I seem to be unable to debug unit tests in

JetBrains Rider 2018.2.3
Build #RD-182.4231.496, built on September 13, 2018
Licensed to Gary Woodfine
Subscription is active until November 5, 2018
JRE: 1.8.0_152-release-1248-b8 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Linux 4.15.0-36-generic

The debugging of Unit Tests was working this morning but it seems to just randomly stopped working.  I have shut down and restarted rider,  rebooted machine. bu the problem keeps recurring.  I am actually able to Run the tests, using Rider and Dotnet Test.  but when I try debug,

I Get Exception below.

I have tried to CLEAN , RESTORE and Rebuild several times.  I created a blank file api.tests.xunit.runner.json

 

System.IO.FileNotFoundException: Could not find file '/home/gary/code/rea-development/backend/candidate/tests/bin/Debug/netcoreapp2.1/api.tests.xunit.runner.json'.
File name: '/home/gary/code/rea-development/backend/candidate/tests/bin/Debug/netcoreapp2.1/api.tests.xunit.runner.json'
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Interop.CheckIo(Error error, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.IO.File.OpenRead(String path)
at Xunit.ConfigReader_Json.File_OpenRead(String path) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Configuration\ConfigReader_Json.cs:line 172
at Xunit.ConfigReader_Json.LoadFile(String configFileName) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Configuration\ConfigReader_Json.cs:line 141
Edited by gary woodfine
1

Unable to edit the above comment 

But I am using .net core 2.1 

I noticed the debug output had some additional info.  I am going to attempt to recreate the unit project and see if the problem recurs

I tried creating a Fresh Unit Test project Using dotnet new xunit and wrote a simple unit test and the problem recurs.

 

Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.MemoryMappedFiles.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll was not found or failed to read
Edited by gary woodfine
0

How about removing 

/usr/share/dotnet/shared/Microsoft.NETCore.App 
And letting Rider or dotnet restore to restore it again?
0

Not sure where exactly where you want me to do this?

What I have tried is basically

sudo apt remove --purge dotnet*

Then I completely reinstalled dotnet using

sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod bionic main" > /etc/apt/sources.list.d/dotnetdev.list'

sudo apt-get update
sudo apt-get install dotnet-sdk-2.1 

Howeever the problem remains

In the Rider Settings --> Build ... -> Toolset and Build

and reset all the options to custom then  dotnet restore  and try test run debug i still get the error

 

 

 

Edited by gary woodfine
0

I meant deleting folder in the file-system:

/usr/share/dotnet/shared/Microsoft.NETCore.App 
1

Ok Cool.

After some more investigating yesterday, and trying out other projects which I have Unit Tests in. It is only one project which has the issue.

This leads me to believe that there must be something about this particular project's configuration which may be a little irregular. I'll do some more digging this morning and get back to you.

 

0

 

Interesting!  I have just tried a dotnet build on the solution.

I found a couple of warnings that point to the Unit Test Project. 

Do you know where the build log is stored? so I can view it?   I can't seem to find it

/usr/share/dotnet/sdk/2.1.403/Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "Microsoft.Extensions.DependencyModel" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/gary/code/rea-development/backend/candidate/tests/api.tests/api.tests.csproj]
api.tests -> /home/gary/code/rea-development/backend/candidate/tests/api.tests/bin/Debug/netcoreapp2.1/api.tests.dll

Build succeeded.

/usr/share/dotnet/sdk/2.1.403/Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "Microsoft.Extensions.DependencyModel" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/gary/code/rea-development/backend/candidate/tests/api.tests/api.tests.csproj]
1 Warning(s)
0 Error(s)

 

Edited by gary woodfine
0

It should be "Open log file" in the Build Toolwindow.


1

OK Cool!   Honestly didn't think of that!!   Thats a cool feature I was blissfully unaware of!  Thanks for letting me know

0

@gary woodfine did you ever figure this out? I'm running into the exact same issue: can't debug an xunit test.

0

I'm afraid I haven't as yet!

I still have the problem, I have a suspicion that it has something to do with referencing the API project, which is developed in .net core 2.1. I am able to debug other unit test projects within the same solution, and other solutions.   The only anomoly I can identify is that the particular project I have an issue with references a .net core API 2.1 project!

I'm not sure if this is correct, it's the only conclusion I can jump too.  I am still waiting for feedback

What project type are you having an issue with?

Hope this helps

0

Unfortunately it seems, that Rider does not fully support non-deterministic xUnit tests, like ones based on AutoFixture. It seems that it's a problem with pre-enumeration.

I've built a small example at https://github.com/drivenet/rider-xunit-discovery-problem

Load solution, restore packages, build the project.

Now, run all tests -- everything seems OK.

Now, try to run the "DetectedProperly" test only -- it works.

Then -- try to run the "NotDetectedProperly" test only -- it either becomes "Unknown, test nor run" or "Inconclusive".

 

Should I file a bug somewhere?

0

Encountered the same problem as Gary. On newly created xUnit test project I started getting those File not found exceptions when I was trying to debug a test (any test in the project)

Creating a new xUnit project (in the same solution) didn't help. 

I went to the Run -> Stop on exception configuration and re-added the System.Exception. Pop up appeared saying that the exception type is already added. This was expected behaviour.

What was unexpected that after this I was able to F5 thru the FileNotFound exception and after that a break point was hit in the test method...

0

Please sign in to leave a comment.

Have more questions?

Submit a request