Problems with .NET unit testing

 

JetBrains Rider automatically detects unit tests and provides a unit test runner to run and debug them. We are continuously improving unit testing with JetBrains Rider, but there are still some areas which are not supported.

So what is currently supported and what is not? 

  Supported since 2017.2
Unit testing Frameworks

xUnit

NUnit

Classic MSTest (Windows-only)

MSTest v2 (.NET Core projects - cross-platform, .NET Framework projects - Windows-only)

Runtime

.NET Framework

.NET Core

Mono (4.8+ recommended)

If you experience any issues with a supported unit testing framework on a supported runtime, please create a request and describe the problem in details, as it will help to sort it out timely. The following information is essential to determine the cause of the issue:

Environment

  1. OS version, for example, Windows 10 or Mac OS X 10.8
  2. JetBrains Rider version: Help -> About
  3. Unit testing framework version, for example, xUnit 2.2.0
  4. MSBuild version: Settings -> Build, Execution, Deployment -> Toolset and Build (find screenshot below)
  5. .NET Core version in case you have .NET Core project:
    • Check a path to dotnet executable via Settings -> Build, Execution, Deployment -> Toolset and Build (find screenshot below)
    • Go to the directory with dotnet executable via terminal
    • Run "dotnet --version" command

Log Files

  1. Build your solution and then collect Rider logs: Help -> Collect Logs
  2. Run Unit Tests if you can see them in unit test explorer 
    • Get launch log files: Launch settings -> Show Last Launch Log (find screenshot below) or click on gear icon -> Diagnostics -> Show Last Launch Log
    • Get unit test process log files as described here
  3. Attach the logs to the request

Screenshot_2020-06-03_at_17.57.49.png

 

Additional information

  1. Sample solution where the issue reproduces is very much appreciated
  2. Can you see your unit tests in unit test explorer?
  3. Are there any steps to reproduce the issue?
  4. Has it worked in the previous version of JetBrains Rider?

 Thank you for helping us make JetBrains Rider better!

4 out of 17 found this helpful
42 comments
Avatar
Permanently deleted user

The marketing page for Rider quotes NUnit test support. Is this only for .Net Framework and not .Net Core projects? There is no template for a .Net core Nunit test project: 

For now I am using NunitLite to debug and run tests.

Edited by Permanently deleted user
0

@Alex
NUnit dotnetcore is not working in the current release.

However it is already fixed in Re# and will get to Rider 2017.2.

https://youtrack.jetbrains.com/issue/RSRP-464912#comment=27-2312068

0
Avatar
Permanently deleted user

To add to this, in Rider 2018.1.3 the settings within the unit test runner settings (see screenshot) are not respected in the unit test runner itself. The "Automatic" option was still selected while the platform architecture was set to x64.

2

Kevin,

Thanks for your feedback! Do I understand correctly that you've changed Platform Type in the NuGet tool window (screenshot 2), but it was not changed accordingly in Unit Testing settings (screenshot 1)?

0
Avatar
Permanently deleted user

Not quite, the second window was the unit test settings. Before taking the screenshot it looks like I'd already fixed the setting, but the unit test tools (window?) didn't use the setting I'd set in the Settings window for the architecture. I took the screenshot after manually setting the arch to x64 and verifying it worked.

0

I seem to be unable to debug unit tests in

JetBrains Rider 2018.2.3
Build #RD-182.4231.496, built on September 13, 2018
Licensed to Gary Woodfine
Subscription is active until November 5, 2018
JRE: 1.8.0_152-release-1248-b8 amd64
JVM: OpenJDK 64-Bit Server VM by JetBrains s.r.o
Linux 4.15.0-36-generic

The debugging of Unit Tests was working this morning but it seems to just randomly stopped working.  I have shut down and restarted rider,  rebooted machine. bu the problem keeps recurring.  I am actually able to Run the tests, using Rider and Dotnet Test.  but when I try debug,

I Get Exception below.

I have tried to CLEAN , RESTORE and Rebuild several times.  I created a blank file api.tests.xunit.runner.json

 

System.IO.FileNotFoundException: Could not find file '/home/gary/code/rea-development/backend/candidate/tests/bin/Debug/netcoreapp2.1/api.tests.xunit.runner.json'.
File name: '/home/gary/code/rea-development/backend/candidate/tests/bin/Debug/netcoreapp2.1/api.tests.xunit.runner.json'
at Interop.ThrowExceptionForIoErrno(ErrorInfo errorInfo, String path, Boolean isDirectory, Func`2 errorRewriter)
at Interop.CheckIo(Error error, String path, Boolean isDirectory, Func`2 errorRewriter)
at Microsoft.Win32.SafeHandles.SafeFileHandle.Open(String path, OpenFlags flags, Int32 mode)
at System.IO.FileStream.OpenHandle(FileMode mode, FileShare share, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, FileOptions options)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share, Int32 bufferSize, Boolean useAsync)
at System.IO.FileStream..ctor(String path, FileMode mode, FileAccess access, FileShare share)
at System.IO.File.OpenRead(String path)
at Xunit.ConfigReader_Json.File_OpenRead(String path) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Configuration\ConfigReader_Json.cs:line 172
at Xunit.ConfigReader_Json.LoadFile(String configFileName) in C:\Dev\xunit\xunit\src\xunit.runner.utility\Configuration\ConfigReader_Json.cs:line 141
Edited by gary woodfine
1

Unable to edit the above comment 

But I am using .net core 2.1 

I noticed the debug output had some additional info.  I am going to attempt to recreate the unit project and see if the problem recurs

I tried creating a Fresh Unit Test project Using dotnet new xunit and wrote a simple unit test and the problem recurs.

 

Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.MemoryMappedFiles.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.IO.Compression.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Buffers.dll was not found or failed to read
Loaded Assembly '/usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll'
Loading module /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll in application domain 1:clrhost
Pdb file for assembly /usr/share/dotnet/shared/Microsoft.NETCore.App/2.1.5/System.Text.Encoding.Extensions.dll was not found or failed to read
Edited by gary woodfine
0

How about removing 

/usr/share/dotnet/shared/Microsoft.NETCore.App 
And letting Rider or dotnet restore to restore it again?
0

Not sure where exactly where you want me to do this?

What I have tried is basically

sudo apt remove --purge dotnet*

Then I completely reinstalled dotnet using

sudo sh -c 'echo "deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-bionic-prod bionic main" > /etc/apt/sources.list.d/dotnetdev.list'

sudo apt-get update
sudo apt-get install dotnet-sdk-2.1 

Howeever the problem remains

In the Rider Settings --> Build ... -> Toolset and Build

and reset all the options to custom then  dotnet restore  and try test run debug i still get the error

 

 

 

Edited by gary woodfine
0

I meant deleting folder in the file-system:

/usr/share/dotnet/shared/Microsoft.NETCore.App 
1

Ok Cool.

After some more investigating yesterday, and trying out other projects which I have Unit Tests in. It is only one project which has the issue.

This leads me to believe that there must be something about this particular project's configuration which may be a little irregular. I'll do some more digging this morning and get back to you.

 

0

 

Interesting!  I have just tried a dotnet build on the solution.

I found a couple of warnings that point to the Unit Test Project. 

Do you know where the build log is stored? so I can view it?   I can't seem to find it

/usr/share/dotnet/sdk/2.1.403/Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "Microsoft.Extensions.DependencyModel" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/gary/code/rea-development/backend/candidate/tests/api.tests/api.tests.csproj]
api.tests -> /home/gary/code/rea-development/backend/candidate/tests/api.tests/bin/Debug/netcoreapp2.1/api.tests.dll

Build succeeded.

/usr/share/dotnet/sdk/2.1.403/Microsoft.Common.CurrentVersion.targets(2110,5): warning MSB3277: Found conflicts between different versions of "Microsoft.Extensions.DependencyModel" that could not be resolved. These reference conflicts are listed in the build log when log verbosity is set to detailed. [/home/gary/code/rea-development/backend/candidate/tests/api.tests/api.tests.csproj]
1 Warning(s)
0 Error(s)

 

Edited by gary woodfine
0

It should be "Open log file" in the Build Toolwindow.


1

OK Cool!   Honestly didn't think of that!!   Thats a cool feature I was blissfully unaware of!  Thanks for letting me know

0
Avatar
Permanently deleted user

@gary woodfine did you ever figure this out? I'm running into the exact same issue: can't debug an xunit test.

0

I'm afraid I haven't as yet!

I still have the problem, I have a suspicion that it has something to do with referencing the API project, which is developed in .net core 2.1. I am able to debug other unit test projects within the same solution, and other solutions.   The only anomoly I can identify is that the particular project I have an issue with references a .net core API 2.1 project!

I'm not sure if this is correct, it's the only conclusion I can jump too.  I am still waiting for feedback

What project type are you having an issue with?

Hope this helps

0

Unfortunately it seems, that Rider does not fully support non-deterministic xUnit tests, like ones based on AutoFixture. It seems that it's a problem with pre-enumeration.

I've built a small example at https://github.com/drivenet/rider-xunit-discovery-problem

Load solution, restore packages, build the project.

Now, run all tests -- everything seems OK.

Now, try to run the "DetectedProperly" test only -- it works.

Then -- try to run the "NotDetectedProperly" test only -- it either becomes "Unknown, test nor run" or "Inconclusive".

 

Should I file a bug somewhere?

1
Avatar
Permanently deleted user

Encountered the same problem as Gary. On newly created xUnit test project I started getting those File not found exceptions when I was trying to debug a test (any test in the project)

Creating a new xUnit project (in the same solution) didn't help. 

I went to the Run -> Stop on exception configuration and re-added the System.Exception. Pop up appeared saying that the exception type is already added. This was expected behaviour.

What was unexpected that after this I was able to F5 thru the FileNotFound exception and after that a break point was hit in the test method...

0

gary woodfineLukas Belovic

The issue was that the debugger didn't respect the "Just my code" setting in exception breakpoints. The workaround for Rider 2018.2.3 is either disable exception breakpoints (Run -> View Breakpoints) or step through all FileNotFound exceptions. In Rider 2018.3, we've fixed "Just my code" setting. If it is turned on, Rider won't stop on the exceptions thrown by xUnit.

Hope this helps!

0
Avatar
Permanently deleted user

I've run into an issue after upgrading. It appears as if the "Run <n> number of assemblies in parallel" option is not being honored. I have some Nunit test assemblies that cannot be run in parallel so I have that setting set to 1. In previous versions, my tests worked out fine. Since upgrading I'm getting massive test failures that seem to indicate that some of these assemblies are either not being unloaded properly. I would like to find out if there is a work around or if anyone has seen this issue.

1

I'm having this issue on 2019.1.1 on Mac OS X with .NET Core 2.2. I can confirm that it's stopping on intermediate code. To be fair, I told it to, but it's really confusing during startup. Especially because (as discussed elsewhere), it doesn't show the exception.

0

Unit tests work fine for me in NUnit 3.12.0 however any time I try to run tests with coverage, I get "Exit code 134" on all my tests and they are all marked as inconclusive.

Am I just doing something wrong here or should I submit a ticket?

0

Hi @John,

It seems that this is a Catalina OSX + Rider 2019.2 case. If so, please try Rider.2019.2.3 or Rider 2019.3EAP and let me know about the results. This issue is known and fixed: https://youtrack.jetbrains.com/issue/RIDER-33547.
If it does not help, we would be grateful if you file an issue on youtrack with collected logs.

 

Sofia.

0

We are unable to run tests without rebuilding all dependencies for dotnet core projects. Our integration test project depends on a huge number of other related projects and every time someone tries to run tests they have to wait for about 1 minute for the build to finish, despite the fact that no code has been changed. 

0

Somehow in my windows machine if you have a global.json configuration it can make the rider works weirdly, you can try to delete it if you have and see if it make it works

Edited by Diegocostacolli
0
Avatar
Permanently deleted user

Unit tests would fail to start with error 134, and I had to reinstall the app to get it to work. No upgrade, already using the latest build. More details below.

 

```

Exit code is 134 (Unhandled exception. System.BadImageFormatException: Could not load file or assembly '/Applications/Rider.app/Contents/lib/ReSharper...

Last runner error: Process /usr/local/share/dotnet/dotnet:14817 exited with code '134': Unhandled exception. System.BadImageFormatException: Could not load file or assembly '/Applications/Rider.app/Contents/lib/ReSharperHost/TestRunner/netcoreapp2.0/ReSharperTestRunner32.dll'. An attempt was made to load a program with an incorrect format. File name: '/Applications/Rider.app/Contents/lib/ReSharperHost/TestRunner/netcoreapp2.0/ReSharperTestRunner32.dll'

```

This issue started today without any particular reason, I was just in the middle of a coding task, running multiple solutions, with multiple Rider instances. I tried to clean the repo, removing .idea, obj/bin folders, etc. I also uninstalled .NET 2.2x just in case.  The .NET app was working fine, but unit tests would fail to start. Every project is based on .NET Core 3.1 but tests seems to be looking for netcoreapp2.0

JetBrains Rider 2020.2
Build #RD-202.6397.244, built on August 10, 2020
Runtime version: 11.0.7+10-b944.20 x86_64
VM: OpenJDK 64-Bit Server VM by JetBrains s.r.o.
.NET Core 3.1.6
macOS 10.15.6

7

Same problem as @Devis described, with the same error message and a .net core 3.1 project.

Rider 2020.2 on MacOS 10.14.5. Tests were running fine then out of nowhere started failing with that exception.

Running Unit Tests with Coverage resolved the problem for the time being.

Edited by Nikola
2

Please sign in to leave a comment.

Have more questions?

Submit a request