Tests become undiscoverable until rebuild

[GrimaceOfDespair]

I've got at least 2 solutions (1 with about 6 projects and another one with about 30 projects) in which this problem occurs. I use the NUnit NuGet adapter to run tests in VS 2015. During developing, every now and then, after a build, tests disappear from the Test Explorer. Only a subsequent rebuild fixes the issue.

It seems more people are suffering from this: http://stackoverflow.com/questions/35589927/why-do-nunit-tests-frequently-disappear-in-visual-studio-2015

[ChrisMaddock]

Is this the NUnit3 adapter, as per the SO question you've linked to?

Which version of the adapter are you using? The issue in that StackOverflow question was fixed in nunit/nunit3-vs-adapter#126, and should be fixed in v3.0.10 of the adapter. If you're on that, it may be a separate issue.

[GrimaceOfDespair]

I'm actually using NUnit 2.6 on NUnitTestAdapter.2.0.0, so maybe the fix was "just" not backported?

[GrimaceOfDespair]

I'm open to aid in resolving this issue. The issue on the NUnit3 adapter has no direct links to the code changes that comprehend the fix, which makes it a bit tough when you're not acquainted with the code base.

Feel free to point me in any direction though.

[CharliePoole]

The bug under the NUnit 3 adapter was never actually worked on, which is why there is no code pointed to. Rather, there were a lot of changes going on at the same time and we basically discovered that it had been fixed by one of them. We never analyzed to figure out why.

However, even if we had the code, we wouldn't necessarily "backport" it to the NUnit Adapter. The reason is that the two adapters are two different products and the code similarities are limited. The reason they are two products, btw, is that we wanted to keep supporting both of them with fixes as needed. Our practice for older releases of the same product is to stop supporting it in that way.

So, we will need for somebody to confirm the bug, preferably by providing a reproducible case. Then somebody can pick it up and try to fix it.

As a second front for attacking the problem, someone could go through the 40 or so commits we made to the NUnit 3 adapter between 3.0.8 and 3.0.9 and identify candidates for porting to this adapter.

[GrimaceOfDespair]

Ok, tnx for the info. I'll post a link here for my own reference purposes and for any extra pair of eyes that might want to have a look:

nunit/nunit3-vs-adapter@3.0-ctp-8...3.0-ctp-9

[GrimaceOfDespair]

It looks like the most important change was DiscoverTests which had a pretty straightforward refactoring comprising 2 changes.

The first one takes the GetRunnerFor out of a try..finally block, and the second one is about the detection of missing fixtures:

if (msgNode != null && msgNode.GetAttribute("value").Contains("contains no tests"))
  TestLog.SendWarningMessage("Assembly contains no NUnit 3.0 tests: " + sourceAssembly);

became

if (msgNode != null && (new[] { "contains no tests", "Has no TestFixtures" }).Any(msgNode.GetAttribute("value").Contains))
  TestLog.SendInformationalMessage("Assembly contains no NUnit 3.0 tests: " + sourceAssembly);

Then, there's also a change on AsyncMethodHelper:

var method = definingType.GetMethod(methodName);

Became

var method = definingType.GetMethods().Where(o => o.Name == methodName).OrderBy(o => o.GetParameters().Length).FirstOrDefault();

[GrimaceOfDespair]

I've got some more info on this issue, and a clue as to what might be happening.

It seems that sometimes, after a build, the adapter kicks in before the whole bin is written. The effect is that the assembly is not read thus no tests are detected. However, eventually, the whole bin does get written, but only after the adapter has done it's scan.

Subsequently, issuing a build again after applying no code changes, the project will not be rebuilt, hence the adapter will not try to scan the assemblies, because they have not changed.

Making a code change to force a rebuild of the project solves the issue in that case. The following is output whenever the described scenario is happening.

------ Discover test started ------
NUnit VS Adapter 2.0.0.0 discovering tests is started
NUnit VS Adapter 2.0.0.0 discovering tests is started
Dependent Assembly FooBar.Test.dll of C:\Projects\FooBar.Test\bin\Debug\FooBar.Test.dll not found. Can be ignored if not a NUnit project.
NUnit VS Adapter 2.0.0.0 discovering test is finished
NUnit VS Adapter 2.0.0.0 discovering test is finished
========== Discover test finished: 0 found (0:00:00.1899422) ==========

[CharliePoole]

If that's true, it seems like it must be a Test Explorer rather than an adapter bug. The adapter only starts discovery when it's called by VS.

[CharliePoole]

On the other hand, we think it was resolved for the NUnit 3 adapter due to some change made. Hmm...

[GrimaceOfDespair]

Unless it's actually a different bug?

[GrimaceOfDespair]

Fyi: I kept suffering from this on a consistent enough basis, that I switched to NUnit 3. The NUnit 3 adapter does not expose this behaviour, mabye thanks to the aforementioned fix.

For me, it's a post mortem, but fwiw, this was my flow:

• Build
• Run tests
• Change some code (not in the test project)
• Try to run test again

The last step would trigger a build of both the project and the test project. After the build, VS would have forgotten about my test, only to be resolved with a retest.

Particularly in the case of dynamically parameterized tests, this was an issue, because it prevented me from running a specific test after a bug fix.