Urgent <> Unable to get any mutation score and stryker is skipping all run - Stryker dot net

[inishantmishra]

Hi I am running following script to run in my pipeline-
dotnet-stryker --test-project "**/*Test.csproj" -m --with-baseline:$(System.PullRequest.TargetBranch) --version $(System.PullRequest.SourceBranch) --open-report --reporter "html" --reporter "progress" --reporter "cleartext" --break-at 80

But I am not getting the score and not getting any result. It seems all mutants are ignore but i haven't added any exclusion logic neither in comments nor in command.

Here are some snippet from logs

[17:30:58 WRN] Failed to load analyzer 'Microsoft.Extensions.Logging.Generators': Exception System.IO.FileLoadException: Assembly with same name is already loaded
at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.AnalyzerAssemblyLoader.LoadFromPath(String fullPath) in //src/Stryker.Core/Stryker.Core/Initialisation/Buildalyzer/IAnalyzerResultExtensions.cs:line 126
at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.GetAssembly()
at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.Extensions1.AddExtensions(Builder builder, String language, Func2 shouldInclude).
[17:30:58 WRN] Failed to load analyzer 'Microsoft.Extensions.Options.SourceGeneration': Assembly with same name is already loaded (error : UnableToLoadAnalyzer, analyzer: All).
[17:30:58 WRN] Failed to load analyzer 'Microsoft.Extensions.Options.SourceGeneration': Exception System.IO.FileLoadException: Assembly with same name is already loaded
at System.Runtime.Loader.AssemblyLoadContext.LoadFromAssemblyPath(String assemblyPath)
at System.Reflection.Assembly.LoadFrom(String assemblyFile)
at Stryker.Core.Initialisation.Buildalyzer.IAnalyzerResultExtensions.AnalyzerAssemblyLoader.LoadFromPath(String fullPath) in //src/Stryker.Core/Stryker.Core/Initialisation/Buildalyzer/IAnalyzerResultExtensions.cs:line 126
at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.GetAssembly()
at Microsoft.CodeAnalysis.Diagnostics.AnalyzerFileReference.Extensions1.AddExtensions(Builder builder, String language, Func2 shouldInclude).
[17:31:34 INF] 6417 mutants created
[17:31:34 INF] Capture mutant coverage using 'CoverageBasedTest' mode.
[17:34:36 INF] 160 mutants got status CompileError. Reason: Mutant caused compile errors
[17:34:36 INF] 596 mutants got status Ignored. Reason: Removed by exclude from code coverage filter
[17:34:36 INF] 5661 mutants got status Ignored. Reason: Removed by mutate filter
[17:34:36 INF] 6417 total mutants are skipped for the above mentioned reasons
[17:34:36 INF] 0 total mutants will be tested
[17:34:36 WRN] It looks like all mutants with tests were ignored. Try a re-run with less ignoring!
[17:34:36 WRN] It looks like all mutants resulted in compile errors. Mutants sure are strange!

Killed: 0
Survived: 0
Timeout: 0

All mutants have been tested, and your mutation score has been calculated

Please help and let me know what is happening or is there anything wrong am i doing?

[rouke-broersma]

You're supplying the option -m without argument, so nothing is mutation tested.

[inishantmishra]

Now I am getting this
[16:36:30 INF] We could not locate a baseline for branch refs/heads/LocalDevelopment, now trying fallback version refs/heads/develop
[16:36:30 INF] We could not locate a baseline report for the current branch, version or fallback version. Now running a complete test to establish a fresh baseline.
[16:36:30 INF] Time Elapsed 00:09:47.9098005
Stryker.NET failed to mutate your project. For more information see the logs
below:

No branch or tag or commit found with given target refs/heads/develop. Please
provide a different GitDiffTarget.

[rouke-broersma]

Your pipeline is probably doing shallow fetch so does not include all refs and this Stryker cannot find the ref you specified

[inishantmishra]

@rouke-broersma
I just want to run mutation test on the feature branch which i have raised as PullRequest to merge the develop branch which is target branch.

There is no mutation ran ever on develop and there is no report stored anywhere. Is it important to store the baseline somewhere? Can I store it in the repo itself?

Can you please explain me in steps what exactly should i do and how can I run mutation tests only for the changes i have done in my feature branch instead of whole develop branch? So that i can test the newly changes not the old source codes.

[rouke-broersma]

If you don't care about storing the report, use the since feature not the baseline feature

[inishantmishra]

So I just need to pass feature branch name or the develop branch name?
Will following syntax work?
--since localDevelopment

[inishantmishra]

@rouke-broersma
Also I am not able to understand your comment

Your pipeline is probably doing shallow fetch so does not include all refs and this Stryker cannot find the ref you specified

[dupdob]

@inishantmishra : CI pipeline works on a light copy of the git repository (via this option https://git-scm.com/docs/git-fetch#Documentation/git-fetch.txt---depthltdepthgt). Which makes sense from a CI pipeline which only builds the software, but if git history is missing, Stryker is not able to identify the reference point. And no way to identify which parts of the code have been changed. Stryker can't do anything about that

You need to look on how to fetch a full git history (or a deep enough one) on your pipeline.

Is it clearer?

[inishantmishra]

@dupdob @rouke-broersma

I was able to run the striker run. but need help on these 2 queries-
1> Why lot of mutations are getting ignored either because of block or because of equality operator etc. How can I make sure these should not get ignored.

2. How can I use the since operator to run the mutations only on the files which are changed in feature branch which I have raised as PullRequest to merge in target branch-develop. I want to run this as PR Build on feature branch

[dupdob]

Regarding ignored mutations: Stryker automatically ignores block mutations (replacing a block with an empty block) when there are mutations in the block; the assumption is that there would be no situation where mutations inside the block are killed while the block mutation survives. This implies these mutations have little marginal value so Stryker skips them to save on test time.

[inishantmishra]

@dupdob @rouke-broersma

Stryker takes more than 9 minutes to run the mutation. And I think once the repo or test cases will increase it will take more. How can we reduce the time? How can we make it fast?

How does it run by default?

[dupdob]

9 minutes is quite fast actually.

Mutation testing is slow by essence.

The best approach is to look on how to make tests faster.

Alternatively, you can restrict which mutators are active and reduce the # of mutations, but it has obvious downsides

[inishantmishra]

Any way to make it faster? Cant we do some byte code change not the actual code change in the mutation process to make it faster?

[inishantmishra]

And how to filter out time taking tests like integration tests from running mutations on it?

[rouke-broersma]

Any way to make it faster? Cant we do some byte code change not the actual code change in the mutation process to make it faster?

Byte code change would not make this any faster. We place all mutants at once. Mutation testing takes a lot of time because we have to run your unit tests multiple times, that's inherent to mutation testing. This would not be different for byte code mutations.

[rouke-broersma]

And how to filter out time taking tests like integration tests from running mutations on it?

Sure, use test filters like you normally would. Stryker supports this.

[inishantmishra]

@rouke-broersma Any reference, how to use test filters?

[rouke-broersma]

@rouke-broersma Any reference, how to use test filters?

https://stryker-mutator.io/docs/stryker-net/configuration/#test-case-filter-string

[inishantmishra]

@rouke-broersma @dupdob

Can you guys please help me in understanding how to use baseline report for comparison during PR build check. Lets say I have a baseline report stored in develop branch of repo.
Now when someone creates the PR from feature branch how to check only the different and compare it with baseline report of develop branch.

And how will it the baseline report of develop branch will get updated lets say if featre branch is merged with develop and also think of some scenarios if feature branch is not merged and PR is cancelled.

How exactly the baseline feature works?

[rouke-broersma]

1> Generate baseline report for develop branch and store it (please tell the location on my repo what should be the folder path)

Storing the baseline in the repository is not natively supported by stryker. You could use the disk provider and add a commit && push step in your pipeline manually to make this work.

4> Now, for both PRs it will run the mutation on whole solution again for 40 minutes with baseline report of develop.

No, your PRs will use the baseline from develop, and only test code changed since that baseline.

5> Now, post run, if PR1 gets merged into develop. I need to run some command to regenerate the baseline report for develop which will take again 40 minutes.

You are in control of when you update the develop baseline. You could do this for example nightly instead of on every merge. Up to you and your use case.

The major issue with since is that lets say there is a file in develop branch called Class1.cs which has 3 methods. Now, if i added a new method 4th method under class and I raise the PR, it will do mutation run on whole file and will increase the threshold. But i wanted to exclude those existing methods and only wanted to run for newly changed code or added code.

Since will only test the new method, not everything in the class. The only difference between Baseline and Since is that Baseline can give you a full report even when it only tested changed code, by reusing an existing result. Since will only report on the changes. Other than that they do the same thing.

[inishantmishra]

@rouke-broersma SO If I am right, in the above example, --since will test only Method4 of class and wont use whole class to mutate and run tests?

But why is it taking into consideration for these constructor blocks for mutations even I have used following config to ignore?

And lets say I want to prevent to allow a single survival mutant to pass through. What should be the threshold to break? Should i implement on break at 0 what does it mean?

[dupdob]

Regarding the ignore-methodsoption: it actually ignore method calls, not the method themselves. You can ignore specific methods, including constructors or even a whole class, using Stryker comments (see here for complete documentation on ignore logic).
There is no existing way to ignore mutations in every constructor as of today.

[inishantmishra]

Hi, But what about -
Lets say I want to prevent to allow a single survival mutant to pass through. What should be the threshold to break? Should i implement on break at 0, what will that mean?

[rouke-broersma]

Threshold break 0 means you cannot have any surviving mutants (in the new code) yes

[inishantmishra]

Hi @rouke-broersma @dupdob

You guys said this-
Since will only test the new method, not everything in the class. The only difference between Baseline and Since is that Baseline can give you a full report even when it only tested changed code, by reusing an existing result. Since will only report on the changes. Other than that they do the same thing.

But my striker when runs mutates the code which was in develop branch since beginning and is not changed in feature branch. And it is referring some of survived mutants from those methods.. How to ignore those methods. I cant go ahead specify manually to ignore those old methods. Why doesn't since automatically ignores the methods of develop branch and only runs on feature branch and that too on methods which is being added new or modified?

What should I do?

[rouke-broersma]

The logs will be in your console in your CI

[inishantmishra]

But What should I see under the log? What should i see to know why the non modified changes are also mutated? Because i can see those methods being mutated in the logs which should not be because no one has modified those methods.
@rouke-broersma

[inishantmishra]

@rouke-broersma @dupdob Anything on this?

[inishantmishra]

Anyone here? Can anyone help me on this?

[dupdob]

I am not familiar withsince nor baseline options, I prefer full run. So I may be wrong.
But, here is the gist of the since logic:

1. If a source file has been modified since the reference point, every mutant within this file will be tested again. You will see Returning all mutants in (nameoffile) because the file is modified.
2. if a test source file has been modified since the reference point, every mutant covered by a test in this file will be tested again. Sadly, this is not reported in log. This is a bug.
3. if another file has been modified (some content file), everything will be tested again as Stryker cannot determine the impacts this change may have. It appears in the log as : Returning all mutants in (nameoffile) because a non-source file is modified

In conclusion, I am not sure there is a feature that achieves what you are trying to do, which looks like: mutates only the bits of code that have changed.
Also, Since feature could use better logging 😄

[inishantmishra]

Hi @dupdob @rouke-broersma
I need help from you guyz related to baseline. Let me explain what am i doign and what i need help with.

I have create a baseline report in HTML as well in json format with name mutation-report.json and placed it over secured azure blob storage.

Now, I want to run mutation as Build Validation when someone raises the Pull request and it should run between the source branch of Pull Request and the baseline report which i had generated earlier.

So, what i do is that i have pipeline in azure devops which first creates a directory in in the agent at -> $(Agent.BuildDirectory) where the code is checked out and mutation is going to run. Director created will be -> StrykerReports/Baselines and here we will download the baseline report from blob storage and will place over here.

And now, post download we run mutation testing via the following command-
$(Agent.BuildDirectory)/tools/dotnet-stryker --test-project "**/*Test.csproj" --with-baseline "$(Agent.BuildDirectory)\StrykerReports\Baselines\mutation-report.json" --config-file StrykerOutput\stryker-config.json --reporter "html" --reporter "progress" --reporter "cleartext" --reporter "html" --verbosity trace

But i am keep on getting the error that unknown command. Followign is the error

Unrecognized command or argument 'D:\a\1\StrykerReports\Baselines\mutation-report.json'

Please help what should I do, i switched from since to baseline but again facing the issue.

[inishantmishra]

@rouke-broersma That time i was using --since. Now i thought to use baseline as you guys have told me this is the way via which i can run only on changed files compared to baseline and hence i can set the threshold as per baseline.

Logs contains the sourcecode assemblies and test cases which can be a security concern to share those here. So, let me know what exactly is needed in log so i can filter out those and share it with you?

[inishantmishra]

@rouke-broersma

This is one of important log i can give it you which informs what settings striker takes into consideration beforing running it. It logged on console while running mutations.

[19:13:55 DBG] Stryker started with options: {"MsBuildPath": null, "DevMode": false, "ProjectPath": "D:\\a\\1\\s\\TestProject", 
"IsSolutionContext": true, "WorkingDirectory": "D:\\a\\1\\s\\TestProject", "OutputPath": "D:\\a\\1\\s\\TestProject\\StrykerOutput\\2024-09-23.19-13-55", 
"ReportPath": "D:\\a\\1\\s\\TestProject\\StrykerOutput\\2024-09-23.19-13-55\\reports", 
"ReportFileName": "mutation-report", "SolutionPath": "D:\\a\\1\\s\\TestProject\\Test.sln", "TargetFramework": null, 
"LogOptions": {"LogToFile": false, "LogLevel": "Verbose", "$type": "LogOptions"}, 
"MutationLevel": "Standard", "Thresholds": {"High": 80, "Low": 60, "Break": 0, "$type": "Thresholds"}, 
"AdditionalTimeout": 5000, "LanguageVersion": "Default", "Concurrency": 1, "SourceProjectName": "", 
"TestProjects": ["D:\\a\\1\\s\\TestProject\\**\\*Test.csproj"], "TestCaseFilter": "(Category!=Integration)", 
"Reporters": ["Html", "Progress", "ClearText", "Baseline"], 
"WithBaseline": true, "BaselineProvider": "AzureFileStorage", "AzureFileStorageUrl": "https://teststorage.file.core.windows.net/filesharepath/StrykerOutput/Baselines", 
"AzureFileStorageSas": "azure file share sas token key", "DashboardUrl": "https://dashboard.stryker-mutator.io", "DashboardApiKey": null, 
"Since": true, "SinceTarget": "master", "DiffIgnoreChanges": [], "FallbackVersion": "master", "ModuleName": "", "ReportTypeToOpen": null, 
"Mutate": [{"Glob": {"Tokens": [{"TrailingPathSeparator": {"Value": "\\", "$type": "PathSeparatorToken"}, 
"LeadingPathSeparator": null, "$type": "WildcardDirectoryToken"}, {"$type": "WildcardToken"}], "$type": "Glob"}, 
"IsExclude": false, "TextSpans": [{"Start": 0, "End": 2147483647, "Length": 2147483647, "IsEmpty": false, "$type": "TextSpan"}], 
"$type": "FilePattern"}], 
"IgnoredMethods": [{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}, 
{"Options": "IgnoreCase", "RightToLeft": false, "MatchTimeout": "-00:00:00.0010000", "$type": "Regex"}], 
"ExcludedMutations": ["Statement"], "ExcludedLinqExpressions": [], "OptimizationMode": "CoverageBasedTest", 
"ProjectName": "", "ProjectVersion": "refs/heads/testagain", "BreakOnInitialTestFailure": false, "$type": "StrykerOptions"}

Now can i please can you take a look and solve my issue?

Also, I have changed my project name here for the security reasons.

[rouke-broersma]

As Dupdob said, look for the following log items:

Returning all mutants in (nameoffile) because the file is modified
Returning all mutants in (nameoffile) because a non-source file is modified

Also your logs should contain a table with:

xx mutants will be tested because: <insert reason here>

That table contains information about how many mutants are being tested with the reasons why.

[inishantmishra]

@rouke-broersma
Thanks for the reply.
Here are the details from logs-

2024-09-23T19:21:06.9629063Z [19:21:06 INF] 1989  mutants will be tested because: Mutant changed compared to target commit
2024-09-23T19:21:06.9637398Z [19:21:06 INF] 1989  total mutants will be tested
2024-09-23T19:21:07.0258047Z [19:21:07 DBG] Mutations will be tested in 101 test runs, instead of 1230.

I am getting following 2 lines for approx all files-

2024-09-23T19:21:06.9267893Z [19:21:06 DBG] Returning all mutants in somefile.cs because the file is modified
2024-09-23T19:21:06.9268304Z [19:21:06 DBG] Returning all mutants on somefile.cs because there is no baseline available

Actually in PullRequest, the source branch has only few files modified maybe 2-3. I don't know why these are coming.

Can you please let me know?

[rouke-broersma]

Returning all mutants on somefile.cs because there is no baseline available

So there is no valid baseline found in your fileshare for the branch you're testing on and not either for the fallback version.

What is the exact filepath of your baseline file in the fileshare?

Please take a look at:

https://stryker-mutator.io/docs/stryker-net/configuration/#baselinefallback-version-string

And:

https://stryker-mutator.io/docs/stryker-net/stryker-in-pipeline/#configuring-baseline-in-pull-requests

[shivambagadia23]

@rouke-broersma @dupdob I went through the entire discussion and I am working on something very similar where I want to make the threshold dynamic in a way that every time a new mutation score comes up for my develop branch the old score gets updated and the new one becomes the new threshold value. I am able to run baseline with develop and generate a mutation score but I am very skeptical about the possibility of how to update the new threshold. Can you please guide me a way out through this. Would be of great help. Thanks!

[shivambagadia23]

I am not aware of anyone currently working on it, but if you want to pick it up I am happy to make time to answer any questions you have to be able to provide this feature and to review a pull request.

@rouke-broersma I was thinking of working on this on this weekend. Just wanted to know what would be a good way to reach out to you incase i am stuck somewhere or need your help?

[rouke-broersma]

I am not aware of anyone currently working on it, but if you want to pick it up I am happy to make time to answer any questions you have to be able to provide this feature and to review a pull request.

@rouke-broersma I was thinking of working on this on this weekend. Just wanted to know what would be a good way to reach out to you incase i am stuck somewhere or need your help?

I regularly check github throughout the day, I'm also available on our slack.

[shivambagadia23]

Sure @rouke-broersma ...I have cloned the repository but I am not sure how do i test it after I have made some code changes. Also I did not find the iirc library you were talking about when I went inside the function that calculates the git diff DiffProviders\GitDiffProvider.cs.

[shivambagadia23]

@rouke-broersma Also, after fetching the line diff from the files how do I have to pass it in the project and how? to calculate the mutation score of only thsoe changed lines?

[shivambagadia23]

@rouke-broersma I think we are using libgit library to find changed files?