Test fails duing run all tests, but running them one by one successes

Question

I have 4 tests spread across 3 test classes. If I run each test one by one they all can succeed. But running all (parallel I think?) they all fail except the first one fired?

My tests require the same setup, so I have a fixture which all tests are set up with:

public class CompositionRootFixture
{
    public Container Container { get; private set; } // Simple Injector Container

    public CompositionRootFixture()
    {
        Container = new Container();

        /* code removed for clearity */

        Container.Verify();
    }
}

And is used in my test classes like so:

public class CommandProcessorTests : IClassFixture<CompositionRootFixture>
{
    private readonly CompositionRootFixture _fixture;

    public CommandProcessorTests(CompositionRootFixture fixture)
    {
        _fixture = fixture;
    }

    [Fact]
    public async Task TestExecutingUsingContainerForResolution()
    {
        var commands = _fixture.Container.GetInstance<IExecuteCommands>();
        var command = new FakeCommandWithoutValidator();
        await commands.Execute(command);

        Assert.Equal("faked", command.ReturnValue);
    }
}

I have a hard time figuring out how to use the IClassFixture<T> and the documentation is not very helpful in setting this up. I am using the latest XUnit 2.0.0-beta5-build2785.

Failed description:

---- System.InvalidOperationException : The configuration is invalid. Creating the instance for type IHandleCommand<FakeCommandWithoutValidator> failed. The registered delegate for type IHandleCommand<FakeCommandWithoutValidator> threw an exception. The configuration is invalid. The type HandleFakeCommandWithoutValidator is directly or indirectly depending on itself.
-------- SimpleInjector.ActivationException : The registered delegate for type IHandleCommand<FakeCommandWithoutValidator> threw an exception. The configuration is invalid. The type HandleFakeCommandWithoutValidator is directly or indirectly depending on itself.
------------ SimpleInjector.ActivationException : The configuration is invalid. The type HandleFakeCommandWithoutValidator is directly or indirectly depending on itself.
---- The following constructor parameters did not have matching fixture data: CompositionRootFixture fixture

Answer 1

Your container is leveraging a singleton which maintains state throughout the duration of the all tests being executed.

Initialize this singleton before each test.

Answer 2

It looks like this might be related to a bug in SimpleInjector:

http://simpleinjector.codeplex.com/discussions/259167

In any case, the problem is with the dependency injection. If that bug isn't fixed, you might try a different IoC container like Ninject, at least for comparison.

Answer 3

I fixed this question by upgrading to xUnit 2.0.0 and using the new Collection Fixture, which is described on their website: http://xunit.github.io/docs/shared-context.html

Answer 4

I am experiencing the same issue with my scenario (some integration tests running tests against an EventStore) although I am not using a ClassFixture.

The idea of xUnit testing is that it can run each Fact in parallel. If you want to avoid that you can add the following on some of your assembly classes

[assembly: CollectionBehavior(DisableTestParallelization = true)]

and they'll run sequentially.

NOTE: Ideally this should be avoided because there's great interest in designing your tests and code in a way that it's idempotent and stateless, in fact I use all my tests as a subclass of this, to have a nice Given Then When structure:

public abstract class Given_When_Then_Test
    : IDisposable
{
    protected Given_When_Then_Test()
    {
        Setup();
    }

    private void Setup()
    {
        Given();
        When();
    }

    protected abstract void Given();

    protected abstract void When();

    public void Dispose()
    {
        Cleanup();
    }

    protected virtual void Cleanup()
    {
    }
}

Things I've discovered:

1. I have experienced errors in the integration tests running in parallel when I use IContainer of Autofac to resolve services instead resolving first IComponentContext and then resolving my services with it.

Answer 5

Hopefully this helps someone, but I had to ensure every Unit/Integration Test Project was referencing Xunit directly as a Nuget package. I thought it was enough to have a base test project and reference Xunit through that. Turns out this leads to many problems like inconclusive tests or tests running successfully but failing in parallel.