Testing Several Inputs with a Pseudo Random Number Generator (PRNG)

Question

Question Background

I already have an O(nlog(n)) solution to a given problem which I'm using as a checker to test a new faster solution (O(n)). Both methods have the same signature: they receive an array as a parameter and return a long value. So, I'm generating several array inputs using the regular PRNG and running N experiments which check both solutions. I always use the same Seed value at the beginning to get the same stream of numbers from the PRNG.

I implemented it all in a single test method which (for what I know) goes beyond of what it's considered a Unit Test. In fact, I saw a similar implementation in the book Beautiful Code but I did not find some explanations about the question below. The closer I got was this question.

The Question(s)

Should I create a Unit Test for every failed experiment? Should I set up something else that does it automatically? What should I do in this scenario?

Explanation of my code

The code is actually pretty simple because the methods that solve the problem are irrelevant for the purpose of the question. Anyways, the main part is contained within the while loop and basically generates the input array, calls the slow solution, calls the fast solution and checks if both results are equals.

        public static int MySeed = 1000000007;
        public static Random random;

        [TestMethod()]
        public void TestSeveralInputSizes()
        {
            random = new Random(MySeed);
            int numberOfExperiments = (int)1e4;
            int maxArraySize = (int)1e2;
            while (numberOfExperiments-- > 0)
            {
                var array = GenerateRandomArray(random.Next(maxArraySize));
                long expectedAnswer = SlowSolution(array);//O(nlog(n))
                long myAnswer = FastSolution(array);//O(n)
                Assert.AreEqual(expectedAnswer, myAnswer);
            }
        }

        public long[] GenerateRandomArray(int maxArraySize)
        {
            //code to generate the array
        }

I may be misunderstanding, but it doesn't sound like this is really unit testing, which is to verify the functionality of a unit, but some sort of acceptance testing verifying that the two solutions are equivalent. I take it that your concern is that you've got multiple assertions in the test, which is discouraged in the world of unit testing, but it's perfectly reasonable to test things this way with non-unit-testing (and the one assertion thing for unit tests is more of a rule of thumb anyway). — Jacob, Jan 06 '18 at 02:24
I'm assuming the Slow Solution is correct, something very simple that does the job. In that case I trust in the fact that different results mean an error in the Fast Solution. So, my question is more about what to do when I find the errors. — Raudel Ravelo, Jan 06 '18 at 02:33

score 4 · Accepted Answer · answered Jan 06 '18 at 02:44

It sounds like your code as provided is doing a different type of testing than unit testing. It's an acceptance test for FastSolution used to discover potential failing test cases. The approach you're using looks to be perfectly reasonable.

Detected failures from this acceptance testing can then be used to create individual unit test cases located elsewhere in your code. Writing these tests cases in a failing state would then be a good starting point for working on the bug fixes. After the fixes, each unit test case would live on through future iterations and will be a good guard against regression bugs.

It's perfectly reasonable to have a mixture of different test types for different purposes.

Testing Several Inputs with a Pseudo Random Number Generator (PRNG)

1 Answers1