render steps in lambda as nested steps

[adrian-herscu]

Sometimes a test is required to repeat a step or a sequence of steps in order to simulate some real behavior.

First I tried to use TestNG's invocationCount + threadPoolSize of @test annotation. Then discovered that every invocation is rendered in the report. Having 100,000 invocations would make the report so big as it would be unusable.

Then I tried to write a step that accepts a TestNG class and repeatedly runs all its tests. Tried to implement it as follows:

public SELF repeating_$_times(
    final int times,
    final Class<?> scenario) {
    IntStream.rangeClosed(1, times)
        .parallel()
        .forEach(i -> {
            log.debug("repeating {}", i);
            final TestNG testng = new TestNG();
            testng.setTestClasses(new Class[] { scenario });
            testng.addListener(new ScenarioTestListener());
            testng.run();
        });
    return self();
}

However, when running it, something is not correctly initialized within JGiven. So fields that should be injected are not. It also requires a public class which also get executed and rendered independently.

Finally I tried to make a repeat step method that accepts another step method and just calls it repeatedly. Tried to implement is as follows:

public SELF repeating_$_times(
    final int times,
    final @DescriptionFormatter.Annotation Function<SELF, SELF> functor) {
    IntStream.rangeClosed(1, times)
        // .parallel() // DOES NOT WORK
        .forEach(i -> {
            log.debug("repeating {}", i);
            safely(dontcare -> functor.apply(self()));
        });
    return self();
}

This one works with two issues:

1. the repeated step is not described in the report (hence I wrote a DescriptionFormatter and wrapped the functor object to include a description, just to have something in the report)
2. running the repetition in parallel does not work if the repeated step uses ThreadLocal variables

Is there a solution I am missing?
This is a real scenario, I think it is worth framework support.

[janschaefer]

Ok, I have not completely understood what you want to achieve and how should JGiven actually behave. - -- Do you want to repeat the same scenario multiple times or do you only want to repeat a single step? I ask, because your first attempt looks like that you wanted to execute a single test (=scenario) multiple times

• What should appear in the report? Only one execution or all executions?
• What is the exact use case?
• Is the main problem that you want to have the steps executed in parallel?

[adrian-herscu]

I am looking for a way to repeat a step, or a sequence of steps, sometimes in parallel. Currently I have in my scenario:

    when().repeating_$_times(2,
        new FunctionWithDescription<>(
            "requesting OAuth token for " + CLIEND_ID,
            action -> action.requesting_OAuth_Token(
                OAuthTokenRequest.builder()
                    .grantType(GrantType.CLIENT_CREDENTIALS)
                    .clientId(CLIEND_ID)
                    .build())));

In the report it looks like:

When repeating  2 times  requesting OAuth token for zQyJJI8eRMuwSUyVYTdbKg

However, it would be nicer to have it render the repeated step itself, like this:

When repeating 2 times requesting OAuth Token
    connecting to https://....
    and appending path /oauth/token
    and posting {client_id=[zQyJJI8eRMuwSUyVYTdbKg],grant_type...]

We have to test a log gathering system that should work in real time with huge amounts of data. Hence by design it may drop logs. This means that the tests should have some degree of tolerance. We cannot just trigger X logs and then verify that X logs where gathered. We have to trigger a relatively high number of logs and then verify that at least one was gathered.

While triggering logs, it would help running this in parallel and thus generating more logs per time unit and shortening test execution time.

This may match the design of Loop Controller from JMeter

[janschaefer]

I am sorry, but I fear I still have not completely understood what you want to achieve. Is it that you want to have a step repeated X times, but the step should only appear once in the report?

[adrian-herscu]

Yes. The repeated block should only appear once in the report.
And the repetition might be required to occur in parallel.
And the repetition might be required to occur for a sequence of steps.

[janschaefer]

Ok, thanks, I think I have finally understood the point :-)

Some questions:

1. Is the block identical for all repetitions?
2. What should happen if there occurs an error in one of the repetitions?
3. How should this be represented in the report?
4. How could JGiven be told that it should collapse multiple steps/blocks of steps?

Possible solution for 4: JGiven could try to detect repeating patterns/blocks in a scenario and collapse these patterns in the report. For cases with 1000 of repetitions it would actually also make sense to have a special way of representing them in the JSON model, to save space.
For single steps that are immediately repeated this should be quite easy, as one could just add a counter to the step. It becomes much more difficult for blocks of steps. The only useful thing would be to apply this to blocks of nested steps.

Example (warning not tested):

  when().requesting_the_OAuth_Token(100);

In the stage class:

 @Nested
 public SELF requesting_the_OAuth_Token(@Hidden int repetitions) {
        for (int i = 0; i < repetitions; i++) {
            this.connecting_to(...);
            this.appending_path(...);
            this.posting(...);
        }
 }

In the report:

 when requesting the OAuth Token [x100]
         connecting to ...
         appending path
         posting

[adrian-herscu]

Is the block identical for all repetitions?

Yes. Think of a block in a "for" loop or a closure in a functional language.

What should happen if there occurs an error in one of the repetitions?

The test should fail.

How should this be represented in the report?

When repeating xxx times in parallel
    step 1
    step 2....

How could JGiven be told that it should collapse multiple steps/blocks of steps?

This one I do not understand :)

[janschaefer]

Regarding the last point: Maybe the user does not want that steps are collapsed, so it might be that the user must tell JGIven somehow that for the following steps the repetitions should be collapsed.

[adrian-herscu]

I think that a good enough general solution would be that JGiven should render Java lambda expression blocks. Currently, it just calls toString() and plots that into the report.

This would open the possibility to implement generic steps such as

• repeat(int, Function<SELF,SELF>) - a step that repeatedly calls Function
• safe(Function<SELF,SELF>) - a step that swallows exceptions generated by Function
  and looking at JMeter and other testing tools there might be more ideas.

Now, one has to code the same loop, or try-catch and build a new step method just for looping or handling exceptions.

[janschaefer]

Ok. I think I understand what you mean. This will not be easy to implement, I think, but I could think of a new annotation to tell JGiven to render a lambda expression by rendering the steps that are called within the lambda as nested steps.

[adrian-herscu]

Maybe the general usecase would be reporting steps provided as parameters to other steps. This becomes common when testing asynchronous systems. Now I have a generic method like this:

public SELF retrying(final Function<SELF, SELF> step)

and almost every other step is implemented like this:

    public SELF selecting_something(final String name) {
        return retrying(step -> step
            .clicking(By.xpath("//*[contains(text(),'" + name + "')]")));
    }

In these cases the internals of select_something are not reported :(