Allow Sharing of Generated Values Across Arbitraries

[jlink]

Testing Problem

Sometimes a value generated by an arbitrary should be the same one for all usages of this arbitrary in the same try.
Currently, this can only be accomplished by flat mapping over this value and handing the value down to all other arbitraries.

It would simplify some properties and arbitrary definitions if there was a simpler way to accomplish the same effect

Suggested Solution

There are at least two situations in which you might want to use this feature:

1. In Arbitrary Creation API. E.g.:

   Arbitrary.share(String key)

   This could create an arbitrary that will only generate a value the first time it is used with the same key!
   Look at a similar Hypothesis feature

2. When using as property parameters, e.g.:

   @Property
   void myProperty(@ForAll("fullNames") String aString, @ForAll(sharedBy="firstName") String firstName) { }
   
   @Provide
   Arbitrary<String> fullNames() {
       Arbitrary<String> firstName = Arbitraries.strings().share("firstName");
       Arbitrary<String> lastName = Arbitraries.strings().share("lastName");
       return Combinators.combine(firstName, lastName).as( (f, l) -> f + " " + l);
   }

Discussion

Implementation has a few complications:

• One has to make sure that the same sharing Id is only used once per try
• Shrinking may be a problem unless there will be flat mapping under the hood.
  This has not been fully researched yet, though.

Sketched some half-working code: https://github.com/jlink/jqwik/blob/main/engine/src/test/java/experiments/SharedArbitraryExperiments.java

[vlsi]

Frankly speaking, I would love to hide combine and flatMap into jqwik implementation.
Of course, there will always be cases when flatMap would be useful, however, I would like to write code like

@Property
void myProperty(@ForAll("fullNames") String aString, @ForAll("firstNames") String firstName) { }

// It might come from extension or something like that
@Provide
Arbitrary<String> firstNames() {...}
@Provide
Arbitrary<String> lastNames() {...}

@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) {
    return firstName + lastName;
}

[jlink]

I would like to write code like

Have you tried it? Should already work.

[jlink]

Would that feature be useful if shared values couldn't be shrunk?

Without shrinking this is rather easy to implement...

[vlsi]

What is the "easy" implementation you have in mind?
I thought the implementation would be to arrange the needed combine behind the scenes, then it should support shrinking, shouldn't it?

[jlink]

The easy implementation is to just store the generated unshrinkable value on first access.

Going for a full flatmap and flat combine behind the scenes would IMO require a data flow analysis across all generated values. I don’t even know if that’s possible. There may be a more hacky solution making use of stores and try lifecycle, but I could go with a first unshrinkable solution, if it provides value.

[vlsi]

I see.

@Provide
Arbitrary<String> fullNames() {
    Arbitrary<String> firstName = Arbitraries.strings().share("firstName");
    Arbitrary<String> lastName = Arbitraries.strings().share("lastName");
    return Combinators.combine(firstName, lastName).as( (f, l) -> f + " " + l);
}

it does indeed need dataflow analysis.

However, is dataflow needed if all the "sharing" is done via method parameters and return values?

[vlsi]

In other words, the definition like

// combine, works
@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) {
    return just(firstName + lastName);
}


// combine, works
@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) {
   ints(..)
...

vs

@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName, @ForAll int ints) {

means fullNames is a combination of firstNames and lastNames. There's no need to analyze the code, and there's no need to execute the code.

[jlink]

@Provide
Arbitrary<String> fullNames(@ForAll("firstNames") String firstName, @ForAll("lastNames") lastName) {
    return firstName + lastName;
}

That code doesn’t compile though. Jqwik needs a clear criterion to decide if it has to flatMap/combine over parameters or if plain map/combine suffices. A return type String would also work but then you could never map/combine to an Arbitrary object, since this would signal a flatMap/combine.

A new annotation would do the trick, but there are already too many for my taste.

[vlsi]

I suggest it just uses combine

[vlsi]

By the way, it might make sense to organize a call to explore options. Wdyt?

[jlink]

Sure, when would it suit you?

[vlsi]

E.g. https://doodle.com/mm/vladimirsitnikov/30min-meeting

[jlink]

Did some experimentation in https://github.com/jlink/jqwik/blob/main/engine/src/test/java/experiments/SharedArbitraryExperiments.java

[jlink]

One more idea. What if @ForAll could also resolve Arbitrary types. Like

@Provide
Arbitrary<String> aString(@ForAll Arbitrary<Integer> ints) {
	return ints.map(i -> i.toString());
}

This would not hide map/flatMap/combine but it could be used directly in provider methods.

[vlsi]

What if @forall could also resolve Arbitrary types. Like

That would be useful as well.

[jlink]

That was so easy to implement that I just did it. Now this works:

@Property
void test(@ForAll("chessSquares") String square) {
	System.out.println(square);
}

@Provide
Arbitrary<String> chessSquares(
	@ForAll Arbitrary<@CharRange(from = 'a', to = 'h') Character> rows,
	@ForAll Arbitrary<@IntRange(min = 1, max = 8) Integer> columns
) {
	return Combinators.combine(rows, columns).as((r, c) -> r.toString() + c);
}

Released and available in "1.6.3-SNAPSHOT"

[jlink]

@vlsi Would @InjectArbitrary be more understandable than @ForAll in this case?

[vlsi]

@InjectArbitrary Arbitrary<Integer> ints sounds like a duplication though.

[jlink]

@Inject seemed too generic, but I'm unsure.

[vlsi]

@Inject might make sense if there is more than one type of injectable item (e.g. Arbitrary, some sort of jqwik-related service, something related to test feedback, etc). I don't know if that is the case.

[jlink]

Currently, I don't see any other types.

[jlink]

What about @ForArbitrary?