Serenity/JS should provide the actual value of a note in reports

[viper3400]

What's the problem you're trying to solve?

When dealing with notes reporting just contain the name of the note, but not the actual value of the note.

user ensures that the text of Answer field ('My answer') does equal a note of answer ('My answer')

How would you like to solve it?

• Challenge: Reporting interactions when values are unknown at the start.

• Issue: Error messages contain values if an interaction fails, but values are unresolved initially.

• Solution for Post-Run Reporters:

  • Serenity BDD:
    • Could include descriptions initially and resolved values later in reports.
    • Generates reports after test runs, allowing for this approach.

• Challenge with Real-Time Console Reporter:

  • Console reporter reports events as they happen, which doesn't align with unresolved values at the start.
  • Possibility to modify the console reporter to report after scenario completion, but...
    • Real-time feedback is crucial, especially for longer scenarios.

• Proposed Solution:

  • Serenity BDD:
    • Report both descriptions and actual values.
  • Console Reporter:
    • Continue real-time reporting.

• Consideration for HTML Reports:

  • Reporting actual values might not be desirable, especially for tests involving lengthy text.

• Extension Idea:

  • Extend functionality, perhaps through notes, to address reporting of actual values: notes().get('recordedItems').andShowTheValueInReports().

• Edge Case Discussion Needed:

  • Further consideration needed for handling edge cases.

Are there any alternatives?

No response

How can we make it happen?

• work on this myself (with Serenity/JS team guidance)
• hire the Serenity/JS team to build it for me
• add it to the Serenity/JS backlog and wait for a volunteer

[jan-molak]

Another consideration is the Masked.valueOf() case, where we'd want to see the masked value rather than the actual value in the report.

[jan-molak]

I like this idea, though.

Some thoughts:

• I think that to make this work, the external caller (actor) would need to be told the name of the activity before it's executed, e.g. #actor enters the value of the note of answer ("my username") into the username field, as well as its name after it's (either successfully or unsuccessfully) executed, e.g. #actor enters "bob@example.org" into the username field, instead of trying to determine it itself based on its description.
• The only way I see right now we could implement it is to make Activity.performAs(actor) return some result object with a new description, rather than Promise<void> as it is now.
• This would be a major breaking change, though, so something for Serenity/JS 4, unless we could make it backwards-compatible (e.g. Activity.performAs(actor) could return either Promise<void> or Promise<ActivityDetails>)
• Implementation-wise, we could use a delayed tag template pattern.

[eyesopen]

Also maybe it would be possible to consider mute or quiet notes, and by that i mean that all the note reports is its value. For the example above it would be:

user ensures that the text of Answer field ('My answer') does equal 'My answer'

This would allow to use notes for sharing test data

[jan-molak]

This is starting to take shape.

Before:

After:

Still, a lot left to do.

[viper3400]

Hi @jan-molak, looks good. Just what to give some feedback: From my point of view, there is still some value to know that the text comes from a note in the report. Otherwise one can tend to search for a hard coded string in the specs.

[jan-molak]

@viper3400 - good point. So are you thinking we should report both the name of the question and its value?

For example:

Tess enters a note of "todo_0" - "buy some cheese" into 'What needs to be done?' input box

Is that what you had in mind?

I'm also wondering if PageElement objects should be considered separately, or the same? I.e. should we report both the description and the locator?

Or maybe we should approach it differently altogether. Maybe we should have the description kept simple, e.g.:

Tess enters a "buy some cheese" into 'What needs to be done?' input box

and have a data attachment showing how the values were arrived at? 🤔

Data:
note of todo_0.toLowerCase() - "buy some cheese"
PageElement(ByCss(#new-todo)) - 'What needs to be done?' input box

[viper3400]

@jan-molak, I ment exactly what you've wrote first -> report both the name of the question and its value.

Tess enters the value "buy some cheese" of note "todo_0" into 'What needs to be done?' input box

The data attachment would be an enhancement one can think about, but it will blow up the report as this would apply to each and every interaction that runs against a PageElement, right?

I would start simple. :)

[jan-molak]

I like your proposed format:

"buy some cheese" of note "todo_0" 

It will be interesting to see what it looks like with questions that have some transformations attached to them, like

"buy some cheese" of note "todo_0".toLowerCase().substr(0, 10)

and so on.. Maybe there's a way to make it look nicer.. 🤔

[eyesopen]

Would it be possible to use both kind of notes? Maybe it is different kind of notebook? One that gives text, other that does not.

For sharing test data informing users that this is a note feels redundant. However the test data can be implemented with getters and then there is no need to complicate Serenity/JS implementation with it.

Does setter of note changes to include the value?

Given the user notes value of input as "price"
→ user takes note of price
...
When the user enters the noted value of "price" to the input
→ user enters note of price to the input
→ user enters "12$" of note "price" to the input
...

[jan-molak]

However the test data can be implemented with getters and then there is no need to complicate Serenity/JS implementation with it.

I'm not sure if I follow, could you please give me an example of what you mean by getters?

[eyesopen]

actually i didnt test it... 🤔

as per discussion in gitter chat

actorInTheSpotlight().attemptsTo(
    TaskThatAssigns.globalVariable(),
    Enter.theValue(globalVariable).into(someInput),
  );

doesnt work as the globalVariable was modified after its value was retrieved. i assume that if globalVariable would be a function it would work as the function would be resolved on execution...

⬇️

i have different sets of test data, some of which is fixed, some is changed during execution as per example. Current file structure is something like this:

{
  country: 'GE',
  currency: 'Eur',
  vat: 19

  //placeholders for dynamic data modified during execution (usually Given steps)
  itemId:'',
}

I reference this file in my imports and modify as desired, "Before" steps cleans the dynamic data

   Given('item {string} is in the basket with quantity {int}', (item, quantity) => actorCalled('Bob').attemptsTo(
1.   Basket.add(quantity).items(item), //this will populate dataSet.itemId provided in API response
2.   Confirm.theBasketItem(dataSet.itemId),
3.   notes().set('quantity', quantity),
   ));
   When(' the user decrease the item quantity by {int}', (quantity) => {
4.  const existingQuantity = actorCalled('Bob').answer(notes.get('quantity'));
     return actorCalled('Bob').attemptsTo(
       Click.on(editIcon.of(dataSet.itemId )),
       Enter.theValue(existingQuantity - quantity).into(quantityField.of(itemLineById(dataSet.itemId) )),
     );
   });

i invented this test case to demonstrate following challenges:

(1.) and (2.) doesnt work as per original question in chat. maybe if my dataSet file had functions instead:

{
  country: 'GE',
  currency: 'Eur',
  vat: 19

  //placeholders for dynamic data modified during execution (usually Given steps)
  _itemId: ''
  itemId:(id)=>{
     if(id){
       _itemId = id;
     }
     return _itemId;
   },
}

then i could write

Confirm.theBasketItem(dataSet.itemId()),

and maybe this would bypass the case, but i just split the actor action to 2 instead of trying it

(3.) and (4.) another attempt to work with data that is defined in step. I found myself doing 4. a lot when the data interactions are needed. "note of quantity" is not what i need for action, i just need its value in test case as well as report.

Enter.theValue(notes().get(quantity)- quantity).into(quantityField.of(itemLineById(dataSet.itemId) )),
// the user enters 5 into quantity field of item

Hence the request 🙃

[jan-molak]

i have different sets of test data, some of which is fixed, some is changed during execution as per example.

That's fine, notes work with either. I think the issue might be a result of mixing synchronous and asynchronous data access models.

Let's say the structure of our custom notepad looks as follows:

interface MyNotes {
  country: string;
  currency: string;
  vat: number;

  //placeholders for dynamic data modified during execution (usually Given steps)
  itemId: string;
}

And the task to manipulate items in the basket is defined like this (I like your DSL, by the way 👍):

import { Answerable, Task, d } from '@serenity-js/core';

class Basket {
  static add = (quantity: Answerable<number>) => ({ 
    items: (name: Answerable<string) =>
      Task.where(d`#actor adds ${ quantity } ${ name } to the basket`,
        // .. add the item
        notes<MyNotes>().set('itemId', LastResponse.body<{ itemId: string }>().itemId)
      )
  })
}

The result of calling Basket.add(2).items('widgets') is that the Notepad now stores the itemId

You can now retrieve it by calling notes<MyNotes>().get('itemId'), for example:

await actor.attemptsTo(
  Enter.theValue(notes<MyNotes>().get('itemId')).into(field)
)

If you want to pass it to another task, you'll need to define its parameter as Answerable<string> to cater for the asynchronous nature of the note:

const myTask = (noteId: Answerable<string>) =>
  Task.where(`#actor ...`,
    Enter.theValue(noteId).into(field)
  )
  
await actor.attemptsTo(
  myTask(notes<MyNotes>().get('itemId'))
)

Adding asynchronous values

Performing arithmetic operations on values retrieved from notes works in a similar way to performing operations on values returned by async operations:

const a = Promise.resolve(1)
const b = Promise.resolve(3)

console.log(a + b)
// prints: `[object Promise][object Promise]`. That's not what we're after.

You need to either resolve the values before performing the operation:

const a = await Promise.resolve(1)
const b = await Promise.resolve(3)

console.log(a + b)
// prints: 4

or you need to chain the operations:

const a = Promise.resolve(1)
const b = Promise.resolve(3)

a.then(valueOfA => b.then(valueOfB => valueOfA + valueOfB)).then(console.log)
// pritns: 4

The same applies to questions:

const a = Question.about('value of a', actor => 1)
const b = Question.about('value of b', actor => 3)

console.log(a + b)
// prints: `value of avalue of b`. That's not what we're after, either.

Here again, you can resolve the values before performing the operations:

const a = Question.about('value of a', actor => 1)
const b = Question.about('value of b', actor => 3)

const valueOfA = await actor.answer(a)
const valueOfB = await actor.answer(b)

console.log(a + b)
// prints: 4

Or you can chain the operations by creating a "higher-order" question:

const sumOf = (first: Answerable<number>, second: Answerable<number>) =>
  Question.about(d`sum of ${ first } and ${ second }`, async actor => {
    const valueOfA = await actor.answer(a)
    const valueOfB = await actor.answer(b)

    return valueOfA + valueOfB
  })
  
await actor.attemptsTo(
  Enter.theValue(sumOf(notes<MyNotes>().get('quantity'), quantity)).into(field)
)  

Alternatively, if the second addend is a static value, you can also look at the operation as a mapping function:

const a = Question.about('value of a', actor => 1)
const b = 3

const increasedBy = (value: number) => 
  (input: number) => 
    input + value;

const result = await actor.answer(
  a.as(increasedBy(b)
)

console.log(result)

// prints: 4

And so:

await actor.attemptsTo(
  Enter.theValue(
    notes<MyNotes>().get('quantity')
      .as(increasedBy(quantity))
  ).into(field)
)  

Hope this helps!

[eyesopen]

Thanks! It is clear.
I think i will stick with simple js files files for test data and use notes for UI element values where actors need to note something per requirements.
Waiting impatiently for this change!