What would be the ideal way to create tests for your Probot app?

[gr2m]

Writing tests for Probot is hard. Usually you want to test what we call a webhook event lifecyle:

1. Your app receives a webhook event from GitHub
2. Your app sends http requests when handling the request

What we currently recommend is documented at https://probot.github.io/docs/testing/

What is hard is

1. Create / maintain a webhook request fixture
2. Create / maintain the http mocks with nock for the subsequent API requests

What I would like to do is to add a Probot feature that would record fixtures for both the webhook request coming from GItHub and the API requests into a fixtures folder, and then generate a test from that automatically. The test would be a "replay" of the previously recorded webhook event lifecycle. It would fail as soon as you change something in your code which result in different requests being sent from your app when receiving the same webhook.

If that happens, two things might have occurred

1. A change broke previous behavior and you need to fix it in your code
2. The change is correct and you need to update the fixtures.

It's clear what to do for 1.. What to do for 2. is less clear, it is something we need to consider. We should implement something like Jest's --updateSnapshot option, which would mean that the fixtures for the API requests would be updated automatically.

I am not sure how all this would work, I would be eager to here your thoughts.

[gr2m]

My happy path would be this:

1. I setup my new GitHub app (configure APP_ID, PRIVATE_KEY, WEBHOOK_SECRET, and WEBHOOK_PROXY_URL) using Probot's setup feature.

2. I install my new GitHub app on a test repository

3. I implement whatever I want my app to do, say I want add a comment to every new issue

4. I restart probot with a --record flag

   probot run ./index.js --record
   

5. I go to my test repository and create a new issue. My app adds the comment. In my logs, I see something like this

   Fixtures written to `test/webhook-lifecycles/issues.created-1599070094284.json`
   

The issues.created-1599070094284.json file would include the event webhook event information (name, id, payload) and the request information that created the comment. All sensitive information has been removed or redacted (webhook signature, authorization header)

6. I run probot test which loads all .json files in the test/webhook-lifecycles folder by default. It receives the webhook event, then intercepts the request to create the issue and compares it to the request information from the fixture.

7. The test should pass

People could still write their own tests using Jest or whatever. But for most applications, the probot run --record and probot test commands should be sufficient.

In an ideal world, probot test would include coverage information.

[travi]

one concern that comes to mind with recording the real webhooks and request payloads for me is focus. i purposely try to include as little content in those payloads in my tests in order to make it more clear which pieces actually have impact on the flow that i'm attempting to test.

i fully agree that it can be difficult to grab a realistic example to reference, but personally i would rather make that piece easier and leave it up to the developer what to do with it. in some cases using the entire payload could be appropriate, but i mostly prefer not to use the full payload in actual tests.

[oscard0m]

one concern that comes to mind with recording the real webhooks and request payloads for me is focus. i purposely try to include as little content in those payloads in my tests in order to make it more clear which pieces actually have impact on the flow that i'm attempting to test.

i fully agree that it can be difficult to grab a realistic example to reference, but personally i would rather make that piece easier and leave it up to the developer what to do with it. in some cases using the entire payload could be appropriate, but i mostly prefer not to use the full payload in actual tests.

Maybe we can brainstorm --record-mode=<simple|full> and cover both use cases @travi ? What do you think.

[gr2m]

I think you could edit the recorded fixtures after the fact and remove all properties you don't need?

Personally I do the same when creating tests/fixtures by hand, but I think because this is an automated recording/testing thing, I'd probably just leave it be. You can still add your own tests, it's not an either/or, you can do both.