Doing changes to a GitHub App (Probot) without affecting production or other devs' development

[oscard0m]

Hi Probot team!

In my new company we have to setup a GitHub App for managing some PR check-runs stuff (I can provide more details if you think it can add value here) and, when considering the developer experience environment we come up with the following questions:

1. How can people test changes locally without affecting the production GitHub App and without all having to share (and then synchronize their usages of) the same staging app?
2. Can we have a setup where we can spin up preview systems for PRs (which would have to include versions of the GitHub app that are spun up on demand as well)?

Checking Probot's docs I assume the following:

• 💪🏽 Each developer can run its local environment GitHub App under localhost, this way, each developer has de control to access its settings, see its delivered payloads, etc and, this way, to sync between developers can be avoided when using it.
• 😢 It implies installing the local GitHub App in the desired repositories you want to play with. It gives full control to the developer but sometimes the exact payloads you want to test can't be generated by a test repository.
  • BUT there's the option to simulate the delivery of payloads locally so it can be debugged how the local GitHub App behaves in local.

Question 1. would be more or less covered by my previous points but if there is any missing best practice or interesting trick it could be useful I would love to hear it.
Question2. in my mind sounds more like a Continuous Integration / Continuous Deployment problem. How is the Node Server deployed and versioned and the entrypoints associated but sounds it will imply a manual change of the WEBHOOK_URL manually or implementing an internal feature-toggle pointing to different environments. Same here, is there any trick or approach I'm missing?

Thanks once again for all the work and for making life easier for developers working with

[gr2m]

1. How can people test changes locally without affecting the production GitHub App and without all having to share (and then synchronize their usages of) the same staging app?

Register as many staging apps as you like. Each developer can register their own, with their own smee.io channel URL set as the webhooks URL so you can run it locally on your respective machines.

2. Can we have a setup where we can spin up preview systems for PRs (which would have to include versions of the GitHub app that are spun up on demand as well)?

I'd love that myself, and I think it would be possible to setup, but nobody did it yet. There are different ways to approach it. You could register a new GitHub app each time you want to deploy a preview. Unfortunately github apps cannot delete themselves, so you'd need to clean up manually.

Alternatively, register a pool of staging apps and configure the credentials in your repository. Then when you want to "deploy" a pull request to staging, setup automation that picks one of the available registrations and use and update the webhooks URL to point to the stating deployment URL of your server. Depending on your environment you'll need to install the app on some test repository. The app can uninstall itself to clean up when the pull request is closed.

The problem you are facing is pretty common, it would be nice to come up with a general solution that folks can apply to their environment. I think the "pool of staging GitHub Apps" could be implemented with a single GitHub Action

[tcbyrd]

The pool of staging apps is basically what I've done in the past. For the Slack and Jira integrations, we had the additional complexity of both a GitHub app and an app for the other side of the integration, so we kept around a few different Slack/Jira workspaces. In a PR, we could deploy to one of those staging environments.

I don't have much to add, but the one thing I would say to keep in mind is keeping the staging app installed can be useful for testing upgrades. Since requesting new permissions sends out an email to every user/org of your app, it's definitely nice to experience that in staging first. A relatively straight forward dev/staging setup comes to mind here. Dev for ephemeral branches that get cleaned up. Staging for longer-lived environments used for support/QA type tasks.