Merge #11595

11595: fix: Prevent a deadlock on provider error in py automation api r=justinvp a=elthariel

<!--- 
Thanks so much for your contribution! If this is your first time contributing, please ensure that you have read the [CONTRIBUTING](https://github.com/pulumi/pulumi/blob/master/CONTRIBUTING.md) documentation.
-->

# Description

<!--- Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. -->

The rationale behind this change is explained in the linked issue. I haven't worked on the Checklist below, as I'd like to get an eye with more pulumi knownledge on the issue beforehand.

Also, reproducing deadlocks in unit tests is always quite tricky, so I'd rather not do it until I'm sure this is the proper approach

Fixes #11594

## Checklist

<!--- Please provide details if the checkbox below is to be left unchecked. -->
- [ ] I have added tests that prove my fix is effective or that my feature works
<!--- 
User-facing changes require a CHANGELOG entry.
-->
- [x] I have run `make changelog` and committed the `changelog/pending/<file>` documenting my change
<!--
If the change(s) in this PR is a modification of an existing call to the Pulumi Service,
then the service should honor older versions of the CLI where this change would not exist.
You must then bump the API version in /pkg/backend/httpstate/client/api.go, as well as add
it to the service.
-->
- [ ] Yes, there are changes in this PR that warrants bumping the Pulumi Service API version
  <!-- `@Pulumi` employees: If yes, you must submit corresponding changes in the service repo. -->


Co-authored-by: Julien 'Lta' BALLET <contact@lta.io>

changelog/pending/20221208--sdk-python--fix-a-deadlock-on-provider-side-error-with-automation-api.yaml

@@ -0,0 +1,4 @@
+changes:
+- type: fix
+  scope: sdk/python
+  description: Fix a deadlock on provider-side error with automation api

sdk/python/lib/pulumi/automation/_server.py

@@ -43,6 +43,15 @@ def on_pulumi_exit():
     def GetRequiredPlugins(self, request, context):
         return language_pb2.GetRequiredPluginsResponse()
 
+    def _exception_handler(self, loop, context):
+        # Exception are normally handler deeper in the stack. If this class of
+        # exception bubble up to here, something is wrong and we should stop
+        # the event loop
+        if "exception" in context and isinstance(context["exception"], grpc.RpcError):
+            loop.stop()
+        else:
+            loop.default_exception_handler(context)
+
     def Run(self, request, context):
         _suppress_unobserved_task_logging()
 
@@ -67,6 +76,7 @@ def Run(self, request, context):
         result = language_pb2.RunResponse()
         loop = asyncio.new_event_loop()
 
+        loop.set_exception_handler(self._exception_handler)
         try:
             loop.run_until_complete(run_in_stack(self.program))
         except RunError as exn: