Merge #11185

11185: [auto] Fix hangs for remote previews r=justinvp a=justinvp

Retrieving engine events programmatically through automation api would result in the operation seemingly hanging for remote operations. This was particularly noticeable during remote previews, because preview in automation API programmatically inspects engine events to look for the summary event (which it doesn't do for up/refresh/destroy).

The problem is due to the lack of a terminating `CancelEvent`. Normally, for local operations, the event stream is always terminated by a `CancelEvent` which is the sentinel value used to indicate the operation is complete. However, when sending events to the service, the `CancelEvent` is not sent. So when we're retrieving events from the service for a remote operation, we weren't getting back a `CancelEvent`.

The fix is to send a synthesized `CancelEvent` to the events channel after all the events have been read from the service ([similar to what is done by the `pulumi replay-events` hidden debug command](https://github.com/pulumi/pulumi/blob/7694cffb6e9d3b513defbc7e324a1d6be5e6738b/pkg/cmd/pulumi/replay_events.go#L180-L184)).

Note: This issue would not happen when running a remote preview via the CLI with the default progress event display, because that display looks for [both `event.Type` of  `engine.CancelEvent` as well as an empty string as terminal events](https://github.com/pulumi/pulumi/blob/fb7400507ac40838054a6bc4d4b436b63c9752fa/pkg/backend/display/progress.go#L1226-L1232); the latter empty string happens when the events channel is closed).

Co-authored-by: Justin Van Patten <jvp@justinvp.com>

pkg/backend/httpstate/backend.go

@@ -1702,6 +1702,7 @@ func (b *cloudBackend) showDeploymentEvents(ctx context.Context, stackID client.
 
 	// The UpdateEvents API returns a continuation token to only get events after the previous call.
 	var continuationToken *string
+	var lastEvent engine.Event
 	for {
 		resp, err := b.client.GetUpdateEngineEvents(ctx, update, continuationToken)
 		if err != nil {
@@ -1712,12 +1713,18 @@ func (b *cloudBackend) showDeploymentEvents(ctx context.Context, stackID client.
 			if err != nil {
 				return err
 			}
+			lastEvent = event
 			events <- event
 		}
 
 		continuationToken = resp.ContinuationToken
 		// A nil continuation token means there are no more events to read and the update has finished.
 		if continuationToken == nil {
+			// If the event stream does not terminate with a cancel event, synthesize one here.
+			if lastEvent.Type != engine.CancelEvent {
+				events <- engine.NewEvent(engine.CancelEvent, nil)
+			}
+
 			close(events)
 			<-done
 			return nil