Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new screen capture constraints and options #22733

Merged
Merged
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
Original file line number Diff line number Diff line change
@@ -0,0 +1,63 @@
---
title: CaptureController()
slug: Web/API/CaptureController/CaptureController
page-type: web-api-constructor
tags:
- API
- Constructor
- Experimental
- Reference
browser-compat: api.CaptureController.setFocusBehavior
---

{{APIRef("Screen Capture API")}}{{seecompattable}}

The {{domxref("CaptureController")}} constructor creates a new `CaptureController` object instance.

## Syntax

```js-nolint
CaptureController()
```

### Parameters

None.

## Examples

```js
// Create a new CaptureController instance
const controller = new CaptureController();

// Prompt the user to share a tab, window, or screen.
const stream =
await navigator.mediaDevices.getDisplayMedia({ controller });

// Query the displaySurface value of the captured video track
const [track] = stream.getVideoTracks();
const displaySurface = track.getSettings().displaySurface;

if (displaySurface == 'browser') {
// Focus the captured tab.
controller.setFocusBehavior('focus-captured-surface');
} else if (displaySurface == 'window') {
// Do not move focus to the captured window.
// Keep the capturing page focused.
controller.setFocusBehavior('no-focus-change');
}
```

## Specifications

{{Specifications}}

## Browser compatibility

{{Compat}}

## See also

- [Screen Capture API](/en-US/docs/Web/API/Screen_Capture_API)
- {{domxref("MediaDevices.getDisplayMedia()")}}
- [Better screen sharing with Conditional Focus](https://developer.chrome.com/docs/web-platform/conditional-focus/)
66 changes: 66 additions & 0 deletions files/en-us/web/api/capturecontroller/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,66 @@
---
title: CaptureController
slug: Web/API/CaptureController
page-type: web-api-interface
tags:
- API
- Interface
- CaptureController
- Reference
- Experimental
browser-compat: api.CaptureController
---

{{APIRef("Screen Capture API")}}{{seecompattable}}

The **`CaptureController`** interface provides methods that can be used to further manipulate a capture session separate from its initiation via {{domxref("MediaDevices.getDisplayMedia()")}}.

A `CaptureController` object is associated with a capture session by passing it into a {{domxref("MediaDevices.getDisplayMedia", "getDisplayMedia()")}} call as the value of the options object's `controller` property.

## Constructor

- {{ domxref("CaptureController.CaptureController", "CaptureController()") }}
- : Creates a new `CaptureController` object instance.

## Instance methods

- {{ domxref("CaptureController.setFocusBehavior", "setFocusBehavior()") }}
- : Controls whether the captured tab or window will be focused or whether the focus will remain with the tab containing the capturing app.

## Examples

```js
// Create a new CaptureController instance
const controller = new CaptureController();

// Prompt the user to share a tab, window, or screen.
const stream =
await navigator.mediaDevices.getDisplayMedia({ controller });

// Query the displaySurface value of the captured video track
const [track] = stream.getVideoTracks();
const displaySurface = track.getSettings().displaySurface;

if (displaySurface == 'browser') {
// Focus the captured tab.
controller.setFocusBehavior('focus-captured-surface');
} else if (displaySurface == 'window') {
// Do not move focus to the captured window.
// Keep the capturing page focused.
controller.setFocusBehavior('no-focus-change');
}
```

## Specifications

{{Specifications}}

## Browser compatibility

{{Compat}}

## See also

- [Screen Capture API](/en-US/docs/Web/API/Screen_Capture_API)
- {{domxref("MediaDevices.getDisplayMedia()")}}
- [Better screen sharing with Conditional Focus](https://developer.chrome.com/docs/web-platform/conditional-focus/)
79 changes: 79 additions & 0 deletions files/en-us/web/api/capturecontroller/setfocusbehavior/index.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
---
title: CaptureController.setFocusBehavior()
slug: Web/API/CaptureController/setFocusBehavior
page-type: web-api-instance-method
tags:
- API
- Experimental
- Method
- Reference
- setFocusBehavior
browser-compat: api.CaptureController.setFocusBehavior
---

{{APIRef("Screen Capture API")}}{{seecompattable}}

The {{domxref("CaptureController")}} interface's **`setFocusBehavior()`** method controls whether the captured tab or window will be focused when an associated {{domxref("MediaDevices.getDisplayMedia()")}} {{jsxref("Promise")}} fulfills, or whether the focus will remain with the tab containing the capturing app.

You can set this behavior multiple times before the {{domxref("MediaDevices.getDisplayMedia()")}} call, or once immediately after its `Promise` resolves. After that, the focus behavior is said to be finalized, and can't be changed.

## Syntax

```js-nolint
setFocusBehavior(focusBehavior)
```

### Parameters

- `focusBehavior`
- : An enumerated value that describes whether the user agent should transfer focus to the captured display surface, or keep the capturing app focused. Possible values are `focus-captured-surface` (transfer focus) and `no-focus-change` (keep focus on the capturing app).

### Return value

`Undefined`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`Undefined`.
None (`undefined`).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change made in next commit.


### Exceptions

- `InvalidStateError` {{domxref("DOMException")}}
- : Thrown if:
- The capture stream has been stopped.
- The user chose to share a screen ({{domxref("MediaTrackSettings.displaySurface", "displaySurface")}} type `monitor`) rather than a `browser` tab or `window` — you can't focus a monitor. In this case the exception is thrown after the {{domxref("MediaDevices.getDisplayMedia()")}} `Promise` resolves.
- Enough time has elapsed after the {{domxref("MediaDevices.getDisplayMedia()")}} `Promise` fulfills that the focus behavior has been finalized.

## Examples

```js
// Create a new CaptureController instance
const controller = new CaptureController();

// Prompt the user to share a tab, window, or screen.
const stream =
await navigator.mediaDevices.getDisplayMedia({ controller });

// Query the displaySurface value of the captured video track
const [track] = stream.getVideoTracks();
const displaySurface = track.getSettings().displaySurface;

if (displaySurface == 'browser') {
// Focus the captured tab.
controller.setFocusBehavior('focus-captured-surface');
} else if (displaySurface == 'window') {
// Do not move focus to the captured window.
// Keep the capturing page focused.
controller.setFocusBehavior('no-focus-change');
}
```

## Specifications

{{Specifications}}

## Browser compatibility

{{Compat}}

## See also

- [Screen Capture API](/en-US/docs/Web/API/Screen_Capture_API)
- {{domxref("MediaDevices.getDisplayMedia()")}}
- [Better screen sharing with Conditional Focus](https://developer.chrome.com/docs/web-platform/conditional-focus/)
51 changes: 28 additions & 23 deletions files/en-us/web/api/mediadevices/getdisplaymedia/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,10 +22,8 @@ browser-compat: api.MediaDevices.getDisplayMedia

{{DefaultAPISidebar("Screen Capture API")}}

The {{domxref("MediaDevices")}} interface's
**`getDisplayMedia()`** method prompts the user to select and
grant permission to capture the contents of a display or portion thereof (such as a
window) as a {{domxref("MediaStream")}}.
The {{domxref("MediaDevices")}} interface's **`getDisplayMedia()`** method prompts the user to select and
grant permission to capture the contents of a display or portion thereof (such as a window) as a {{domxref("MediaStream")}}.

The resulting stream can then be
recorded using the [MediaStream Recording API](/en-US/docs/Web/API/MediaStream_Recording_API) or transmitted as part of a [WebRTC](/en-US/docs/Web/API/WebRTC_API) session.
Expand All @@ -35,28 +33,37 @@ See [Using the Screen Capture API](/en-US/docs/Web/API/Screen_Capture_API/Using_
## Syntax

```js-nolint
getDisplayMedia(constraints)
getDisplayMedia(options)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
getDisplayMedia(options)
getDisplayMedia()
getDisplayMedia(options)

I guess, since this is optional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense; added in next commit

```

### Parameters

- `constraints` {{optional_inline}}
- : An optional object specifying requirements for
the returned {{domxref("MediaStream")}}. Since `getDisplayMedia()` requires
a video track, the returned stream will have one even if no video track is expressly
requested by the `constraints` object. For more details, see the [constraints](/en-US/docs/Web/API/MediaDevices/getUserMedia#parameters)
section under the {{domxref("MediaDevices.getUserMedia()")}} method, as well
as the article [Capabilities, constraints, and settings](/en-US/docs/Web/API/Media_Capture_and_Streams_API/Constraints).
- `options` {{optional_inline}}
- : An optional object specifying requirements for the returned {{domxref("MediaStream")}}. The options for `getDisplayMedia()` work in the same as the [constraints](/en-US/docs/Web/API/MediaDevices/getUserMedia#parameters) for the {{domxref("MediaDevices.getUserMedia()")}} method, although in that case only `audio` and `video` can be specified. The list of possible option properties for `getDisplayMedia()` is as follows:
Comment on lines +42 to +43
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This <dl> is broken, probably because of the indentation. (it's a bit of a shame that our <dl> is quite fragile).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to fix this for ages when I was originally writing this, and gave up ;-)

I just had another look, and realized that it was because the note following the <dl> was indented. Fixed in next commit.

- `video`
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Presumably also optional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed, ta.

- : A boolean; a value of `true` indicates that the returned {{domxref("MediaStream")}} will contain a video track. Be aware that, since `getDisplayMedia()` requires a video track, the returned stream will have one even if no video track is expressly requested. If the `video` property is explicitly given a value of `false`, the returned {{jsxref("Promise")}} will reject with a `TypeError`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The IDL has this and audio as (boolean or MediaTrackConstraints) (https://w3c.github.io/mediacapture-screen-share/#dom-displaymediastreamoptions).

This seems like a puzzling option. It's boolean, but if I set it to false the promise rejects?! And if I set it to true it will contain a video track. Does "even if no video track is expressly requested" here mean "even if this option is omitted? If so we could say something like:

  • if this is option is omitted or set to true, the stream will contain a video track
  • if this option is set to false the promise will reject with an error

Which makes me wonder why we would ever pass it. Am I confused here?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes made. as requested. Yup, it is quite confusing. I don't have a good answer for your question.

- `audio` {{optional_inline}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think for all optional options we should say what the default is.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done; I think I've worked these out correctly...

- : A boolean; a value of `true` indicates that the returned {{domxref("MediaStream")}} will contain an audio track, if audio is supported and available for the display surface chosen by the user.
- `controller` {{optional_inline}}
- : A {{domxref("CaptureController")}} object instance containing methods that can be used to further manipulate the capture session.
- `preferCurrentTab` {{non-standard_inline}} {{optional_inline}}
- : A boolean; a value of `true` instructs the browser to offer the current tab as the most prominent capture source, i.e. as a separate "This Tab" option in the "Choose what to share" options presented to the user. This is useful as many app types generally just want to share the current tab. For example, a slide deck app might want to let the user stream the current tab containing the presentation to a virtual conference.
- `selfBrowserSurface` {{optional_inline}}
- : An enumerated value specifying whether the browser should allow the user to select the current tab for capture. This helps to avoid the "infinite hall of mirrors" effect experienced when a video conferencing app inadvertently shares its own display. Possible values are `include`, which hints that the browser should include the current tab in the choices offered for capture, and `exclude`, which hints that it should be excluded.
- `surfaceSwitching` {{optional_inline}}
- : An enumerated value specifying whether the browser should display a control to allow the user to dynamically switch the shared tab during screen-sharing. This is much more convenient than having to go through the whole sharing process again each time a user wants to switch the shared tab. Possible values are `include`, which hints that the browser should include the control, and `exclude`, which hints that it should not be shown.
- `systemAudio` {{optional_inline}}
- : An enumerated value specifying whether the browser should include the system audio among the possible audio sources offered to the user. Possible values are `include`, which hints that the browser should include the system audio in the list of choices, and `exclude`, which hints that it should be excluded.

> **Note:** See the article [Capabilities, constraints, and settings](/en-US/docs/Web/API/Media_Capture_and_Streams_API/Constraints) for a lot more detail on how these options work.

### Return value

A {{jsxref("Promise")}} that resolves to a {{domxref("MediaStream")}} containing a
video track whose contents come from a user-selected screen area, as well as an optional
audio track.

> **Note:** Browser support for audio tracks varies, both in terms of
> whether or not they're supported at all by the media recorder and in terms of the
> audio sources supported. Check the [compatibility table](#browser_compatibility) for details for each browser.
> **Note:** Browser support for audio tracks varies, both in terms of whether or not they're supported at all by the media recorder and in terms of the audio sources supported. Check the [compatibility table](#browser_compatibility) for details for each browser.

### Exceptions

Expand All @@ -73,17 +80,15 @@ audio track.
- `NotFoundError` {{domxref("DOMException")}}
- : Returned if no sources of screen video are available for capture.
- `NotReadableError` {{domxref("DOMException")}}
- : Returned if the user selected a screen, window, tab, or other source of screen data, but a
- : Returned if the user selected a screen, window, tab, or another source of screen data, but a
hardware or operating system level error or lockout occurred, preventing the sharing
of the selected source.
- `OverconstrainedError` {{domxref("DOMException")}}
- : Returned if, after creating the stream, applying the specified `constraints` fails
- : Returned if, after creating the stream, applying any specified constraints fails
because no compatible stream could be generated.
- {{jsxref("TypeError")}}
- : Returned if the specified `constraints` include constraints which are not permitted
when calling `getDisplayMedia()`. These unsupported constraints are
`advanced` and any constraints which in turn have a member named
`min` or `exact`.
- : Returned if the specified `options` include values that are not permitted
when calling `getDisplayMedia()`, for example a `video` property set to false, or if any specified constraints are not permitted.

## Security

Expand All @@ -92,9 +97,9 @@ source of significant privacy and security concerns. For that reason, the specif
details measures browsers are required to take in order to fully support
`getDisplayMedia()`.

- The specified `constraints` can't be used to limit the options available
- The specified options can't be used to limit the choices available
to the user. Instead, they must be applied after the user chooses a source, in order
to generate output that matches the constraints.
to generate output that matches the options.
- The go-ahead permission to use `getDisplayMedia()` cannot be persisted
for reuse. The user must be prompted for permission every time.
- [Transient user activation](/en-US/docs/Web/Security/User_activation) is required. The user has to interact with the page or a UI element in order for this feature to work.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -47,21 +47,21 @@ video {{domxref("MediaStreamTrack")}}, then checking the value of the returned
{{domxref("MediaTrackSettings.displaySurface", "displaySurface")}} object.

For example, if your app needs to know that the surface being shared is a monitor or
application—meaning that there's possibly a non-content backdrop—it can use code similar
window—meaning that there's possibly a non-content backdrop—it can use code similar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure which non-content backdrop appears in a single window?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point; I've changed it so that it only applies to a fullscreen/monitor capture.

to this:

```js
let mayHaveBackdropFlag = false;
let displaySurface = displayStream.getVideoTracks()[0].getSettings().displaySurface;

if (displaySurface === "monitor" || displaySurface ==="application") {
if (displaySurface === "monitor" || displaySurface ==="window") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we want to replace "application" with "window" there. If so, you may want to update the paragraph above that still uses "application" then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good spot; I missed that mention of "application". I've updated it to "window" to match the code snippet, as the "application" value was removed from the spec. I'm not sure if it has been removed from browsers yet, but I thought this would be good defensive writing for either eventuality.

mayHaveBackdropFlag = true;
}
```

Following this code, `mayHaveBackdrop` is `true` if the display
surface contained in the stream is of type `monitor` or
`application`; either of these _may_ have non-content backdrop areas.
`window`; either of these _may_ have non-content backdrop areas.
Later code can use this flag to determine whether or not to perform special processing,
such as to remove or replace the backdrop, or to "cut" the individual display areas out
of the received frames of video.
Expand All @@ -73,8 +73,6 @@ use of the `displaySurface` property. In addition, see
{{SectionOnPage("/en-US/docs/Web/API/Media_Capture_and_Streams_API/Constraints", "Example:
Constraint exerciser")}} for a complete example showing how constraints are used.

TBD

## Specifications

{{Specifications}}
Expand Down
5 changes: 3 additions & 2 deletions files/en-us/web/api/mediatrackconstraints/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -158,8 +158,6 @@ These constraints apply to the `video` property of the object passed into {{domx

- : A [`ConstrainDOMString`](#constraindomstring) which specifies the types of display surface that may be selected by the user. This may be a single one of the following strings, or a list of them to allow multiple source surfaces:

- `application`
- : The stream contains all of the windows of the application chosen by the user rendered into the one video track.
- `browser`
- : The stream contains the contents of a single browser tab selected by the user.
- `monitor`
Expand All @@ -170,6 +168,9 @@ These constraints apply to the `video` property of the object passed into {{domx
- {{domxref("MediaTrackConstraints.logicalSurface", "logicalSurface")}}
- : A [`ConstrainBoolean`](#constrainboolean) value which may contain a single Boolean value or a set of them, indicating whether or not to allow the user to choose source surfaces which do not directly correspond to display areas. These may include backing buffers for windows to allow capture of window contents that are hidden by other windows in front of them, or buffers containing larger documents that need to be scrolled through to see the entire contents in their windows.

- {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback", "suppressLocalAudioPlayback")}}
- : A [`ConstrainBoolean`](#constrainboolean) value describing the requested or mandatory constraints placed upon the value of the {{domxref("MediaTrackSettings.suppressLocalAudioPlayback","suppressLocalAudioPlayback")}} constrainable property. This property controls whether the audio playing in a tab will continue to be played out of a user's local speakers when the tab is captured.

## Specifications

{{Specifications}}
Expand Down