Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add new screen capture constraints and options #22733

Merged

Conversation

chrisdavidmills
Copy link
Contributor

Description

This PR adds a bunch of content covering new constraints and options from the screen capture API. You can see what's been added in my research document, at https://docs.google.com/document/d/1HUZMVu_hGTUdkThLhqrUDMBKqvFS5YNRs0xGEHIYaMY/edit#heading=h.frl8u3cowbcx

Motivation

Additional details

Related issues and pull requests

@chrisdavidmills chrisdavidmills requested review from a team as code owners December 5, 2022 06:47
@chrisdavidmills chrisdavidmills requested review from wbamberg and dipikabh and removed request for a team December 5, 2022 06:47
@github-actions github-actions bot added the Content:WebAPI Web API docs label Dec 5, 2022
@github-actions
Copy link
Contributor

github-actions bot commented Dec 5, 2022

Preview URLs (12 pages)
Flaws (12)

Note! 3 documents with no flaws that don't need to be listed. 🎉

URL: /en-US/docs/Web/API/Screen_Capture_API
Title: Screen Capture API
Flaw count: 3

  • macros:
    • /en-US/docs/Web/API/MediaTrackConstraints/cursor redirects to /en-US/docs/Web/API/MediaTrackConstraints
    • /en-US/docs/Web/API/MediaTrackSupportedConstraints/cursor redirects to /en-US/docs/Web/API/MediaTrackSupportedConstraints
    • /en-US/docs/Web/API/MediaTrackConstraints/cursor redirects to /en-US/docs/Web/API/MediaTrackConstraints

URL: /en-US/docs/Web/API/Screen_Capture_API/Using_Screen_Capture
Title: Using the Screen Capture API
Flaw count: 2

  • macros:
    • The fourth to sixth parameters of 'EmbedLiveSample' are deprecated
  • broken_links:
    • Can't resolve /en-US/docs/Web/API/WebRTC_API/Taking_still_photos

URL: /en-US/docs/Web/API/MediaTrackConstraints
Title: MediaTrackConstraints
Flaw count: 1

  • bad_bcd_links:
    • no explanation!

URL: /en-US/docs/Web/API/MediaTrackConstraints/suppressLocalAudioPlayback
Title: MediaTrackConstraints.suppressLocalAudioPlayback
Flaw count: 1

  • bad_bcd_queries:
    • No BCD data for query: api.MediaTrackConstraints.suppressLocalAudioPlayback

URL: /en-US/docs/Web/API/MediaTrackSupportedConstraints/suppressLocalAudioPlayback
Title: MediaTrackSupportedConstraints.suppressLocalAudioPlayback
Flaw count: 1

  • bad_bcd_queries:
    • No BCD data for query: api.MediaTrackSupportedConstraints.suppressLocalAudioPlayback

URL: /en-US/docs/Web/API/CaptureController
Title: CaptureController
Flaw count: 1

  • bad_bcd_queries:
    • No BCD data for query: api.CaptureController

URL: /en-US/docs/Web/API/CaptureController/setFocusBehavior
Title: CaptureController.setFocusBehavior()
Flaw count: 1

  • bad_bcd_queries:
    • No BCD data for query: api.CaptureController.setFocusBehavior

URL: /en-US/docs/Web/API/CaptureController/CaptureController
Title: CaptureController()
Flaw count: 1

  • bad_bcd_queries:
    • No BCD data for query: api.CaptureController.setFocusBehavior

URL: /en-US/docs/Web/API/MediaTrackSettings/suppressLocalAudioPlayback
Title: MediaTrackSettings.suppressLocalAudioPlayback
Flaw count: 1

  • bad_bcd_queries:
    • No BCD data for query: api.MediaTrackSettings.suppressLocalAudioPlayback
External URLs (3)

URL: /en-US/docs/Web/API/CaptureController
Title: CaptureController


URL: /en-US/docs/Web/API/CaptureController/setFocusBehavior
Title: CaptureController.setFocusBehavior()


URL: /en-US/docs/Web/API/CaptureController/CaptureController
Title: CaptureController()

(comment last updated: 2022-12-19 11:00:20)

@dipikabh
Copy link
Contributor

dipikabh commented Dec 6, 2022

Hi @wbamberg, hope it is okay to leave the review for this PR to you

Copy link
Contributor

@beaufortfrancois beaufortfrancois left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @chrisdavidmills!

### Exceptions

- `InvalidStateError` {{domxref("DOMException")}}
- : Thrown if the capture stream has been stopped, or if enough time has elapsed after the {{domxref("MediaDevices.getDisplayMedia()")}} `Promise` fulfills that the focus behavior has been finalized.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Calling setFocusBehavior() also throws after the getDisplayMedia() returned promise resolves, if the user shared a screen (not a tab or a window).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, updating this in my next commit. Since the number of different circumstances for throwing an InvalidState error got to three, I've split them out into a bulleted list to make them easier to read.

@@ -54,14 +54,14 @@ to this:
let mayHaveBackdropFlag = false;
let displaySurface = displayStream.getVideoTracks()[0].getSettings().displaySurface;

if (displaySurface === "monitor" || displaySurface ==="application") {
if (displaySurface === "monitor" || displaySurface ==="window") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure we want to replace "application" with "window" there. If so, you may want to update the paragraph above that still uses "application" then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, good spot; I missed that mention of "application". I've updated it to "window" to match the code snippet, as the "application" value was removed from the spec. I'm not sure if it has been removed from browsers yet, but I thought this would be good defensive writing for either eventuality.

attempt to obtain media with local audio playback enabled or disabled as specified, if
possible, but will not fail if this can't be done. If, instead, the value is given as an
object with an `exact` field, that field's Boolean value indicates a required
setting for the noise suppression feature; if it can't be met, then the request will
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Exact values are not allowed in getDisplayMedia calls. They are in applyConstraints calls.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've had a go at updating this section, but I do find these constraint object values rather confusing. You'll have to let me know if you think I'm any closer to getting it right. This also makes me think of two other issues:

  • We ought to look at the other screen capture-specific constraints, and see if they are correct in terms of not using exact.
  • I ought to put the note about exact not being permitted in getDisplayMedia calls somewhere in the main http://localhost:5042/en-US/docs/Web/API/MediaTrackConstraints page, so it is more visible. But probably not on every individual constraint page.
    What do you think?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call. I've checked, and it is kind of covered on the getDisplayMedia() page, in the TypeError entry in the Exceptions section, but I've added in a line explicitly to say that min and exact are not allowed.

I've also followed my plan as outlined above.

async function capture() {
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints();
let displayMediaOptions = {
video: {},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This line is not needed as default value for video is true.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, removed.


```js
async function capture() {
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Use const instead of let.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed; changed.

async function capture() {
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints();
let displayMediaOptions = {
video: {},
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated


```js
async function capture() {
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ditto

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated

}
},
surfaceSwitching: true,
selfBrowserSurface: false
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should be either "include" or "exclude" for both surfaceSwitching and selfBrowserSurface

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You could also suppressLocalAudioPlayback in audio, video: { displaySurface: "window" }, and systemAudio: "exclude" as well

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good call, it's nice to have those constraints/options shown. I've added them.

@chrisdavidmills
Copy link
Contributor Author

@beaufortfrancois OK, fixes pushed; thanks for all the comments! There are a couple that I think require a bit more input from you, but overall this is getting close.

@@ -47,7 +47,7 @@ video {{domxref("MediaStreamTrack")}}, then checking the value of the returned
{{domxref("MediaTrackSettings.displaySurface", "displaySurface")}} object.

For example, if your app needs to know that the surface being shared is a monitor or
application—meaning that there's possibly a non-content backdrop—it can use code similar
window—meaning that there's possibly a non-content backdrop—it can use code similar
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure which non-content backdrop appears in a single window?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point; I've changed it so that it only applies to a fullscreen/monitor capture.

selfBrowserSurface: false
surfaceSwitching: "include",
selfBrowserSurface: "exclude",
systemAudio: "exclude"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You may want to update the explanation text as well then.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good point; explanation updated.

attempt to obtain media with local audio playback enabled or disabled as specified, if
possible, but will not fail if this can't be done. If, instead, the value is given as an
object with an `exact` field, that field's Boolean value indicates a required
setting for the noise suppression feature; if it can't be met, then the request will
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@chrisdavidmills
Copy link
Contributor Author

@beaufortfrancois OK, more fixes made. Thanks again!

@beaufortfrancois
Copy link
Contributor

LGTM! Thank you!

@dipikabh dipikabh removed their request for review December 12, 2022 16:06
@wbamberg
Copy link
Collaborator

Related BCD PR -> mdn/browser-compat-data#18264.

Copy link
Collaborator

@wbamberg wbamberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry for the delay, Chris!

"MediaTrackSupportedConstraints.displaySurface",
"MediaTrackSupportedConstraints.logicalSurface"
"MediaTrackSettings.suppressLocalAudioPlayback",
"MediaTrackSupportedConstraints"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

"MediaTrackSupportedConstraints" isn't a property so this can't be right. Looking at the spec there seem to be three dictionaries that are relevant (i.e. that are defined as partials and that are documented as standalone entities on MDN):

MediaTrackSupportedConstraints
MediaTrackConstraintSet (although this is documented in the subclass MediaTrackConstraints)
MediaTrackSettings

These all have properties with the same names:

displaySurface
logicalSurface
cursor
restrictOwnAudio
suppressLocalAudioPlayback

So I guess I would expect to see 15 entries here, 5 for each of those dictionaries?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was kinda trying to reduce the amount of verbiage in the sidebar, as I felt it looked a bit horrible ;-)

But I appreciate that what I did doesn't really work.

I have added the separate MediaTrackSupportedConstraints.* pages back in, in my next commit.

Some notes:

  1. I've not included *.restrictOwnAudio pages, as restrictOwnAudio is not part of this PR; I don't think it is supported yet.
  2. I've removed MediaTrackSupportedConstraints.cursor and MediaTrackConstraints.cursor; those URLs redirect to the parent class page, as they don't exist any more. I'm not really sure why.

- {{domxref("MediaTrackConstraints.logicalSurface")}}
- : Indicates whether or not the video in the stream represents a logical display surface (that is, one which may not be entirely visible onscreen, or may be completely offscreen). A value of `true` indicates a logical display surface is to be captured.
- {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback")}}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This and the other dictionaries omits restrictOwnAudio, is there a reason for that?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See above; not part of this PR, I don't think it is supported yet


### Return value

`Undefined`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
`Undefined`.
None (`undefined`).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Change made in next commit.

@@ -35,28 +33,37 @@ See [Using the Screen Capture API](/en-US/docs/Web/API/Screen_Capture_API/Using_
## Syntax

```js-nolint
getDisplayMedia(constraints)
getDisplayMedia(options)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
getDisplayMedia(options)
getDisplayMedia()
getDisplayMedia(options)

I guess, since this is optional?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Makes sense; added in next commit

Comment on lines +41 to +42
- `options` {{optional_inline}}
- : An optional object specifying requirements for the returned {{domxref("MediaStream")}}. The options for `getDisplayMedia()` work in the same as the [constraints](/en-US/docs/Web/API/MediaDevices/getUserMedia#parameters) for the {{domxref("MediaDevices.getUserMedia()")}} method, although in that case only `audio` and `video` can be specified. The list of possible option properties for `getDisplayMedia()` is as follows:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This <dl> is broken, probably because of the indentation. (it's a bit of a shame that our <dl> is quite fragile).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to fix this for ages when I was originally writing this, and gave up ;-)

I just had another look, and realized that it was because the note following the <dl> was indented. Fixed in next commit.

}

try {
videoElem.srcObject = await navigator.mediaDevices.getDisplayMedia(displayMediaOptions);
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure what I'm missing here. We're passing an options object into getDisplayMedia. The object has an audio property which is itself an object, with a boolean suppressLocalAudioPlayback property. But in the page for getDisplayMedia it says that audio is a boolean, not an object.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It can also be a MediaTrackConstraints object.


## Value

A Boolean value which is `true` if the {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback", "suppressLocalAudioPlayback")}} constraint is supported by the device and user agent.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
A Boolean value which is `true` if the {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback", "suppressLocalAudioPlayback")}} constraint is supported by the device and user agent.
A boolean value which is `true` if the {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback", "suppressLocalAudioPlayback")}} constraint is supported by the device and user agent.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated


## Examples

The below function sets up the constraints object specifying the options for the call to {{domxref("MediaDevices.getDisplayMedia", "getDisplayMedia()")}}. It adds the `suppressLocalAudioPlayback` constraint (requesting that captured audio is not played out of the user's local speakers) only if it is known to be supported by the browser. Capturing is then started by calling `getDisplayMedia()` and attaching the returned stream to the video element referenced by the variable `videoElem`.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The below function sets up the constraints object specifying the options for the call to {{domxref("MediaDevices.getDisplayMedia", "getDisplayMedia()")}}. It adds the `suppressLocalAudioPlayback` constraint (requesting that captured audio is not played out of the user's local speakers) only if it is known to be supported by the browser. Capturing is then started by calling `getDisplayMedia()` and attaching the returned stream to the video element referenced by the variable `videoElem`.
The function below sets up the options object for the call to {{domxref("MediaDevices.getDisplayMedia", "getDisplayMedia()")}}. It adds the `suppressLocalAudioPlayback` constraint (requesting that captured audio is not played out of the user's local speakers) only if it is known to be supported by the browser. Capturing is then started by calling `getDisplayMedia()` and attaching the returned stream to the video element referenced by the variable `videoElem`.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

update made


> **Note:** Unlike most uses of constraints in media APIs, here it's solely used to define the stream configuration, and _not_ to filter the available choices.
The `video` and `audio` objects pased in to the options object can also hold additional constraints particular to those media tracks. See [Properties of shared screen tracks](/en-US/docs/Web/API/MediaTrackConstraints#properties_of_shared_screen_tracks) for details about additional constraints for configuring a screen-capture stream that are added to {{domxref("MediaTrackConstraints")}}, {{domxref("MediaTrackSupportedConstraints")}}, and {{domxref("MediaTrackSettings")}}).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
The `video` and `audio` objects pased in to the options object can also hold additional constraints particular to those media tracks. See [Properties of shared screen tracks](/en-US/docs/Web/API/MediaTrackConstraints#properties_of_shared_screen_tracks) for details about additional constraints for configuring a screen-capture stream that are added to {{domxref("MediaTrackConstraints")}}, {{domxref("MediaTrackSupportedConstraints")}}, and {{domxref("MediaTrackSettings")}}).
The `video` and `audio` objects passed into the options object can also hold additional constraints particular to those media tracks. See [Properties of shared screen tracks](/en-US/docs/Web/API/MediaTrackConstraints#instance_properties_of_shared_screen_tracks) for details about additional constraints for configuring a screen-capture stream that are added to {{domxref("MediaTrackConstraints")}}, {{domxref("MediaTrackSupportedConstraints")}}, and {{domxref("MediaTrackSettings")}}).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@@ -159,7 +166,7 @@ There isn't all that much code needed in order to make this work, and if you're

First, some constants are set up to reference the elements on the page to which we'll need access: the {{HTMLElement("video")}} into which the captured screen contents will be streamed, a box into which logged output will be drawn, and the start and stop buttons that will turn on and off capture of screen imagery.

The object `displayMediaOptions` contains the constraints to pass into `getDisplayMedia()`; here, the {{domxref("MediaTrackConstraints.cursor", "cursor")}} property is set to `always`, indicating that the mouse cursor should always be included in the captured media.
The object `displayMediaOptions` contains the options to pass into `getDisplayMedia()`; here, the {{domxref("MediaTrackConstraints.cursor", "cursor")}} property is set to `always`, indicating that the mouse cursor should always be included in the captured media.

> **Note:** Some properties are not widely implemented and might not be used by the engine. `cursor`, for example, [has limited support](/en-US/docs/Web/API/MediaTrackConstraints#browser_compatibility).
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cursor doesn't appear in the linked table. (In general I don't really like notes like this. People always need to check the compat tables.)

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a bad example, it looks like cursor has been removed, as a number of the cursor-related pages no longer exist. I've changed it to use displaySurface.

Question - the live sample should update automatically, right? I've completely forgotten how they work ;-)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, live samples just work.

@chrisdavidmills
Copy link
Contributor Author

Thanks for the review @wbamberg ! I think I've addressed everything

Copy link
Collaborator

@wbamberg wbamberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 thank you for the updates, Chris!

@wbamberg wbamberg merged commit 83eccbf into mdn:main Dec 19, 2022
@chrisdavidmills chrisdavidmills deleted the add-screencapture-constraints-options branch December 19, 2022 19:30
@chrisdavidmills
Copy link
Contributor Author

Woo hoo! Thanks @wbamberg!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Content:WebAPI Web API docs
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants