New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new screen capture constraints and options #22733
Add new screen capture constraints and options #22733
Conversation
Preview URLs (12 pages)
Flaws (12)Note! 3 documents with no flaws that don't need to be listed. 🎉 URL:
URL:
URL:
URL:
URL:
URL:
URL:
URL:
URL:
External URLs (3)URL:
URL:
URL:
(comment last updated: 2022-12-19 11:00:20) |
Hi @wbamberg, hope it is okay to leave the review for this PR to you |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thank you @chrisdavidmills!
### Exceptions | ||
|
||
- `InvalidStateError` {{domxref("DOMException")}} | ||
- : Thrown if the capture stream has been stopped, or if enough time has elapsed after the {{domxref("MediaDevices.getDisplayMedia()")}} `Promise` fulfills that the focus behavior has been finalized. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Calling setFocusBehavior() also throws after the getDisplayMedia() returned promise resolves, if the user shared a screen (not a tab or a window).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, updating this in my next commit. Since the number of different circumstances for throwing an InvalidState
error got to three, I've split them out into a bulleted list to make them easier to read.
@@ -54,14 +54,14 @@ to this: | |||
let mayHaveBackdropFlag = false; | |||
let displaySurface = displayStream.getVideoTracks()[0].getSettings().displaySurface; | |||
|
|||
if (displaySurface === "monitor" || displaySurface ==="application") { | |||
if (displaySurface === "monitor" || displaySurface ==="window") { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure we want to replace "application" with "window" there. If so, you may want to update the paragraph above that still uses "application" then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah, good spot; I missed that mention of "application". I've updated it to "window" to match the code snippet, as the "application" value was removed from the spec. I'm not sure if it has been removed from browsers yet, but I thought this would be good defensive writing for either eventuality.
attempt to obtain media with local audio playback enabled or disabled as specified, if | ||
possible, but will not fail if this can't be done. If, instead, the value is given as an | ||
object with an `exact` field, that field's Boolean value indicates a required | ||
setting for the noise suppression feature; if it can't be met, then the request will |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Exact values are not allowed in getDisplayMedia calls. They are in applyConstraints calls.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I've had a go at updating this section, but I do find these constraint object values rather confusing. You'll have to let me know if you think I'm any closer to getting it right. This also makes me think of two other issues:
- We ought to look at the other screen capture-specific constraints, and see if they are correct in terms of not using
exact
. - I ought to put the note about
exact
not being permitted in getDisplayMedia calls somewhere in the main http://localhost:5042/en-US/docs/Web/API/MediaTrackConstraints page, so it is more visible. But probably not on every individual constraint page.
What do you think?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe the getDisplayMedia page as well? See https://w3c.github.io/mediacapture-screen-share/#dfn-floor-value:~:text=While%20min%20and%20exact%20constraints%20produce%20TypeError
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call. I've checked, and it is kind of covered on the getDisplayMedia()
page, in the TypeError entry in the Exceptions section, but I've added in a line explicitly to say that min and exact are not allowed.
I've also followed my plan as outlined above.
async function capture() { | ||
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints(); | ||
let displayMediaOptions = { | ||
video: {}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This line is not needed as default value for video is true.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
OK, removed.
|
||
```js | ||
async function capture() { | ||
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nit: Use const
instead of let
.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Agreed; changed.
async function capture() { | ||
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints(); | ||
let displayMediaOptions = { | ||
video: {}, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated
|
||
```js | ||
async function capture() { | ||
let supportedConstraints = navigator.mediaDevices.getSupportedConstraints(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ditto
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
updated
} | ||
}, | ||
surfaceSwitching: true, | ||
selfBrowserSurface: false |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It should be either "include" or "exclude" for both surfaceSwitching
and selfBrowserSurface
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You could also suppressLocalAudioPlayback
in audio
, video: { displaySurface: "window" },
and systemAudio: "exclude"
as well
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good call, it's nice to have those constraints/options shown. I've added them.
@beaufortfrancois OK, fixes pushed; thanks for all the comments! There are a couple that I think require a bit more input from you, but overall this is getting close. |
@@ -47,7 +47,7 @@ video {{domxref("MediaStreamTrack")}}, then checking the value of the returned | |||
{{domxref("MediaTrackSettings.displaySurface", "displaySurface")}} object. | |||
|
|||
For example, if your app needs to know that the surface being shared is a monitor or | |||
application—meaning that there's possibly a non-content backdrop—it can use code similar | |||
window—meaning that there's possibly a non-content backdrop—it can use code similar |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm not sure which non-content backdrop appears in a single window?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point; I've changed it so that it only applies to a fullscreen/monitor capture.
selfBrowserSurface: false | ||
surfaceSwitching: "include", | ||
selfBrowserSurface: "exclude", | ||
systemAudio: "exclude" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
You may want to update the explanation text as well then.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Good point; explanation updated.
attempt to obtain media with local audio playback enabled or disabled as specified, if | ||
possible, but will not fail if this can't be done. If, instead, the value is given as an | ||
object with an `exact` field, that field's Boolean value indicates a required | ||
setting for the noise suppression feature; if it can't be met, then the request will |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Maybe the getDisplayMedia page as well? See https://w3c.github.io/mediacapture-screen-share/#dfn-floor-value:~:text=While%20min%20and%20exact%20constraints%20produce%20TypeError
@beaufortfrancois OK, more fixes made. Thanks again! |
LGTM! Thank you! |
Related BCD PR -> mdn/browser-compat-data#18264. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the delay, Chris!
files/jsondata/GroupData.json
Outdated
"MediaTrackSupportedConstraints.displaySurface", | ||
"MediaTrackSupportedConstraints.logicalSurface" | ||
"MediaTrackSettings.suppressLocalAudioPlayback", | ||
"MediaTrackSupportedConstraints" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
"MediaTrackSupportedConstraints" isn't a property so this can't be right. Looking at the spec there seem to be three dictionaries that are relevant (i.e. that are defined as partials and that are documented as standalone entities on MDN):
MediaTrackSupportedConstraints
MediaTrackConstraintSet (although this is documented in the subclass MediaTrackConstraints)
MediaTrackSettings
These all have properties with the same names:
displaySurface
logicalSurface
cursor
restrictOwnAudio
suppressLocalAudioPlayback
So I guess I would expect to see 15 entries here, 5 for each of those dictionaries?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I was kinda trying to reduce the amount of verbiage in the sidebar, as I felt it looked a bit horrible ;-)
But I appreciate that what I did doesn't really work.
I have added the separate MediaTrackSupportedConstraints.*
pages back in, in my next commit.
Some notes:
- I've not included
*.restrictOwnAudio
pages, asrestrictOwnAudio
is not part of this PR; I don't think it is supported yet. - I've removed
MediaTrackSupportedConstraints.cursor
andMediaTrackConstraints.cursor
; those URLs redirect to the parent class page, as they don't exist any more. I'm not really sure why.
- {{domxref("MediaTrackConstraints.logicalSurface")}} | ||
- : Indicates whether or not the video in the stream represents a logical display surface (that is, one which may not be entirely visible onscreen, or may be completely offscreen). A value of `true` indicates a logical display surface is to be captured. | ||
- {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback")}} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This and the other dictionaries omits restrictOwnAudio
, is there a reason for that?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
See above; not part of this PR, I don't think it is supported yet
|
||
### Return value | ||
|
||
`Undefined`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
`Undefined`. | |
None (`undefined`). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Change made in next commit.
@@ -35,28 +33,37 @@ See [Using the Screen Capture API](/en-US/docs/Web/API/Screen_Capture_API/Using_ | |||
## Syntax | |||
|
|||
```js-nolint | |||
getDisplayMedia(constraints) | |||
getDisplayMedia(options) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
getDisplayMedia(options) | |
getDisplayMedia() | |
getDisplayMedia(options) |
I guess, since this is optional?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Makes sense; added in next commit
- `options` {{optional_inline}} | ||
- : An optional object specifying requirements for the returned {{domxref("MediaStream")}}. The options for `getDisplayMedia()` work in the same as the [constraints](/en-US/docs/Web/API/MediaDevices/getUserMedia#parameters) for the {{domxref("MediaDevices.getUserMedia()")}} method, although in that case only `audio` and `video` can be specified. The list of possible option properties for `getDisplayMedia()` is as follows: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This <dl>
is broken, probably because of the indentation. (it's a bit of a shame that our <dl>
is quite fragile).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I tried to fix this for ages when I was originally writing this, and gave up ;-)
I just had another look, and realized that it was because the note following the <dl>
was indented. Fixed in next commit.
} | ||
|
||
try { | ||
videoElem.srcObject = await navigator.mediaDevices.getDisplayMedia(displayMediaOptions); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not sure what I'm missing here. We're passing an options object into getDisplayMedia
. The object has an audio
property which is itself an object, with a boolean suppressLocalAudioPlayback
property. But in the page for getDisplayMedia
it says that audio
is a boolean, not an object.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It can also be a MediaTrackConstraints
object.
|
||
## Value | ||
|
||
A Boolean value which is `true` if the {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback", "suppressLocalAudioPlayback")}} constraint is supported by the device and user agent. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A Boolean value which is `true` if the {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback", "suppressLocalAudioPlayback")}} constraint is supported by the device and user agent. | |
A boolean value which is `true` if the {{domxref("MediaTrackConstraints.suppressLocalAudioPlayback", "suppressLocalAudioPlayback")}} constraint is supported by the device and user agent. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated
|
||
## Examples | ||
|
||
The below function sets up the constraints object specifying the options for the call to {{domxref("MediaDevices.getDisplayMedia", "getDisplayMedia()")}}. It adds the `suppressLocalAudioPlayback` constraint (requesting that captured audio is not played out of the user's local speakers) only if it is known to be supported by the browser. Capturing is then started by calling `getDisplayMedia()` and attaching the returned stream to the video element referenced by the variable `videoElem`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The below function sets up the constraints object specifying the options for the call to {{domxref("MediaDevices.getDisplayMedia", "getDisplayMedia()")}}. It adds the `suppressLocalAudioPlayback` constraint (requesting that captured audio is not played out of the user's local speakers) only if it is known to be supported by the browser. Capturing is then started by calling `getDisplayMedia()` and attaching the returned stream to the video element referenced by the variable `videoElem`. | |
The function below sets up the options object for the call to {{domxref("MediaDevices.getDisplayMedia", "getDisplayMedia()")}}. It adds the `suppressLocalAudioPlayback` constraint (requesting that captured audio is not played out of the user's local speakers) only if it is known to be supported by the browser. Capturing is then started by calling `getDisplayMedia()` and attaching the returned stream to the video element referenced by the variable `videoElem`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
update made
|
||
> **Note:** Unlike most uses of constraints in media APIs, here it's solely used to define the stream configuration, and _not_ to filter the available choices. | ||
The `video` and `audio` objects pased in to the options object can also hold additional constraints particular to those media tracks. See [Properties of shared screen tracks](/en-US/docs/Web/API/MediaTrackConstraints#properties_of_shared_screen_tracks) for details about additional constraints for configuring a screen-capture stream that are added to {{domxref("MediaTrackConstraints")}}, {{domxref("MediaTrackSupportedConstraints")}}, and {{domxref("MediaTrackSettings")}}). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The `video` and `audio` objects pased in to the options object can also hold additional constraints particular to those media tracks. See [Properties of shared screen tracks](/en-US/docs/Web/API/MediaTrackConstraints#properties_of_shared_screen_tracks) for details about additional constraints for configuring a screen-capture stream that are added to {{domxref("MediaTrackConstraints")}}, {{domxref("MediaTrackSupportedConstraints")}}, and {{domxref("MediaTrackSettings")}}). | |
The `video` and `audio` objects passed into the options object can also hold additional constraints particular to those media tracks. See [Properties of shared screen tracks](/en-US/docs/Web/API/MediaTrackConstraints#instance_properties_of_shared_screen_tracks) for details about additional constraints for configuring a screen-capture stream that are added to {{domxref("MediaTrackConstraints")}}, {{domxref("MediaTrackSupportedConstraints")}}, and {{domxref("MediaTrackSettings")}}). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Updated
@@ -159,7 +166,7 @@ There isn't all that much code needed in order to make this work, and if you're | |||
|
|||
First, some constants are set up to reference the elements on the page to which we'll need access: the {{HTMLElement("video")}} into which the captured screen contents will be streamed, a box into which logged output will be drawn, and the start and stop buttons that will turn on and off capture of screen imagery. | |||
|
|||
The object `displayMediaOptions` contains the constraints to pass into `getDisplayMedia()`; here, the {{domxref("MediaTrackConstraints.cursor", "cursor")}} property is set to `always`, indicating that the mouse cursor should always be included in the captured media. | |||
The object `displayMediaOptions` contains the options to pass into `getDisplayMedia()`; here, the {{domxref("MediaTrackConstraints.cursor", "cursor")}} property is set to `always`, indicating that the mouse cursor should always be included in the captured media. | |||
|
|||
> **Note:** Some properties are not widely implemented and might not be used by the engine. `cursor`, for example, [has limited support](/en-US/docs/Web/API/MediaTrackConstraints#browser_compatibility). |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
cursor
doesn't appear in the linked table. (In general I don't really like notes like this. People always need to check the compat tables.)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is a bad example, it looks like cursor
has been removed, as a number of the cursor-related pages no longer exist. I've changed it to use displaySurface
.
Question - the live sample should update automatically, right? I've completely forgotten how they work ;-)
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes, live samples just work.
Thanks for the review @wbamberg ! I think I've addressed everything |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
👍 thank you for the updates, Chris!
Woo hoo! Thanks @wbamberg! |
Description
This PR adds a bunch of content covering new constraints and options from the screen capture API. You can see what's been added in my research document, at https://docs.google.com/document/d/1HUZMVu_hGTUdkThLhqrUDMBKqvFS5YNRs0xGEHIYaMY/edit#heading=h.frl8u3cowbcx
Motivation
Additional details
Related issues and pull requests