HDR streams not playing due to unsupported properties

[ValentinBesnard]

Environment

• The MPD passes the DASH-IF Conformance Tool on https://conformance.dashif.org/
• The stream has correct Access-Control-Allow-Origin headers (CORS)
• There are no network errors such as 404s in the browser console when trying to play the stream
• The issue observed is not mentioned on https://github.com/Dash-Industry-Forum/dash.js/wiki/FAQ
• The issue occurs in the latest reference client on http://reference.dashif.org/dash.js/ and not just on my page

• Link to playable MPD file: Zip containing the stream:
  stream-for-dashjs.zip
• Dash.js version: 4.7.3 and last nightly (4.7.4)
• Browser name/version: Firefox and Chrome
• OS name/version: Windows 10

Steps to reproduce

Play the stream

Observed behavior

When playing the stream with essential properties properties for HDR, stream is not playing

<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:ColourPrimaries" value="5"/>
<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:MatrixCoefficients" value="5"/>
<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:TransferCharacteristics" value="5"/>

DashJS prints a debug log "[228046][CapabilitiesFilter] [Stream] EssentialProperty not supported: urn:mpeg:mpegB:cicp:ColourPrimaries"

When I remove the "urn:mpeg:mpegB:cicp:ColourPrimaries" property, a new error appears "“[452419][CapabilitiesFilter] [Stream] EssentialProperty not supported: urn:mpeg:mpegB:cicp:MatrixCoefficients".

When I remove the three essential properties, stream is playing.

Console output

228046][CapabilitiesFilter] [Stream] EssentialProperty not supported: urn:mpeg:mpegB:cicp:ColourPrimaries

Expected behavior

Stream is playing even if DashJS does not know some essential properties.

[ValentinBesnard]

To what I see, it is a capabilities issues but the only capabilities that are supported relates to thumbnails and fonts.

function _filterUnsupportedEssentialProperties(manifest) {

        if (!manifest || !manifest.Period_asArray || manifest.Period_asArray.length === 0) {
            return;
        }

        manifest.Period_asArray.forEach((period) => {
            period.AdaptationSet_asArray = period.AdaptationSet_asArray.filter((as) => {

                if (!as.Representation_asArray || as.Representation_asArray.length === 0) {
                    return true;
                }

                as.Representation_asArray = as.Representation_asArray.filter((rep) => {
                    const essentialProperties = adapter.getEssentialPropertiesForRepresentation(rep);

                    if (essentialProperties && essentialProperties.length > 0) {
                        let i = 0;
                        while (i < essentialProperties.length) {
                            if (!capabilities.supportsEssentialProperty(essentialProperties[i])) {
                                logger.debug('[Stream] EssentialProperty not supported: ' + essentialProperties[i].schemeIdUri);
                                return false;
                            }
                            i += 1;
                        }
                    }

                    return true;
                });

                return as.Representation_asArray && as.Representation_asArray.length > 0;
            });
        });

    }

[stschr]

Hi @ValentinBesnard ,

thank you for raising this issue.

From my understanding, this is not a bug but rather a feature request.
The existing code just acts as specified for EssentialProperties; it sees them using unknown schemeIdUri-strings and consequently removes the containing AdaptationSets.

Although I see the benefit for having native support for these HDR-related identifiers, adding this is little more complicated for a quick fix.

However, a first fix might be available for you via:
#4385
With this code, you can add your schemeIdUri to the settings-structure so they pass the capability-detection.

[jpiesing]

Forgive me if I'm being stupid but I just stumbled across this.

I thought the whole point of an EssentialPropertyDescriptor was that the entity with an unknown such descriptor was ignored. Hence the behaviour that is described is exactly what I would expect assuming dash.js does not support these essential property descriptors.

To get the behaviour that the original proposer seems to want then the EssentialPropertyDescriptors should be changed to SupplementalPropertyDescriptors.

Happy to be corrected but I think this is not a bug and changing it would be a bug.

[dsilhavy]

Problem Analysis / Status quo

• I agree with what @stschr and @jpiesing wrote. dash.js is filtering AdaptationSets and Representations which contain unknown/unsupported EssentialProperty elements. This is the correct behavior.
• dash.js is currently only supporting the EssentialProperties for Thumbnails and DVBDownloadableFonts.

Immediate solution

• You have basically two options with the current implementation in dash.js
  • As @jpiesing wrote: EssentialPropertyDescriptors can be changed to SupplementalPropertyDescriptors. This will prevent the AdaptationSets and Representations to be filtetered prior to playback.
  • Disable filterUnsupportedEssentialProperties: https://cdn.dashjs.org/latest/jsdoc/module-Settings.html#~Capabilities . Note that this is a global setting and might result in other unsupported EssentialProperties not being filtered

Longterm solution

• I like the proposed change by @stschr . This gives the application the possibility to disable the filtering mechanism for specific EssentialProperty elements. Obviously the application does this at its own risk as there will be no specific handling for unsupported EssentialProperty elements in dash.js.
• I am wondering if there is any specific handling required in dash.js to enable the following EssentialProperties:

<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:ColourPrimaries" value="5"/>
<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:MatrixCoefficients" value="5"/>
<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:TransferCharacteristics" value="5"/>

We probably need to use the MediaCapabillitiesAPI to check the HDR rendering capabilities as described here? https://github.com/w3c/media-capabilities/blob/main/hdr_explainer.md#mediacapabilities-does-the-ua-support-decodingrendering-my-brand-of-hdr-video

[jpiesing]

Allowing an app to register that it handles certain essential property descriptors as described by @stschr seems to be a desirable change in principle - although I've not reviewed the details. This is valid even if, in the future, dash.js would support these specific essential property descriptors as first class citizens.

Completely ignoring / disabling essential property descriptors makes me spectacularly uncomfortable. It is likely that the organisation that created the MPD chose to use an essential property descriptor for a reason. Why not ask them to change the essential property descriptor to a supplemental one?

[dsilhavy]

#4385 adds a settings attribute supportedEssentialProperties. By adding schemeIds to this array, applications can prevent dash.js from filtering EssentialProperties. Note that adding entries to this settings array does not trigger any native handling of the "new" EssentialProperties in dash.js. If there is additional logic required to handle an AdaptationSet or a Representation that contains such an EssentialProperty playback might fail.

[ValentinBesnard]

Hi @jpiesing @stschr @dsilhavy ,

Thanks for your feedbacks and here some answers/remarks to your comments.

I agree on the point that EssentialProperty are the required properties that a decoder should supports to be able to read the stream. I also agree on the point that SupplementalProperty are the additional properties that a decoder may support to be able to read the stream with additionnal features/properties. In the case of HDR, it happens that some HDR streams are backward compatible so the HDR capabilities is set in a SupplementalProperty and players that are not HDR will read the EssentialProperty which is not HDR.

Among the essential properties that are currently supported by Dash.js, there only one for thumbnails and one for fonts but in pratice it supports many others. Bacause Dash.js is able to play non-HDR streams, it supports:

<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:ColourPrimaries" value="`X"/>
<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:MatrixCoefficients" value="Y"/>
<EssentialProperty schemeIdUri="urn:mpeg:mpegB:cicp:TransferCharacteristics" value="Z"/>

with X, Y, Z contained in a given range. Does the Dash.js team know these ranges ? Maybe not because it depends on the browser decoder. @dsilhavy Have you an idea ?

To get the behaviour that the original proposer seems to want then the EssentialPropertyDescriptors should be changed to SupplementalPropertyDescriptors.

Doing this it a short-term solution on packager side to solve interoperability issue. To do that:

• either the stream need to be duplicated (one version with essential property and one version with supplemental property) but very costly for the client
• either the adaptation sets need to be duplicated inside the same MPD (acceptable but MPD is longer so it is also more expensive for the client in terms of bandwidth)

On player side, the short term solution is to ignore essential properties and plays the stream. If one property is not supported, I think the decoder will trigger an error. Not a good solution but at least it is a workaround.

In my view, long-term solution is to add in the player all (or at least a subset) essential properties supported by the decoder.

Thank you all again for you feedbacks

Valentin

[dsilhavy]

Partially addresses in #4405 . I leave this issue open until we have support for checking the HDR related settings with the Media Capabilities API.