Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support containerless formats #2337

Closed
joeyparrish opened this issue Jan 11, 2020 · 25 comments · Fixed by #4009
Closed

Support containerless formats #2337

joeyparrish opened this issue Jan 11, 2020 · 25 comments · Fixed by #4009
Assignees
@theodab
Labels
component: HLS The issue involves Apple's HLS manifest format priority: P2 Smaller impact or easy workaround status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
HLS Improvements
Milestone
v4.0

Comments

@joeyparrish
Copy link
Member

Have you read the FAQ and checked for duplicate open issues?
Yes

Is your feature request related to a problem? Please describe.

Some HLS content uses containerless formats, such as raw AAC or raw MP3. In these formats, there is no timestamp information. This not only complicates the building of the segment index in the HLS parser, but it requires special timestamp management at the MediaSource level because these formats implicitly force their SourceBuffer into sequence mode.

Without special handling, the timestamps are never right after seeking, even if we synthesize correct SegmentReference timestamps during the playlist parsing.

Describe the solution you'd like

We should skip these formats in the HLS parser's timestamp discovery stage. We should also mark these streams as being containerless. Finally, StreamingEngine should use this containerless flag to manage timestamps at the MediaSource level (set timestampOffset every time the buffer is cleared, and include the first appended segment reference's start time).

Describe alternatives you've considered

We could reject containerless formats, but I'd prefer to improve our support for them for the sake of legacy HLS content.

Additional context

Related to #1083 (raw AAC).
Sequence mode docs: https://developer.mozilla.org/en-US/docs/Web/API/SourceBuffer/mode
Spec text around sequence mode: https://www.w3.org/TR/media-source/#dom-appendmode-sequence
@joeyparrish joeyparrish added type: enhancement New feature or request component: HLS The issue involves Apple's HLS manifest format labels Jan 11, 2020
@joeyparrish joeyparrish mentioned this issue Jan 11, 2020
Closed
shaka-bot pushed a commit that referenced this issue Jan 13, 2020
@joeyparrish
Recognize and reject raw AAC in HLS
d2ccc62 
This makes the HLS parser recognize raw AAC, but refuse to play it.
It will not play correctly until we have full support for
containerless formats at the MediaSource level.

Closes #1083 (raw AAC, can't do anything more for this right now)
Issue #2337 (full support for containerless formats in general)

Change-Id: I50c7f06a1aa08d515f4d9f74603954a6d015dc29
joeyparrish added a commit that referenced this issue Jan 15, 2020
@joeyparrish
Recognize and reject raw AAC in HLS
f3c4518 
This makes the HLS parser recognize raw AAC, but refuse to play it.
It will not play correctly until we have full support for
containerless formats at the MediaSource level.

Closes #1083 (raw AAC, can't do anything more for this right now)
Issue #2337 (full support for containerless formats in general)

Backported to v2.5.x

Change-Id: I50c7f06a1aa08d515f4d9f74603954a6d015dc29
shaka-bot pushed a commit that referenced this issue Jan 18, 2020
@joeyparrish
Fix skipping of raw format streams in HLS
1ee3284 
Raw, containerless streams can't be played yet (#2337), but our logic
for skipping or rejecting them was broken.  This broken logic affected
the whole v2.5.x series of releases up to and including v2.5.8.

This fixes the logic and improves it in several ways:
 - Skip streams that can't be played, instead of rejecting the whole
   master playlist
 - Handle raw AC3 and EC3, in addition to MP3 and AAC
 - Handle and skip WebM+HLS in the same way
 - Add the playlist and segment URLs to
   HLS_COULD_NOT_PARSE_SEGMENT_START_TIME errors

This allows us to re-enable the Apple HLS+TS asset as video-only.

Change-Id: Ia00857d87b085aa7e2b810b0b949993cebabe4ba
@joeyparrish joeyparrish added this to the Backlog milestone Jan 29, 2020
joeyparrish added a commit that referenced this issue Feb 4, 2020
@joeyparrish
Fix skipping of raw format streams in HLS
dbf7182 
Raw, containerless streams can't be played yet (#2337), but our logic
for skipping or rejecting them was broken.  This broken logic affected
the whole v2.5.x series of releases up to and including v2.5.8.

This fixes the logic and improves it in several ways:
 - Skip streams that can't be played, instead of rejecting the whole
   master playlist
 - Handle raw AC3 and EC3, in addition to MP3 and AAC
 - Handle and skip WebM+HLS in the same way
 - Add the playlist and segment URLs to
   HLS_COULD_NOT_PARSE_SEGMENT_START_TIME errors

This allows us to re-enable the Apple HLS+TS asset as video-only.

Backported to v2.5.x

Change-Id: Ia00857d87b085aa7e2b810b0b949993cebabe4ba
@joeyparrish joeyparrish mentioned this issue Aug 5, 2020
Closed
@heyhippari heyhippari mentioned this issue Oct 21, 2020
Merged
@joeyparrish joeyparrish mentioned this issue Dec 28, 2020
Closed
@avelad
Copy link
Collaborator

avelad commented Apr 20, 2021

@ismena , can you consider this issue for your next roadmap planning? (github label) There is a lot of legacy content to support and this would be very useful.

@ismena
Copy link
Contributor

ismena commented Apr 20, 2021

Yes!

@avelad avelad mentioned this issue May 5, 2021
Closed
@joeyparrish joeyparrish added this to To do in HLS Improvements Sep 13, 2021
This was referenced Sep 13, 2021
Closed
Closed
@TheModMaker TheModMaker added the priority: P2 Smaller impact or easy workaround label Sep 29, 2021
@joeyparrish joeyparrish mentioned this issue Sep 29, 2021
Closed
@johansvedberg
Copy link

Does the addition of the P2 label mean that this will be worked on in the near future?

@theodab
Copy link
Collaborator

theodab commented Oct 13, 2021

I wouldn't count on it. At the moment we have 21 P1 issues and 88 P2 issues to get through, and we're a small team.

@joeyparrish joeyparrish moved this from To do to In progress in HLS Improvements Nov 19, 2021
@Ankita20
Copy link

Hello

Is this issue resolved? We are trying to playback hls video on shaka player but not able to do so because of this issue. Any updates on by when this will be resolved and which version of shaka to use then?

@theodab
Copy link
Collaborator

theodab commented Jan 13, 2022

This issue is not currently being worked on. We do not have any estimates for when it will be resolved.

Honestly, given how much our team size has shrunk since this issue was filed, realistically it might be best to wait for community contributions. I'll change the tags on the issue.

@theodab theodab added the flag: seeking PR We are actively seeking PRs for this; we do not currently expect the core team will resolve this label Jan 13, 2022
@joeyparrish joeyparrish removed the flag: seeking PR We are actively seeking PRs for this; we do not currently expect the core team will resolve this label Jan 14, 2022
@joeyparrish
Copy link
Member Author

Quick correction: @michellezhuogg started this work before going on leave, and we may be able to complete the first phase of it in the next few weeks. I need to plot out our schedule and priorities for the near future, as we've had a lot of changes internally lately.

@theodab
Copy link
Collaborator

theodab commented Feb 15, 2022

Actually, the captions error might be the same as #3659? Not sure, I'll have to check the content they provided for that to see if it looks like the same problem. If it is the same, I shouldn't open a new issue.

theodab added a commit to theodab/shaka-player that referenced this issue Feb 15, 2022
@theodab
fix(text): Fix webvtt offset in sequence mode
93ec703 
When running in sequence mode, we ignore the normal timestamps
of video and audio segments. This lead to problems in some Apple-
encoded webvtt content, which used the X-TIMESTAMP-MAP tag to account
for the timestamp offsets in their video. Thus, those subtitles would
end up 10 seconds offset.
This changes the webvtt parser to ignore the X-TIMESTAMP-MAP when in
sequence mode.

Issue shaka-project#2337

Change-Id: I1596e064be42ad8fb536729816a64514dac07c3e
theodab added a commit to theodab/shaka-player that referenced this issue Feb 15, 2022
@theodab
fix(text): Fix webvtt offset in sequence mode
82fe602 
When running in sequence mode, we ignore the normal timestamps
of video and audio segments. This lead to problems in some Apple-
encoded webvtt content, which used the X-TIMESTAMP-MAP tag to account
for the timestamp offsets in their video. Thus, those subtitles would
end up 10 seconds offset.
This changes the webvtt parser to ignore the X-TIMESTAMP-MAP when in
sequence mode.

Issue shaka-project#2337

Change-Id: I1596e064be42ad8fb536729816a64514dac07c3e
@theodab theodab mentioned this issue Feb 15, 2022
Merged
11 tasks
@avelad
Copy link
Collaborator

avelad commented Feb 16, 2022

The root cause is the same (I think), but in the open issue it talks about H265 and here it is H264.

theodab added a commit to theodab/shaka-player that referenced this issue Feb 16, 2022
@theodab
fix(text): Fix webvtt offset in sequence mode
858fe4a 
When running in sequence mode, we ignore the normal timestamps
of video and audio segments. This lead to problems in some Apple-
encoded webvtt content, which used the X-TIMESTAMP-MAP tag to account
for the timestamp offsets in their video. Thus, those subtitles would
end up 10 seconds offset.
This changes the webvtt parser to ignore the X-TIMESTAMP-MAP when in
sequence mode.

Issue shaka-project#2337

Change-Id: I1596e064be42ad8fb536729816a64514dac07c3e
@joeyparrish joeyparrish mentioned this issue Feb 16, 2022
Closed
joeyparrish pushed a commit that referenced this issue Feb 16, 2022
@theodab
fix(text): Fix webvtt offset in sequence mode (#3955)
a4e9267 
When running in sequence mode, we ignore the normal timestamps
of video and audio segments. This lead to problems in some Apple-
encoded webvtt content, which used the X-TIMESTAMP-MAP tag to account
for the timestamp offsets in their video. Thus, those subtitles would
end up 10 seconds offset.

This changes the webvtt parser to ignore the X-TIMESTAMP-MAP when in
sequence mode.

Issue #2337
@joeyparrish
Copy link
Member Author

We have an internal report of an HLS live stream with A/V sync issues in the master branch. @theodab is looking into it.

After discussing HLS live implementation with @JulianDomingo, here are things we should consider doing next:

  • Parse EXT-X-PROGRAM-DATE-TIME and use it for A/V sync. Since HLS will now always use sequence mode in MSE, we should make sure our assigned timestamps are aligned with EXT-X-PROGRAM-DATE across streams.
  • Extrapolate EXT-X-PROGRAM-DATE-TIME if it's missing. The spec says we should extrapolate forward from segments with EXT-X-PROGRAM-DATE-TIME to those without, based on segment duration. I think we could also extrapolate backward if needed, to approximate the wall-clock-time of the first segment if it doesn't have one in the playlist for some reason.
  • If EXT-X-PROGRAM-DATE-TIME is missing, use a combination of target duration and media sequence number to approximate alignment between playlists.
  • Add a config to force us to ignore EXT-X-PROGRAM-DATE-TIME and only use sequence numbers. Anecdotally, a similar options was useful for certain content in the legacy Cast player.

This was referenced Feb 17, 2022
Merged
Merged
joeyparrish pushed a commit that referenced this issue Feb 18, 2022
feat!(hls): HLS disabled in old browsers/platforms due to incompatibi…
0daa00f 
…lities (#3964)

In Tizen 2/3 and WebOS 3.x there is no support for SourceBuffer.mode=sequence so as a result of change #2337, it necessary disable support for the HLS parser so that the user can at least use the native one with src= .
joeyparrish pushed a commit that referenced this issue Feb 24, 2022
fix(cea): make a more robust CEA MP4 parser (#3965)
2687b95 
Fix a HLS error reported in #2337 (comment)

Tested with https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8
@github-actions github-actions bot mentioned this issue Mar 3, 2022
Merged
@michellezhuogg michellezhuogg removed their assignment Mar 15, 2022
joeyparrish pushed a commit that referenced this issue Mar 22, 2022
@joeyparrish
fix(cea): make a more robust CEA MP4 parser (#3965)
b292fb3 
Fix a HLS error reported in #2337 (comment)

Tested with https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8
joeyparrish pushed a commit that referenced this issue Mar 22, 2022
@joeyparrish
fix(cea): make a more robust CEA MP4 parser (#3965)
70cbd25 
Fix a HLS error reported in #2337 (comment)

Tested with https://devstreaming-cdn.apple.com/videos/streaming/examples/bipbop_adv_example_hevc/master.m3u8
@github-actions github-actions bot mentioned this issue Mar 22, 2022
Closed
theodab added a commit that referenced this issue Mar 24, 2022
@theodab
feat(hls): Read EXT-X-PROGRAM-DATE-TIME (#4034)
89409ce 
This makes the HLS parser read the EXT-X-PROGRAM-DATE-TIME value
on manifests, and use it to make sure that segments are inserted at
the correct place in the timeline, when in sequence mode.

Issue #2337
@avelad
Copy link
Collaborator

avelad commented Apr 4, 2022

@theodab I've seen that support for EXT-X-PROGRAM-DATE-TIME has already been added, what's missing?

@joeyparrish
Copy link
Member Author

I think we're ready to close this, but we could use some feedback from the community before launch. @avelad, @johansvedberg, and anyone else reading this, have you been able to try the new implementation with your own live streams?

@avelad
Copy link
Collaborator

avelad commented Apr 13, 2022

I have tried several HLS Live streams and I see a correct result in all of them.

@joeyparrish
Copy link
Member Author

Great news! Thank you. We will close this issue, then.

If anyone has content that doesn't work correctly after v4 is released, please file a new issue with links to playlists.

@joeyparrish joeyparrish moved this from In progress to Done in HLS Improvements Apr 14, 2022
@avelad avelad modified the milestones: Backlog, v4.0 May 4, 2022
@github-actions github-actions bot added the status: archived Archived and locked; will not be updated label Jun 12, 2022
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jun 12, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@theodab theodab

Labels
component: HLS The issue involves Apple's HLS manifest format priority: P2 Smaller impact or easy workaround status: archived Archived and locked; will not be updated type: enhancement New feature or request
Projects
No open projects
HLS Improvements
Done
Milestone
v4.0
8 participants
@TheModMaker @ismena @johansvedberg @joeyparrish @avelad @Ankita20 @theodab @michellezhuogg