2

I have been working with the HTML Video Media Source Extension (MSE) and here is an overview of the graph from w3.org for how MSE interact with HTML Video Element:

MSE - Video Element

If I understand correctly, MSE only feed the source. The video decoding job is still done by HTML Video Element and it is the only entry to access hardware decoding support, per StackOverflow JS Video decoding post saying.

I have two questions:

  1. When accessing videos buffered attribute, does it refer to decoded buffer in HTML Element or downloaded/parsed buffer in MSE? If it refers to downloaded buffer, as MDN Doc saying, is that possible to get the decoded buffer range?
  2. On certain bad-performance computers, they cannot decode high resolution videos fast enough because of lack of GPU hardware support. At the same time, with really good internet bandwidth, Adaptive Bit Rate (ABR) algorithm will always try to feed high resolutions to those computers, leading to a choppy playback experience. Is there any solution to this?

Thank you so much for any advice!

Fei
  • 1,906
  • 20
  • 36

1 Answers1

1

Looking at your questions in turn:

  1. It is the downloaded buffer - the decoded content is not available to the Javascript app level typically, and is not even available to the OS if it is an encrypted media stream and the device supports a secure media path. Assuming, the video is not encrypted there is nothing in theory to stop you decoding it yourself in Javascript, but it would obviously be slow. There are some ffmpeg ports to Javascript available (e.g. https://github.com/Kagami/ffmpeg.js/) but these will still be relatively slow.

  2. Most HMTML5 players will include a way to manually or programatically set or limit the maximum resolution that the player requests from the resolutions available in the manifest. Different players may have different ABR algorithms also and some will include CPU as a factor in the algorithm. Some players may even support multiple or custom ABR algorithms so you can add you own criteria. If you want to see an example of an algorithm that allows for CPU look at the 'DroppedFramesRule' in DASH.js: https://github.com/Dash-Industry-Forum/dash.js/wiki/ABR-Logic

Mick
  • 24,231
  • 1
  • 54
  • 120