JPEG live stream in html slow

Question

From a raw video source I'm trying to stream jpeg images to html as fast as possible in a embedded platform/board using linux.

At the gstreamer side I can see that the jpeg image is updated at ~37fps the pipeline looks like this:

appscr -> videoconvert -> jpegenc -> multifilesink

based in this question I created the next embedded html:

<!DOCTYPE html>
<html>
    <head>
        <meta charset='UTF-8' />
        <script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.0/jquery.min.js"></script>
        <script src="app.js"></script>
    </head>
    <body>  
        <img id="snapshot" src="snapshot.jpeg"/>
    </body>
</html>

and the java script:

$(function() {
    function refreshImage(){
    $("#snapshot").attr("src", 'snapshot.jpeg?' + Math.random());
    setTimeout(refreshImage, 20);
    }
    refreshImage();
})

Opening a web browser from a PC and typing the platform/board IP I can see the video stream, but the problem is that the image is updated too slow and I would expect a more fluid/fast video given the source frame rate (37fps).

does anyone know what could be the reason why the update is slow?

Maybe try `setInterval()` instead of `setTimeout()`? 37fps -> 27ms frame period, so if the browsers decode and render time is > 7ms you'll miss frames with those 20ms chained timeouts. — jamieguinan, Jan 23 '20 at 19:37
Thanks for the suggestion, I replaced setTimeout by setInterval it but unfortunately it did work even slower...any more ideas about what could it be? — joe, Jan 24 '20 at 09:24
@joe When image is refreshed in browser, browser actually pulls (http get) image from embedded server, embedded server reads image from persistent memory and serves it via LAN (WiFi or cable) to browser. All those factors might increase latency proportionally to size of the image. Did you try, just for testing purposes, to decrease size of image? You can measure those periods by using browser's debugging/developer tools. — Dusan Kovacevic, Jan 24 '20 at 09:50
@DusanKovacevic thanks for your idea, I tried different resolutions and it did improve it at low resolutions (640x480) but still quite a bit lag is there any way to speed it up or it is a limitation? — joe, Jan 25 '20 at 10:49

score 2 · Accepted Answer · answered Jan 25 '20 at 13:30

I think this deserves proper analysis since it is interesting subject (at least for me).

Testing environment

I completely replicated scenario on 2 PCs within same LAN.

PC 1 is creating jpeg images from live stream with following pipeline

gst-launch-1.0 -v rtspsrc location="rtsp://freja.hiof.no:1935/rtplive/_definst_/hessdalen03.stream" \
! rtph264depay ! avdec_h264 \
! timeoverlay halignment=right valignment=bottom \
! videorate ! video/x-raw,framerate=37000/1001 ! jpegenc ! multifilesink location="snapshot.jpeg"

and serving index.html, app.js and (endlessly updated) snapshot.jpeg with python's simple http server

python -m SimpleHTTPServer 8080

PC 2 is accessing index.html using Chrome browser (with developer tools window) and showing images.

For testing purposes

I've added timeoverlay in gstreamer pipeline that adds timestamp on each image in right bottom corner.
I increased refresh period in JS function on 1000 ms.

Analysis of test results

Here is browser's network log

Time column shows periods (in ms) that browser spends to fetch (download) one image from server. Those periods are not always the same with average of ~100ms for images with size ~87KB.

Fetch time interval actually includes:

interval that HTTP GET needs to reach server from browser,
interval that server needs to read image from disk and send it back as HTTP response,
interval that HTTP response needs to reach browser.

1st and 3rd interval are directly depend on "internet" environment: if browser is "farther" away from server interval will be greater.

2nd interval is proportional to server "speed": how fast server can read images from disk and handle HTTP request/response

There is another interval proportional to "speed" of PC which runs browser: how fast PC can handle HTTP GET request/response and re-render image.

Conclusion

There are many unavoidable delay intervals that depend on testing environment - internet and capabilities of server machine and client machine with browser - and your code in browser is executing as fast as it possibly can.

In any case, 37 fps sounds like some live stream video. There are specialized protocols to stream video that can be shown in browser (e.g. MPEG DASH or HLS) by serving video chunk-by-chunk (where each chunk contains many video frames).

Many thanks for your time and analysis, I'm quite new on this topic and your answer really help me to understand it better. I'll have a look to the HLS option, gstreamer has a dedicated plugin for it (hlssink) it might be good option. — joe, Jan 25 '20 at 21:30

JPEG live stream in html slow

1 Answers1