4

Is it possible to dynamically produce large files (10Gb+) for the client to download?

I'm building a webapp that dynamically creates files and downloads them to the client. I've implemented this by creating a Blob of data and using an objectUrl to download them.

(example from: Download large files in Dartlang):

import 'dart:html';

void main() {
  var data = new Blob(['Hello World!\n'*1000000]);
  querySelector("#downloadLink")
      ..setAttribute('href', Url.createObjectUrl(data));
}

However, this does not work for large files. Is there a way to stream files to the client, or append data to a file, so that the file can be generated piece by piece rather than generating the entire thing before downloading?

Community
  • 1
  • 1
Aro
  • 609
  • 9
  • 15
  • There is a maximum string length in Chrome - [see this post](http://stackoverflow.com/a/4695265/1961059). – Maximilian Riegler Nov 18 '16 at 08:57
  • But it doesn't work with a `List` [because we're running out of memory then ...](https://dartpad.dartlang.org/07baf1ea98f2800dfa2d3cc13d5ae9db) – Maximilian Riegler Nov 18 '16 at 09:03
  • 1
    You can stream. I used it here to consume from a Docker service https://github.com/bwu-dart/bwu_docker/blob/0244aa90079e4e8515b5ca412e73b8f858cab809/lib/src/remote_api.dart#L136 but it's `dart:io` code, but might help you to get some ideas. – Günter Zöchbauer Nov 18 '16 at 14:26
  • I doubt the max String length would be an issue. I construct my Blob by combining a previously constructed blob with a new small string. – Aro Nov 18 '16 at 17:56
  • @GünterZöchbauer It looks like HTTP response stream may work if I can trigger the browser to save the file. If that works, I'll be streaming from the client to itself since I don't have a backend. – Aro Nov 18 '16 at 18:16
  • Do you really need the whole file in memory at once. What about just processing the stream as the data arrives and then drop the data? – Günter Zöchbauer Nov 18 '16 at 19:21

1 Answers1

3

for < 800MB i would recommend FileSaver but for 10GB+ you are going to need something like StreamSaver (It will only work in Blink doe) 16GB+ haven't been any problem for me

const fileStream = streamSaver.createWriteStream('filename.txt')
const writer = fileStream.getWriter()
const encoder = new TextEncoder()

// When you have any data you write it as Uint8array
let data = 'a'.repeat(1024)
let uint8array = encoder.encode(data + "\n\n")

writer.write(uint8array) // chunk
writer.write(uint8array) // chunk
writer.write(uint8array) // chunk
writer.write(uint8array) // chunk

// After you have written all bytes you need to close it
writer.close()

(just don't write all chunks at once, do it progressively as you get more data avalible)

Endless
  • 34,080
  • 13
  • 108
  • 131
  • This appears to be what I need. I haven't tested it yet; I'm going to wrap it in Dart first. Until then, this seems to answer my question, thanks! – Aro Nov 18 '16 at 23:40
  • One thing I just noticed is StreamSaver.js only supports Chrome 52+ and Opera 39+ – Aro Nov 19 '16 at 00:05
  • That is what i said (Blink only...) – Endless Nov 19 '16 at 19:20
  • @Aro Did you ever get this to work as needed? If so, would you mind sharing more about how, perhaps, as answer to this ( https://stackoverflow.com/questions/51006619/firefox-streams-api-streamsaver-to-stream-dynamically-generated-data-on-client) – Gary Jun 24 '18 at 20:02
  • @Gary Unfortunately I didn't figure it out completely. I asked while I was working on a side project and moved to other tasks without finishing this problem. – Aro Jun 24 '18 at 20:10
  • @Aro Thanks. If it comes up again, in his comment to his answer to https://stackoverflow.com/questions/50997762/what-and-where-exactly-are-privileged-code-chrome-code-gecko, Andrew Swan mentioned using webRequest and the filteResponseData API as a possible solution in an extension. It has a StreamFilter and you can intercept and write to a web request. Not sure it will work on a download request but am going to attempt it. If it works, I'll let you know. – Gary Jun 24 '18 at 21:50