File Download Example

[higoka]

Is there any example on how we can download remote files to my machine?

For example, I have an array of remote files i need to download:

const files = [
  'https://www.example.com/image-1.png',
  'https://www.example.com/image-2.gif',
  'https://www.example.com/file-1.txt',
  'https://www.example.com/file-2.json'
]

I want to download them to my machine with the same file name and extension.

How can i implement this?
Which function do i need to use? Dispatcher.request, Dispatcher.stream or Dispatcher.pipeline

A small example would help a lot.
Thanks!

[jimmywarting]

I would recommend using fsPromises.writeFile instead of fs.createWritableStream as createWritableStream is meant to be piped to/from a node stream.

using writeFile is somewhat better IMO b/c it accepts a async iterator so you can write both node & web streams
fsPromises.writeFile(file, data[, options])

so:

import fs from 'node:fs/promises'
import path from 'node:path'

for await (const url of files) {
  const res = await fetch(url)
  const cd = res.headers.get('content-disposition')
  // figure out if the file name is in the content-disposition header and parse that instead
  // or use url's basename
  filename = path.basename(url)
  await fs..writeFile(filename, res.body)
}

[higoka]

Thanks! Are the requests executed one after another or are they run in parallel?

[jimmywarting]

one after the other, if you want some easy parallel work without dependencies you could eg do something like: https://stackoverflow.com/a/51020535/1008999

function doWork(iterator) {
  for await (const url of files) {
    const res = await fetch(url)
    const cd = res.headers.get('content-disposition')
    // figure out if the file name is in the content-disposition header and parse that instead
    // or use url's basename
    filename = path.basename(url)
    await fs..writeFile(filename, res.body)
  }
}

const workers = Array(4).fill(files.values()).map(doWork) // starts 4 parallel jobs by sharing same iterator
await Promise.all(workers)