Support multipart/form-data

[NetOpWibby]

I think this is what's preventing my API from seeing my uploads.

[NetOpWibby]

My initial foray into creating a helper function with multiparty and formidable brought up TypeError: req.on is not a function.

[evert]

Makes sense. req.on would be a node request specific function, which curveball attempts to abstract to facilitate things like http/2 push, internal requests and native deployment on things like Lambda.

To get access to a Buffer representing the entire request body, you can use the rawBody function available on a request object.

If this is not sufficient, and you need access to something that's more stream-like, let me know. There's probably something we can do about this, although it might require some changes in core.

I'm going to need this very feature next week, so if you're hitting a wall this will come soon.

[NetOpWibby]

Oh that's fantastic. Perfect timing I guess! I can definitely work on other things in the meantime.

[evert]

@NetOperatorWibby

Looking into this a bit. I first thought I would just add something to this package and just put everything in ctx.request.body.

However, the implication kind of is that this is all going to be stored in memory. That might be ok for our cases (I don't think we'll exceed a few megabytes) but it might not be something you'd want to do, and maybe not a pattern we'd want to encourage.

So I'm thinking now to just suggest people use this package:

https://www.npmjs.com/package/busboy

Which really just means right now you need some way to get access to the underlying stream. So, I'll make that my focus and not add something by default to bodyparser just yet.

For our usecase we'll also use this stream and busbuy to extract the attachments.

[evert]

Actually, we have this already:

ctx.request.getStream().

So I think this should work:

const busboy = new Busboy({ headers: ctx.request.headers.getAll()});
// hook up various events
busboy.pipe(ctx.request.getStream());

[NetOpWibby]

I'll give it a try!

[NetOpWibby]

Alright, here's the relevant code I have in my copy of curveball's bodyparser:

parseFormData comes from this issue comment.

function parse(ctx: Context): Promise<void> {
  if (ctx.request.is("multipart/form-data"))
    return parseForm(ctx);

  if (ctx.request.is("json"))
    return parseJson(ctx);

  if (ctx.request.is("x-www-form-urlencoded"))
    return parseUrlEncoded(ctx);

  if (ctx.request.type.startsWith("text/"))
    return parseText(ctx);

  return Promise.resolve();
}

const parseFormData = (event: any) => {
  return new Promise((resolve, reject) => {
    const busboy = new Busboy({
      // headers: {
      //   ...event.headers,
      //   "content-type":
      //     event.headers["Content-Type"] || event.headers["content-type"]
      // }
      headers: event.headers.getAll()
    });

    const result = {
      files: []
    };

    busboy.on("file", (fieldname, file, filename, encoding, mimetype) => {
      console.log(fieldname, file, filename, encoding, mimetype);
      file.on("data", data => {
        result.files.push({
          // @ts-ignore
          contentType: mimetype,
          // @ts-ignore
          file: data,
          // @ts-ignore
          fileName: filename
        });
      });
    });

    busboy.on("field", (fieldname, value) => {
      try {
        // @ts-ignore
        result[fieldname] = JSON.parse(value);
      } catch(err) {
        // @ts-ignore
        result[fieldname] = value;
      }
    });

    // @ts-ignore
    busboy.on("error", error => reject(`Parse error: ${error}`));

    busboy.on("finish", () => {
      event.body = result;
      resolve(event);
    });

    // console.log("+++");
    // console.log(event);
    busboy.write(event.body, event.isBase64Encoded ? "base64" : "binary");
    busboy.end();
  });
};

async function parseForm(ctx: Context) {
  if (!String(ctx.request.headers.getAll()["content-type"]).includes("boundary"))
    return; // This is not a valid multipart request

  ctx.request.body = await parseFormData(ctx.request);
}

The data I'm sending to my API looks like this:

There's a good chance I'm doing something wrong because I'm not seeing busboy write anything. My original code basically copied what's in their README but I deleted that.

[evert]

So I probably wouldn't do this as a middleware, but rather use busbuy directly in the controller where you need it.

In the middleware kind of implies that you parse the whole thing, and then place the result in ctx.request somewhere, but this implies to me that you're storing the entire file in memory.. which feels like a bad idea.

However, if you are going down this route. I'm not seeing where you call pipe() to turn the request stream into busboy and kick off the process. Maybe I'm missing something.

[NetOpWibby]

You're not missing anything, I just don't know what I'm doing. I've never had to deal with file uploads in any of my projects until now so I'm just slapping things together until something works.