Skip to content

Announcing fromfrom

Jump to bottom
Tomi Turtiainen edited this page Mar 25, 2019 · 11 revisions

I’m announcing a new library called fromfrom. Watch this video (NOTE: the video has been recorded before I decided a name for the library) or read on to learn more!

What is it

fromfrom is a library to transform sequences of data from one format to another. Here’s a simple example of how it works.

import { from } from "fromfrom";

const users = [
  { id: 1, name: "John", age: 31, score: 2244, active: true },
  { id: 2, name: "Jane", age: 32, score: 2492, active: false },
  { id: 3, name: "Luke", age: 33, score: 2500, active: false },
  …
  { id: 1000, name: "Mary", age: 34, score: 2290, active: true },
];

from(users)
  .filter(user => user.active)
  .pick("id", "name", "score")
  .sortByDescending(user => user.score)
  .take(5)
  .toArray();

// End result:
// [
//   { id: 214, name: "May",   score: 5622 },
//   { id: 353, name: "Peter", score: 5212 },
//   { id: 32,  name: "Macy",  score: 5112 },
//   { id: 972, name: "Laura", score: 5023 },
//   { id: 429, name: "Bob",   score: 4900 }
// ]

As you can probably guess, the name of the library comes from its API, which is basically a single function from. from takes a single parameter, the data you want to transform and wraps it to a Sequence. Now you can apply and chain all types of different transformations to the data until you want to convert it back to for example an array. To see the full list of available transformation, see the API documentation.

Motivation

Have you ever tried transforming an array of data from one format to another in JavaScript? JavaScript array has all these nice functions like filter, map and reduce, which do a pretty nice job. However, what do you do when you need to something more complicated, like taking only specific items, sorting them by some attribute and taking the first 10? Or grouping the items by some attribute and taking only specific properties per item? Or want to use some data structure than array?

I come from a .NET background and when I started working with JS some years ago, I was surprised and frustrated how lacking the JS “standard” library was. Even though the situation has improved with each new ECMAScript version, you are still pretty much forced to do any more complicated operations using the reduce function, which is not as readable as using a proper function for the task.

Things get even worse when you want to use different data structures than array (and you should use a correct data structure for the situation). Map and Set don’t have any transformation methods and you’re left with the only option to convert them to arrays and back. With objects the situation is a little better. ES5.1 gave us Object.keys, ES2017 Object.entries and the Object.fromEntries TC39 proposal is as of writing this in stage 4. Still the support could be better.

Some might say you can always use a library like lodash. For me, the way you do chaining in lodash has always felt a bit clumsy and implicit. When you chain operations in lodash with the chain function you invoke the execution with .value(). It’s hard to see what the actual data type and result of the chain is going to be without reading the entire chain. Also it doesn’t support native Sets or Maps. What Having been accustomed to the powerful LINQ expressions in .NET, I wanted to have a similar experience in JS. That is, to have a comprehensive set of transform functions as well as great tooling support. Now there are already many existing LINQ implementations in JS, but they don’t feel like JS. They use the exact same method names as in .NET and they might not support Sets, Maps or objects. To tackle these shortcomings and make my life a little easier I decided to write a new library and I set these design goals for myself:

  • Minimalistic: Do one thing and do it well
  • Simple: Have a clean and simple API
  • Familiar: Use familiar function names from JS
  • Small: No external dependencies
  • Type safe: Written in TypeScript and has type definitions included
  • Comprehensive: Support all native JS collections
  • Fast: Use lazy evaluation and pipelining

How does it work

Transforming data with fromfrom can be thought as a pipeline. Each element of the source sequence flows through the pipeline and the given transformation are applied one by one. Under the hood fromfrom uses iteration protocols, deferred execution and lazy evaluation.

Iteration protocols

Iteration protocols were added in ECMAScript 2015 to provide a coherent way to produce a sequence of values and iterate them. For example when you loop through all the elements of an array with a for..of construct you are using the iteration protocols.

There are two iteration protocols: the iterable protocol and the iterator protocol. The TypeScript definitions of these two protocols are as follows:

interface Iterable<T> {
  [Symbol.iterator](): Iterator<T>;
}

interface Iterator<T> {
  next(value?: any): IteratorResult<T>;
  return?(value?: any): IteratorResult<T>;
  throw?(e?: any): IteratorResult<T>;
}

interface IteratorResult<T> {
  done: boolean;
  value: T;
}

As can be seen, for an object to implement the iterable protocol, it must have a single function named Symbol.iterator that returns an iterator. Iterator provides functions to loop through all the data. Here’s a bit modified example from MDN that shows how this works in practice with an array, that implements the iterable protocol:

var someArray = [1, 2];
var iterator = someArray[Symbol.iterator]();

iterator.next();       // { value: 1, done: false }
iterator.next();       // { value: 2, done: false }
iterator.next();       // { value: undefined, done: true }

fromfrom uses the iteration protocols to iterate through the input data. All transformations create a new iterable that when iterated reads from the previous iterable, applies the transformation and then provides the output data. fromfrom works with any input data that implements the iterable protocol. From the built-in types for example Array, Set and Map implement the iteration protocols, but for example Objects do not. fromfrom provides its own iterable implementation for objects, so they can also be used as input data.

Deferred execution and lazy evaluation

Pipelines created by fromfrom use deferred execution and lazy evaluation. Deferred execution means that the pipeline is not executed until there’s a call that forces the evaluation. Calls that produce a built-in JS type force the evaluation, such as .toArray(), .some() or .first(). Here’s an example that demonstrates that:

import { from } from "fromfrom";

var sequence = from([1, 2, 3, 4])
  .map(x => {
    console.log(x);
    return x;
  })
  .filter(x => x > 2);

// Nothing is printed the console yet

var result = sequence.toArray(); // .toArray() forces the evaluation

Lazy evaluation means that each element from the source sequence is processed through the entire pipeline as needed. The following example demonstrates that:

import { from } from "fromfrom";

var sequence = from([1, 2, 3, 4])
  .map(x => {
    console.log(x);
    return x;
  })
  .map(x => x * 2)
  .take(1)
  .toArray(); // Needed to force the evaluation

// Only 1 is printed to the console

Exception to this rule are certain transformations that require the entire sequence to be eagerly evaluated. For example .sortBy() needs sort the entire collection first before it can return the first element.

Conclusion

fromfrom is available now in NPM. Try it out and let me know for example in Twitter what you think!

Clone this wiki locally