Skip to content

Hamsters v5.6.1 Released!
Compare
Choose a tag to compare
@austinksmith austinksmith released this 23 Sep 14:10
· 2 commits to v5.6.1 since this release
v5.6.1
eaed5bd

I'm happy to announce the release of Hamsters.js version 5.6.1 while this is a minor bump in versions its actually quite a big leap in feature improvements.

Observable

There is a new feature and class called hamsters.observable, you can use this class as an event emitter to log changes in objects you want to track inside your project, additionally this is now being used inside the library to allow for change monitoring of internal library variables to give better insights into what is taking place behind the scenes when the library is running. Please see the list below of examples on variables you can monitor.

  hamsters.pool.threads.on('change', function(threads) {
    console.log("Thread Pool Threads Changed");
  });

  hamsters.pool.running.on('change', function(runningThreads) {
    console.log("Thread Pool Running Threads Changed");
  });

  hamsters.pool.pending.on('change', function(pendingThreads) {
    console.log("Thread Pool Pending Threads Changed");
  });

  hamsters.distribute.remoteConnections.on('change', function(connections) {
    console.log("Distributed Remote Connections Changed");
  });

  hamsters.distribute.receiveChannels.on('change', function(receiveChannels) {
    console.log("Distributed P2P Receive Channels Changed");
  });

  hamsters.distribute.sendChannels.on('change', function(sendChannels) {
    console.log("Distributed P2P Send Channels Changed");
  });

  hamsters.distribute.clientInfo.on('change', function(clientInfo) {
    console.log("Distributed Connected Client Info Changed");
  });

  hamsters.distribute.pendingPromises.on('change', function(pendingPromises) {
    console.log("Distributed Pending Promises (Tasks) Changed");
  });

Scaffold

In previous releases of Hamsters.js you were confined to making use of the libraries built in web worker scaffolds, as Hamsters.js is meant to be a higher level abstraction to provide universal parallel computing support, pre-defined methods are required to preserve the use of the thread pool, however this presented some limitations in some of the workloads you can complete with Hamsters.js, one of these challenges was the 1 billion row challenge where you must parse a billion rows of sensor measurements from a txt file as fast as possible.

When attempting this in the past using Hamsters.js, there was a problem where since this is a txt file of objects I didn't have an efficient way to ensure that the workers themselves got the data from the file requiring parsing of the data on the main thread and then sending that data through a postMessage to each worker, this was a huge bottleneck resulting in a completion time using 8 threads on an 8th Gen Core i5 processor of 16 minutes and 32 seconds, surely there is a better way right?

Introducing custom scaffold support, you can now initialize Hamsters.js to use a custom scaffold that includes all the libraries you need accessable inside your threads! You will need to define a relative path, absolute path, or url to the worker file you want to use like below.

  hamsters.init({
    scaffold: 'myCustomWorker.js'
  }):

By making use of the scaffold param your threads will make use of your worker file, you can make use of the existing scaffolds the library uses as boilerplate under the src/scaffold folder and the src/common folder as well. Just remember not to remove too many of the libraries methods or you'll lose the benefits of having Hamsters.js in the first place.

Back to that billion row challenge, by using a custom scaffold and moving the file reading into the threads themselves, we can reap the benefits of Hamsters.js parallel computing and features like memoization if we desire. So the results speak for themselves, moving to version 5.6.1 and making the above changes resulted in a processing time of a mere 261.431 seconds! This beats multi-threaded python and concurrent ruby using jRuby which allows for actual parallel processing with ruby. Ruby took 695.027056 seconds, while Python took 446.83 seconds on the same machine.

Making the Hamsters.js approach 165.85% faster than Ruby, and 70.93% faster than python, overall a 279.39% improvement in performance compared to previous releases of Hamsters.js running the same challenge, if you would like to run the challenge yourself or view the source code to see how the custom scaffold was used you can view the repo here.

1-billion-row-challenge

Trainer

Similar to the above in previous versions of Hamsters.js there was no way to customize the response handler for when a thread returns a response, this is now possible by passing a trainer parameter at initialization. This enables you to create custom message handlers and implement things like progress bars and status updates into your workflow, ideal for longer running tasks.

    const hamsterTrainer = (index, task, threadId, hamster, resolve, reject) => {
      const onThreadResponse = (message) => {
        if(message.data.type === "log") { //Custom log message type from our worker thread
          console.log(`We have a log from thread id#${threadId}: ${message.data.message}`);
        } else {  // Process with Hamsters.js like normal
          this.hamsters.pool.processReturn(index, message, task);
          if (this.hamsters.habitat.debug) {
            task.scheduler.metrics.threads[threadId].completed_at = Date.now();
          }
          this.hamsters.pool.removeFromRunning(task, threadId);
          if (task.scheduler.workers.length === 0 && task.scheduler.count === task.scheduler.threads) {
            this.hamsters.pool.returnOutputAndRemoveTask(task, resolve);
          }
          if (!this.hamsters.habitat.persistence) {
            hamster.terminate();
          }
          if (this.hamsters.pool.pending.length() !== 0) {
            const queueHamster = this.hamsters.pool.fetchHamster(this.hamsters.pool.running.length());
            this.hamsters.pool.processQueuedItem(queueHamster, this.hamsters.pool.pending.shift());
          }
        }
      };
      this.hamsters.pool.setOnMessage(hamster, onThreadResponse, reject);
    }
    
    hamsters.init({
       trainer: hamsterTrainer
    });

Async

You can now make use of async await inside of your Hamsters.js functions and everything should work seamlessly as if you were using async await on the main thread. Hamsters.js will automatically detect which functions are async and make the appropriate adjustments behind the scenes.

  hamsters.run(params, async function() {
     await longRunningPromise;
  }, function(results) {
    //Do something with results
  });

Notes

This release of Hamsters.js opens up many more possibilities than in the past, I hope they are made use of and they improve your overall experience using the library, thank you for using the library and helping to make the web a more modern experience. If you enjoy Hamsters.js and want to continue to see updates like this please DONATE today.

Assets 2
Loading