Let’s use the generator function for array iteration in javascript for better performance

This post is regarding how we can use generator functions for faster executions of iterable objects in javascript to achieve more performance and better execution.

JavaScript with some experiment(Generator Function)

So let us first understand what are generator functions and why generator functions?

So What are Generator Functions?

• Generator functions are the normal function that does not execute completely as soon as the function is called.
• It has the capability to suspend its execution and yield the data only when the caller calls the function.
• They return a special type of iterator object called a Generator which helps for the iteration
• To access each iteration, we have two approaches i.e a .next() function and a “for of loop” approach.

So I think we are good with the basic understanding of what are generators, so let's look at some examples that will add more inputs to our understanding.

#Defining a generator function
function* myFirstGeneratorFunction() {
    yield 1;
    yield 2;
    yield 3;
    yield 4;
}
#calling the generator function
//returns a special iterator object
const iterableFunction = myFirstGeneratorFunction(); 
#Approach 1 : Accessing with .next() function
console.log(iterableFunction.next().value) //output 1
console.log(iterableFunction.next().value) //output 2
console.log(iterableFunction.next().value) //output 3
console.log(iterableFunction.next().value) //output 4
#Approach 2: Accessing with for of loop
for(let data of iterableFunction) {
     console.log(data)         // output 1 2 3 4
}

A Generator function is always defined by the asterisk sign (*).

When we execute the .next() method, the generator function returns the following object

{
   "value" : "some-value",
   "done": false
}

The value property contains the yield value of the generator function and the done property indicates whether the generator function is completely executed or not.

Refer to the below image for a better understanding

Fig: Understanding of how generator function works internally

Following are the advantages of the generator function

• Lazy Loading/ Lazy Evaluation
• Memory Optimization

I hope the above example helped us to understand the basics of the generator function. But now let's deep dive into solving some real problem.

Use Case

In the real world of data processing and manipulation, we have always come across a problem where the execution time for lengthy arrays is always a challenge and it becomes more challenging when the dataset is very large.

We always go with a typical approach i.e we make an API call to fetch the data then iterate the data for data manipulation and processing. But just consider a scenario where the dataset fetched is too large and in that case we need to consider the optimal execution time for better performance. It will be really cool if we lazily process our iterable dataset which optimizes the execution time and also gives us a better performance. We can all achieve it by introducing generator functions.

With all this talking let’s experiment with a real example:

• In this experiment, we have a large data set mostly a JSON file of approx 25MB. Let’s refer to these file as large-file.json
• With this JSON file, we will check what is the actual execution time of the complete dataset when we use the generator function and when we don’t use a generator function
• For the experiment, I have chosen the NodeJS 14+ environment for testing our execution.

Let's get started.

Step 1: Initiated a NodeJS project with all default configuration

> npm init

Step 2: Created the index.js file and placed the large-file.json in the folder

Fig1: Placing large-file.json and creating index.js

Step3: Let’s experiment

• Importing the large-file.json in index.js

const fs = require('fs');
let rawdata = fs.readFileSync('large-file.json');
let largeData = JSON.parse(rawdata);

• Creating a generator function

/**
* @desc: Function is defined to yield data lazily (Generator function)
* @param {*} largeData : Array<Object>
*/
function* largeDataSetProcessingGenerator(largeData) {
   const arrLength = largeData.length;
   for(let i = 0; i<arrLength; i++) {
      yield arr[i];
   }
}

• Now let’s create two functions one which normally iterates the data set array with a for loop and one which iterates a generator function using a “for of” loop. As we are experimenting with performance, let’s also log the execution time for each approach.

1. Creating a function for iterating the large dataset without generator functions

//Function defination to calculate execution without generators
function itterateValueWithoutGenerator() {
    const start = new Date().getTime();
    const newFormatedArray = [];
    const arrLength = largeData.length
    for(let i = 0; i<arrLength; i++) {
        newFormatedArray.push(largeData[i])
    }
  
    const end = new Date().getTime();
    const time = end - start; //calculating the execution time
    console.log('New Manipulated Data set length',  newFormatedArray.length); //logging the new array length
    console.log('Execution time: ' + time + 'ms'); //execution time
}

2. Creating a function for iterating the large dataset with generator functions

//Function defination to calculate execution with generators
function itterateValueUsingGenerator() {
    const start = new Date().getTime();
    //Calling the generator function
    const returnvalue = largeDataSetProcessingGenerator(largeData);
    const newFormatedArray = []
    //Iterating iterable object returned by generator
    for(let obj of returnvalue) {
       newFormatedArray.push(obj)
    }
    const end = new Date().getTime();
    const time = end - start;//calculating the execution time
    
   
    console.log('New Manipulated Data set length', newFormatedArray.length); //logging the new array length
   console.log('Execution time: ' + time + 'ms');//execution time
}

• Since we have created both functions, let’s call them and check their execution time

//To get the execution time without implementing generator function
itterateValueWithoutGenerator();
//To get the execution time with by implementing generator function
itterateValueUsingGenerator();

Step 5: Let’s run our experiment

> node index.js

Step 6: Verify the Output

Fig: Output

• In the output, we can clearly verify that the execution time for iterating a generator function is less as compared to the execution time for iterating a data set via for loop approach.

Isn’t it cool? This use case is just the basic implementation to test the capability of the generator function. With the laziness and optimized execution feature of the generator function, we can utilize it whenever we want to handle a very large data set for its manipulation and processing.

You can find the above code in the following Github repository.

https://github.com/kushhingol/Generator-functions-js

Conclusion

1. With the capability of laziness and optimized execution, the generator functions can be utilized on the browser side (web applications) as well where we can render a large amount of data lazily and can also optimize its execution time as well.

2. Since the generator function returns a special iterable object, it becomes very easy for data modification/manipulation or processing as we have access to each object of the iterable array.

3. In the single thread environment of javascript, the generator functions help us a lot by pushing the limits and allowing us to run and execute heavy non I/0 processes in an optimized way.

4. Generator Functions are pretty much a well-known approach in python where large data sets need to be processed in many data science studies and analysis.

I hope the above experiment has given us a more optimized way for array iteration approaches in a single-threaded javascript environment.

Thank you for the read. Stay Safe and Healthy!