top

Using Web Worker To Compress The Data At Client Side

Problem Statement:You have identified various instances, where you want to send a lot of data from frontend side to your backend. For example, user actions on your website for analysis purposes, a long-form data etc. And if you are doing it recursively, it is a problem as you are using a lot of user’s bandwidth. These are some instances where we need to think about how to compress data at client side with web workers. I have identified 80% difference in the size after the compression.Let’s see, how we can do it:First of all, we need an external javascript library that can compress the data for us. But the main problem here is, we can’t perform compression on the main thread as javascript is single threaded language. Website UI can be blocked if we perform the action on the main thread. So, we will use the web worker for this purpose which improves the usability of web applications. Here are the steps to be followed for compressing data at client side using web worker.Let’s start with an external javascript library. In this case, we are using pako, high-speed zlib port to javascript, works in browser & node.jsWe write little code using this library and create it as Blob, that we will use in Web worker to speed up data compressing.We will post message our data to our library and expect the compressed data in the return.We will use FormData to send the data to the backend.Note: This blog post is about optimizing data compressing with web workers at the client side. Corresponding logic should be written to the backend to read and process the binary form data.We will refer this repository (https://github.com/hemkaran/frontend-data-compress-demo) to reference any code snippet. Clone the repo and play with it, if you want to test out the real things.Let’s create a web worker file first. That will accept the data through post message and compress it in the background thread. After compression, it will be sent to the main thread again through post message. For this, we will use RequireJS AMD build to combine pako library to our code. Below code is listening to post a message and compress the data it will receive.define(['pako'], function (pako) {     'use strict';     onmessage = function (e) {         var data = e.data;         var dataToCompress = data.data;         if (data.action === 'compress') {             for(var key in dataToCompress) {                 if(dataToCompress.hasOwnProperty(key)) {                     dataToCompress[key] = pako.gzip(dataToCompress[key], {level: 4});                 }             }             postMessage({                 action: 'compressed',                 id: data.id,                 data: dataToCompress             });         }     }; });Ref: https://github.com/hemkaran/frontend-data-compress-demo/blob/master/worker.jsLet’s see how to use this library. Since this file will run in the background thread, we will need to load it into web worker through BLOB.1. First of all, fetch the file using ajax call as follows:2. Use the file content data to create a blob and a blob URL, so that we can serve the file using the blob URL3. Now create a worker using this blob URL:4. Now, since the worker is ready. Let’s a create a function that will be called when the worker will be initialized. This will add the  listener of post message to send the data. And also, send two calls, one without compressing the data and one after compressing the data. (Read the comments inline)Note: This URL is created using Postman mock server. To check how to create a mock server, check here: https://www.getpostman.com/docs/postman/mock_servers/setting_up_mock5. Let’s see the full code of test page:/*  This file demonstrates the use of web worker to compress the data.We want to send a very big data to our backend. We will send this data to worker to get it compressed as we can't do compression on the main thread. Worker will compress the data and return it here, so that we can send it to backend  */ (function () {      // Check if browser supports worker @returns {boolean} - true/false           var isWorkerSupported = function () {         return window.Worker && window.URL && window.Blob;     }; //URL of the worker file to load Check if worker is supported in the browser     var workerUrl = '/dist/worker.js';        if (isWorkerSupported()) { // If supported, send a call to fetch the web worker file.Create a blob out of it         $.ajax({             url: workerUrl,             dataType: 'text',             success: function (data) {                 try {    /*Create a blob of the file content, so that we can create blob url out of it, that can be passed to create a web worker file (that runs in the background thread) */                     var blob =  new window.Blob([data], {type: 'text/javascript'});                     var workerBlobUrl = window.URL.createObjectURL(blob);     //workerBlobUrl now is the url to worker file, now we need to create a web worker using this url.                     var worker = new Worker(workerBlobUrl);   // Since, worker is ready, call the function of worker initialized with the worker instance                     workerInitialized(worker);                 } catch (e) {                     console.log('Error in initializing worker file');                 }             }         });     }     function sendData (data) {         $.ajax({ /*This url is created using Postman (https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en) mock server, to check how to create a mock servercheck here: https://www.getpostman.com/docs/postman/mock_servers/setting_up_mock */             url: 'https://cf9d49d3-f0ce-4cf9-90c5-76ad542d1982.mock.pstmn.io/event',             type: 'POST',             data: data         });     }    /*Called when worker will be initialized @param worker. This will be called when worker finished compressing data. you can access data at e.data */     function workerInitialized (worker) {         worker.onmessage = function (e) {             console.log(e.data);             sendData(e.data);         }; // Lets wait for the document to be ready, so that we will enough data to send         $(document).ready(function () {             // Let's create a lot of data to test the compression             var aLongString = '',                 dataToSend;             for (var i = 0; i < 1000; i++) {                 aLongString = aLongString + JSON.stringify(window.performance);             }             dataToSend = {                 name: 'Hemkaran Raghav',                 value: aLongString             };             /*Since data is ready, will be send two calls:             1. First call without compressing the data 2. Second call after worker will compress our data */             sendData(dataToSend); worker.postMessage({                 id: 1,                 data: dataToSend,                 action: 'compress'             });         })     } })(); Ref: https://github.com/hemkaran/frontend-data-compress-demo/blob/master/index.js6. Let’s create an index.html file, that will load our js file. (Note: This file is using jquery for sending calls using $.ajax, you can load it from anywhere or create your own demo without jquery)<!DOCTYPE html> <html lang="en"> <head>     <meta charset="UTF-8">     <title>Frontend compress Demo</title>     <script src="node_modules/jquery/dist/jquery.min.js"></script>     <script src="index.js"></script> </head> <body>     <div>This is a sample page</div> </body> </html>Ref: https://github.com/hemkaran/frontend-data-compress-demo/blob/master/index.htmlFull Source code demo can be found here: https://github.com/hemkaran/frontend-data-compress-demoLet’s run the demo now in the browser. You will see two calls going in the network panel. Compare the request payload of both the calls.Without compression ( 846027 KB)With compression (132470 KB)Here, you can see, we have achieved around 84% compression.((846027–132470)/846027)*100 = ~84Where we can use this:Automated build of web workers compressing client data is generally used where we are tracking a lot of information and wants to send all to the backend. Some of the instances will be:User mouse movement on the page.A long form dataPage’s current HTML snapshotA very large canvas data etc. etc.
Rated 4.0/5 based on 31 customer reviews
Normal Mode Dark Mode

Using Web Worker To Compress The Data At Client Side

Hemkaran Raghav
Blog
23rd Apr, 2018
Using Web Worker To Compress The Data At Client Side

Problem Statement:

You have identified various instances, where you want to send a lot of data from frontend side to your backend. For example, user actions on your website for analysis purposes, a long-form data etc. And if you are doing it recursively, it is a problem as you are using a lot of user’s bandwidth. These are some instances where we need to think about how to compress data at client side with web workers. I have identified 80% difference in the size after the compression.


Let’s see, how we can do it:

First of all, we need an external javascript library that can compress the data for us. But the main problem here is, we can’t perform compression on the main thread as javascript is single threaded language. Website UI can be blocked if we perform the action on the main thread. So, we will use the web worker for this purpose which improves the usability of web applications. Here are the steps to be followed for compressing data at client side using web worker.

  1. Let’s start with an external javascript library. In this case, we are using pako, high-speed zlib port to javascript, works in browser & node.js
  2. We write little code using this library and create it as Blob, that we will use in Web worker to speed up data compressing.
  3. We will post message our data to our library and expect the compressed data in the return.
  4. We will use FormData to send the data to the backend.

Note: This blog post is about optimizing data compressing with web workers at the client side. Corresponding logic should be written to the backend to read and process the binary form data.

We will refer this repository (https://github.com/hemkaran/frontend-data-compress-demo) to reference any code snippet. Clone the repo and play with it, if you want to test out the real things.

Let’s create a web worker file first. That will accept the data through post message and compress it in the background thread. After compression, it will be sent to the main thread again through post message. For this, we will use RequireJS AMD build to combine pako library to our code. Below code is listening to post a message and compress the data it will receive.

define(['pako'], function (pako) {
    'use strict';

    onmessage = function (e) {
        var data = e.data;
        var dataToCompress = data.data;
        if (data.action === 'compress') {
            for(var key in dataToCompress) {
                if(dataToCompress.hasOwnProperty(key)) {
                    dataToCompress[key] = pako.gzip(dataToCompress[key], {level: 4});
                }
            }

            postMessage({
                action: 'compressed',
                id: data.id,
                data: dataToCompress
            });
        }
    };
});

Ref: https://github.com/hemkaran/frontend-data-compress-demo/blob/master/worker.js

Let’s see how to use this library. Since this file will run in the background thread, we will need to load it into web worker through BLOB.

1. First of all, fetch the file using ajax call as follows:

2. Use the file content data to create a blob and a blob URL, so that we can serve the file using the blob URL

3. Now create a worker using this blob URL:

4. Now, since the worker is ready. Let’s a create a function that will be called when the worker will be initialized. This will add the  listener of post message to send the data. And also, send two calls, one without compressing the data and one after compressing the data. (Read the comments inline)

Note: This URL is created using Postman mock server. To check how to create a mock server, check here: https://www.getpostman.com/docs/postman/mock_servers/setting_up_mock

5. Let’s see the full code of test page:

/*
 This file demonstrates the use of web worker to compress the data.We want to send a very big data to our backend. We will send this data to worker to get it compressed as we can't do compression on the main thread. Worker will compress the data and return it here, so that we can send it to backend
 */

(function () {
   
 // Check if browser supports worker @returns {boolean} - true/false
     
    var isWorkerSupported = function () {
        return window.Worker && window.URL && window.Blob;
    };

//URL of the worker file to load Check if worker is supported in the browser

    var workerUrl = '/dist/worker.js';

       if (isWorkerSupported()) {

// If supported, send a call to fetch the web worker file.Create a blob out of it
        $.ajax({
            url: workerUrl,
            dataType: 'text',
            success: function (data) {
                try {
   /*Create a blob of the file content, so that we can create blob url out of it, that can be passed to create a web worker file (that runs in the background thread) */
                    var blob =  new window.Blob([data], {type: 'text/javascript'});
                    var workerBlobUrl = window.URL.createObjectURL(blob);

    //workerBlobUrl now is the url to worker file, now we need to create a web worker using this url.
                    var worker = new Worker(workerBlobUrl);

   // Since, worker is ready, call the function of worker initialized with the worker instance
                    workerInitialized(worker);
                } catch (e) {
                    console.log('Error in initializing worker file');
                }
            }
        });
    }

    function sendData (data) {
        $.ajax({
  /*This url is created using Postman (https://chrome.google.com/webstore/detail/postman/fhbjgbiflinjbdggehcddcbncdddomop?hl=en) mock server, to check how to create a mock servercheck here: https://www.getpostman.com/docs/postman/mock_servers/setting_up_mock */
            url: 'https://cf9d49d3-f0ce-4cf9-90c5-76ad542d1982.mock.pstmn.io/event',
            type: 'POST',
            data: data
        });
    }

   /*Called when worker will be initialized @param worker. This will be called when worker finished compressing data. you can access data at e.data */

    function workerInitialized (worker) {

          worker.onmessage = function (e) {
            console.log(e.data);
            sendData(e.data);
        };

// Lets wait for the document to be ready, so that we will enough data to send

        $(document).ready(function () {
            // Let's create a lot of data to test the compression
            var aLongString = '',
                dataToSend;
            for (var i = 0; i < 1000; i++) {
                aLongString = aLongString + JSON.stringify(window.performance);
            }

            dataToSend = {
                name: 'Hemkaran Raghav',
                value: aLongString
            };

            /*Since data is ready, will be send two calls:
               1. First call without compressing the data
               2. Second call after worker will compress our data */
            sendData(dataToSend);
            worker.postMessage({
                id: 1,
                data: dataToSend,
                action: 'compress'
            });
        })
    }
})();

Ref: https://github.com/hemkaran/frontend-data-compress-demo/blob/master/index.js

6. Let’s create an index.html file, that will load our js file. (Note: This file is using jquery for sending calls using $.ajax, you can load it from anywhere or create your own demo without jquery)

<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <title>Frontend compress Demo</title>
    <script src="node_modules/jquery/dist/jquery.min.js"></script>
    <script src="index.js"></script>
</head>
<body>
    <div>This is a sample page</div>
</body>
</html>

Ref: https://github.com/hemkaran/frontend-data-compress-demo/blob/master/index.html

Full Source code demo can be found here: https://github.com/hemkaran/frontend-data-compress-demo

Let’s run the demo now in the browser. You will see two calls going in the network panel. Compare the request payload of both the calls.

  1. Without compression ( 846027 KB)

    Without compression

  2. With compression (132470 KB)

    With compression

Here, you can see, we have achieved around 84% compression.
((846027–132470)/846027)*100 = ~84


Where we can use this:

Automated build of web workers compressing client data is generally used where we are tracking a lot of information and wants to send all to the backend. Some of the instances will be:

  1. User mouse movement on the page.
  2. A long form data
  3. Page’s current HTML snapshot
  4. A very large canvas data etc. etc.

Hemkaran

Hemkaran Raghav

Blog author

I am a Sr. Software Engineer at Wingify, hacking with JavaScript professionally.

Leave a Reply

Your email address will not be published. Required fields are marked *

SUBSCRIBE OUR BLOG

Follow Us On

Share on

other Blogs

20% Discount