How to build an ESM module customization hook in Node.js

Tags:
  • Node.js
  • Node.js v21
  • ESM
  • Customization Hooks
  • Module Resolution
  • Remote Modules
  • URLs
  • Deno

Introduction

Node.js developers often don't realize this but almost every time they make changes to their code, the code doesn't magically run on the V8 engine right away. It goes through a series of steps before it is executed. One of those steps is the module resolution process, which is responsible for loading and executing modules in the correct order.

Disclaimer: This article is for educational purposes only. The code provided is not production-ready and should not be used in a production environment yet. The Node.js team is working on making this feature stable and secure, so it is recommended to wait for the official release before using it in production. This article was written with Node.js v21.7.2 in mind. The code may not work with older versions of Node.js.

In this article, we will explore how to build a custom ESM loader in Node.js. This concept is called a module customization hook in Node.js. Basically, it allows you to intercept the module resolution process and modify it to your liking. One practical example of this is to resolve remote modules based on a web URL, similar to how Deno works.

Imagine being able to run this code:

// @ts-check

// jokes.mjs
import joke from 'https://v2.jokeapi.dev/joke/Any' assert { type: 'json' };
console.log(joke.setup + '\n' + joke.delivery);

Yeah. you can do a node:fetch, you can do any sophisticated stuff you want, but this looks elegant and clean. Nothing beats clean code, especially if it is secure and efficient.

Importing from URLs is experimental in Node.js, you can see from this link that this feature has reached stability 1 (as of April 2024) which means it's not stable and likely to change. On the other hand, Deno has this feature built-in and it's stable, so you can use it without any issues. Looking at the Node.js docs for customization hooks, we can see that the stability is 1.2 which means it is a release candidate and based on the official Node.js website we notice the following message.

(1.2) Experimental features at this stage are hopefully ready to become stable. No further breaking changes are anticipated but may still occur in response to user feedback. We encourage user testing and feedback so that we can know that this feature is ready to be marked as stable

This message is actually much much better than the network imports, so we can safely assume that the customization hooks are more stable than network imports. That being said, let's build our own custom ESM loader in Node.js that resolves remote modules based on a web URL.

Implementing our own loader in Node.js

Custom ESM loaders are modules that intercept and modify the loading process of ESM files. They give you fine-grained control over how modules are loaded, allowing you to perform tasks such as transpilation, redirection, or modification of module contents.

In the real world, you might be already using a custom loader without even knowing it. It abstracts the complexity of the loading process and provides you with a seamless experience. You might be familiar with Webpack loaders which are used to transform files before they are added to the dependency graph.

How do custom ESM loaders work?

  1. Injection: When Node.js starts, it checks for injected resources within the binary. If a bundled script (in a blob) is found, Node.js executes it. Otherwise, it proceeds as usual.
  2. Chaining: Custom ESM loaders can work in tandem (chained) during startup. They execute in reverse order, similar to a promise chain. Loaders can alter almost anything within the loading process.

How to enable such functionality in Node.js?

We can and will use the register method from node:module which you can run before your application code by using the --import flag.

node --import ./register-hooks.js ./my-app.js
// register-hooks.js
import { register } from 'node:module';
register('./hooks.mjs', import.meta.url);

You can also use --import to import a file from a node_modules package. For example if you are working on a ts-node project, you can do:

node --import 'data:text/javascript,import { register } from "node:module"; import { pat
hToFileURL } from "node:url"; register("ts-node/esm", pathToFileURL("./"));' index.ts

Example of a dynamic module resolution

This is very exciting! we're going to build our very own custom ESM loader that dynamically resolves remote modules based on a web URL. Wait a minute, this is what deno is advertising, right? Well, yes, but we are going to do it in Node.js.

To save some time, I have prepared the loader (fully documented) in a file called network-loader.mjs, you can see it below. This file is designed to resolve modules that start with https:// or http:// to their respective URLs. Take a moment to look into this custom loader and understand how it works. It is very well documented and easy to understand.

// @ts-check

/**
 * This function is used to load a resource from a given URL. If the URL does not start with 'https://' or 'http://',
 * it will call the nextLoad function with the url and context as parameters. If the URL starts with 'https://' or 'http://',
 * it will fetch the resource and return a Promise. The Promise will resolve with an object containing the format of the
 * resource, a shortCircuit flag, and the source of the resource as a string. If there is an error during the fetch operation,
 * the Promise will be rejected with the error.
 *
 * @param {string} url - The URL of the resource to load, can be a local file or a remote URL.
 * @param {object} context - The context in which the load function is called.
 * @param {function} nextLoad - The function to call if the URL does not start with 'https://' or 'http://'.
 * @returns {Promise} - A Promise that resolves with an object containing the format, shortCircuit flag, and source of the resource,
 * or rejects with an error.
 */
export function load(url, context, nextLoad) {
  // If the URL doesn't start with 'https://' or 'http://', we don't need to do anything.
  if (!url.startsWith('https://') && !url.startsWith('http://')) return nextLoad(url, context);

  // For JavaScript to be loaded over the network, we need to fetch and return it.
  return new Promise((resolve, reject) => {
    fetch(url)
      .then((res) => res.json())
      .then((data) =>
        resolve({
          format: 'json',
          shortCircuit: true,
          source: JSON.stringify(data),
        }),
      )
      .catch((err) => reject(err));
  });
}

Create a file called index.js which will look like this:

// @ts-check
import joke from 'https://v2.jokeapi.dev/joke/Any' assert { type: 'json' };
console.log(joke.setup + '\n' + joke.delivery);

On build time, the loader will fetch the JSON from the URL and replace the import with the fetched JSON. You can use your imagination to build more complex loaders that can do more complex things, like fetching menus, configuration files, feature flags, etc.

Registering the module loader

To register the custom ESM loader, you can use the --loader flag when running your Node.js application. We are using the --loader flag specifically for this example in order to show you that this is not the recommended way of running your application. You should use the register method from node:module instead. Let's start with --loader for now and see what issues we face. You can run the following command to run your application with the custom loader:

node --loader ./network-loader.mjs index.js

Although our program seems to be working, there are couple things going on behind the scenes.

(node:31840) ExperimentalWarning: `--experimental-loader` may be removed in the future; instead use `register()`:
--import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("./network-loader.mjs", pathToFileURL("./"));'
(Use `node --trace-warnings ...` to show where the warning was created)
What does Santa suffer from if he gets stuck in a chimney?
Claustrophobia!
(node:31840) ExperimentalWarning: Importing JSON modules is an experimental feature and might change at any time

In order to get rid of this warning, let's break down the newer way (Node.js' register function we discussed earlier) of registering loaders. Create a new file called register.js and add the following code:

import { register } from 'node:module';
import { pathToFileURL } from 'node:url';

register('./network-loader.mjs', pathToFileURL('./'));

Now we can run our application with the following command:

node --import ./register.js --no-warnings index.js

Let's see the output of our program:

What does Santa suffer from if he gets stuck in a chimney?
Claustrophobia!

Not bad, right? We have successfully built a custom ESM loader that resolves remote modules based on a web URL. If you want to run this code without register.js you can use the following one-liner command:

node --import 'data:text/javascript,import { register } from "node:module"; import { pathToFileURL } from "node:url"; register("./network-loader.mjs", pathToFileURL("./"));' --no-warnings index.js

Conclusion

In this article, we explored how to build a custom ESM loader in Node.js. We learned how to intercept the module resolution process and modify it to resolve remote modules based on a web URL. We also discussed how to register the custom loader using the --loader flag and the register method from node:module.

You can take this concept further and build more complex loaders that can perform any workflow you need. I have been using the --loader way of running my applications for a while now and it has been working great, but I am excited to see the register method being used in the future. You can now open a PR to your Node.js project and show off your correct and future-proof way of importing module resolvers.

Thank you for reading, unfortunately, I have not enabled comments on my blog yet, but you can reach out to me via email on [email protected] if you have any questions or feedback.

Bonus 🌶️🌶️

I have been told multiple times to publish my articles on Medium 🤮, although I started on Medium back in 2017, I have decided to move to my own blog. The main reason is that I want everything I share to be free and open-source, and I don't want to be part of a platform that is not aligned with my values. I have tried many times to read on Medium, only to find out that I have to pay to read the full article, which is something I will never do ☠️. Information should be free and accessible to everyone, and that's why I have decided to publish my articles on my own blog.

It's not a bad idea to monetize your content, but it's a bad idea to put a paywall in front of it without letting people know beforehand. I hope a message like this inspires you to start your own blog and share your knowledge with the world. Spread the word, share your knowledge, and make the world a better place 🌍, you never know how a person will interpret your words and how it will change their life.