Webpack

Yarn helps us manage our dependencies, and webpack is what allows us to refer to the dependencies in our code.

It’s a little tricky to talk about webpack in the context of our project because webpack’s behavior depends on its configuration file, and one of the things Rails Webpacker does is generate that configuration file from different inputs so we don’t actually see the real webpack configuration in a file.

There’s a partial workaround that will allow us to print out the configuration file, or at least most of it, and I’m going to use it to guide our way through the way webpack works, and then I’m going to talk about how Webpacker simplifies webpack.

The workaround to let us actually see our webpack configuration is to go to the file config/webpack/development.js and add the following lines at the end of the script:

Object.defineProperty(RegExp.prototype, "toJSON", {
  value: RegExp.prototype.toString
});

console.log(JSON.stringify(module.exports, null, 2))

At this point, module.exports is an object containing the webpack config as JSON, so we’re just asking to print it to the console with a two-space indent. We’re also adding a little bit of extra code so that the regular expressions print properly. This won’t get us the full text representation of the webpack config, but I think it’s as close as we can easily get.

With that code in place, if you run the command bin/webpack, you’ll see the entire webpack configuration in your terminal window.

I’m not going to show the entire thing here in one piece; I’m going to break it down to cover webpack’s main concepts. For each section, I’ll discuss the webpack syntax and what it means, and then I’ll discuss how Rails Webpacker uses defaults and convention to generate the configuration.

Mode

I’m going to show all these snippets one by one, but please remember that they are part of the larger configuration file. Also, these snippets may not exactly match the order that your file prints in.

{
  "mode": "development",
  "devtool": "cheap-module-source-map",
}

Unlike Rails, webpack only cares about two modes: development and production (okay, strictly speaking you can set the mode to none, but please don’t do that). This setting sets the NODE_ENV environment variable, and that value might be used within the code to manage features. Setting production is typically used to run some optimization plugins that we are not going to detail here.

In Rails, Webpacker sets this variable based on your Rails environment, if you are not in development, you are in production. So, for example, a staging environment gets the production plugins. But, just to be a little confusing, Webpacker allows you to maintain separate configuration files based on your Rails environment.

So, Webpacker allows you to generate, say, a different environment for testing, but the NODE_ENV environment variable in that configuration will still be set to either development or production by the Webpacker config.

The devtool parameter controls the generation of source maps. A source map is a separate file that relates lines in your originally coded file in TypeScript, CoffeeScript, or what have you with the JavaScript file eventually sent to the browser. With a source map, debugging tools in the browser can show you your original file and not the derived file, making the code easier to navigate.

The cheap-module-source-map map is a somewhat faster version of a regular source map that has line-to-line fidelity, but not column-to-column fidelity. In production, Rails Webpacker sets this parameter to source-map, which generates a separate source map for each file, and connects it to the webpack pack so that the browser can find the map.

The use of source-map as a production default is actually recent in Webpacker—originally it didn’t generate production source maps by default. In fact, the webpack documentation specifically says, “You should configure your server to disallow access to the Source Map file for normal users!” in a yellow alert box. But David Heinemeier Hansson and the Basecamp team had a change of heart after reflecting on how the openness and availability of HTML and JavaScript were pretty important in helping people learn the early web, and the default was changed.

If you want to change the behavior back, you can see the options at the webpack documentation.

Entry

Our generated configuration file has an entry setting, with one entry for each pack. We have defined three packs: the application pack that has the Stimulus code and our CSS; the venue_display pack that covers the React code; and the hello_react pack that Webpacker generated and which it appears I have neglected to remove from the codebase…

{
  "entry": {
    "application": [
      "/<...>/nxns/app/javascript/packs/application.scss",
      "/<...>/nxns/app/javascript/packs/application.ts"
    ],
    "hello_react": "/<...>/nxns/app/javascript/packs/hello_react.jsx",
    "venue_display": "/<...>/nxns/app/javascript/packs/venue_display.tsx"
  },
}

The entry in a webpack application is a file or files that webpack uses as the starting point for a chunk of code. The expectation is that the entry file will import all the other dependencies in the code (sort of like an asset pipeline manifest file), and webpack will use this information to build a dependency graph. A dependency graph is a list of all the dependencies used by the code so that it can resolve imports and then convert all the code to a browser-friendly file.

By default, a webpack configuration will have one entry point at “main”: "./src/index.js". There are two options for the entry syntax. Our application pack, which has two entry files, uses the array syntax. In that syntax, all the entry files are built together, and the build still uses the default name, main.

More commonly, and as our other packs handle it, you can use object syntax, where the key is the name of the pack, and the value is the file name that is the entry point for that pack.

You can also specify a function as the value of the entry, which we’re not going to do, but which you might do if your webpack build depended on a dynamically updated set of files from some external source. It seems like an unusual use case to me, which probably means that I’ll see a dozen examples of it next week.

In general, the webpack documents recommend one entry point for each separate HTML page that uses JavaScript components. The entry points can all reference the same underlying code, but by separating a different entry point for each HTML downloaded page, you can focus each page on only the dependencies relevant for that page. This causes a smaller download on each page load to the client at the slight cost of maintaining multiple different pages on the server, where each page likely would have some overlap with the others. This implies that a Single Page App would have only one entry point.

Although our configuration doesn’t use it, many webpack configurations also have a "context" setting, which gives an absolute path that is used as the base path for all the entry points, preventing you from needing to explicitly specify the entire absolute path for each entry point.

For a Rails app, Webpacker uses a convention to prevent us from spelling out all the entry points explicitly. Webpacker creates an entry point for each file in the application/javascript/packs directory. Our Rails app doesn’t need to specify a separate context setting because Webpacker is happy to generate the entire absolute path for each entry point. Webpacker doesn’t care how you structure the rest of your files, but it does expect every separate pack you want to generate to have a suitable entry file in that directory.

Output

Our generated configuration has some things to say about the output of webpack’s run:

{
  "output": {
    "filename": "js/[name]-[chunkhash].js",
    "chunkFilename": "js/[name]-[chunkhash].chunk.js",
    "hotUpdateChunkFilename": "js/[id]-[hash].hot-update.js",
    "path": "/Users/noel/code/github/noelrappin/nxns/public/packs",
    "publicPath": "/packs/",
    "pathinfo": true
  },
}

There’s a lot here, and all of this has to do with where webpack puts the files that it generates.

Let’s start with the path and publicPath options. The path is the absolute internal drive location where you want webpack to place its completed, browser-ready files.

The publicPath option is the flip side of that. It’s the relative location of the output files when requested by the browser. The public path is made part of any URL that webpack needs to create to refer to assets, so it must be consistent with your server configuration, or the browser won’t be able to access any resources.

The pathinfo option is actually unrelated to where things are stored, If the option is true, then when webpack combines all the source files into the module to be sent to the browser, it will include a comment before each new file indicating where the file comes from. This info is true by default in development and is definitely false by default in production—you normally don’t want to leak that kind of information about your file setup.

Webpack uses the filename property to determine the pattern used to name each individual outputted pack. In this case, the pattern "js/[name]-[chunkhash].js" indicates that JavaScript files will go into the js subdirectory.

The token [name] will be replaced by the name of the pack, and the [chunkhash] token will be replaced by a hash digest based on the entire content of the pack being outputted. The hotUpdateChunkFilename is similar, but for, well, hot updates. Hot updates and “chunks” are webpack code optimization features, and we’re not going to talk about them here.

All of these outputs are generated by Rails Webpacker based on defaults we can edit. There are a lot of other options for output, but most of them are not relevant to us from our Rails application. A later part of the configuration generated by Webpacker uses plugins to place generated CSS files in "filename": "css/[name]-[contenthash:8].css".

Loaders

The fourth big webpack concept is loaders. A loader is somewhat awkwardly named; it’s just a transformation that changes a source file in some way. So, for example, there’s a Babel loader that applies Babel to our code, which is where our TypeScript compiler comes in. Other default loaders handle converting SASS to plain CSS and changing the file name of static file assets. Loaders go into the configuration file as a list inside the rules parameter.

Our default configuration includes a few loaders. The configuration for the Babel loader will give you the general idea:

{
  "test": "/\\.(js|jsx|mjs|ts|tsx)?(\\.erb)?$/",
  "include": [
    "/Users/noelrappin/Noel/nxns/app/javascript"
  ],
  "exclude": "/node_modules/",
  "use": [
    {
      "loader": "babel-loader",
      "options": {
        "cacheDirectory": "tmp/cache/webpacker/babel-loader-node-modules",
        "cacheCompression": false,
        "compact": false
      }
    }
  ]
}

There are two parts here: the test and the use. In our case, the test provides a regular expression. If a file matches that regular expression, then the rule for that loader is invoked (there are other test options that are less commonly used). The regular expression here matches any file that ends with .js, jsx, mjs, .ts, .tsx, .ts.erb or tsx.erb—the erb files can be configured to be parsed by ERb before TypeScript, though I don’t think we are currently so configured. Have a look at the exclude and include rules.

The use option specifies what to do if the file passes the test. In this case, it specifies a loader to pass the file to, and a set of options to pass to the loader.

Rails Webpacker includes a loader for static images, PostCSS processing of CSS or SASS, and Babel for processing JavaScript files. Other loaders can be added manually.

Resolve

You’ll see that our webpack config has sections called resolve and resolveLoader. These sections manage how code in our app can address code that is not in the same file.

The resolve section starts with a list of extensions:

"extensions": [
  ".jsx", ".tsx", ".ts", ".mjs", ".js", ".sass",
  ".scss", ".css", ".module.sass", ".module.scss",
  ".module.css", ".png", ".svg", ".gif", ".jpeg", ".jpg"
],

This list of extensions has a small but important part to play in module resolution. When you import a file without its extension, as in import * as ActiveStorage from "@rails/activestorage", webpack searches for a file with one of these extension—in order—to determine which file to actually import. If you want to explicitly include the extension of the file in the import statement, you must have a wildcard ("*") entry in the list.

Another important entry in this part of the configuration is the list of modules:

"modules": [
  "/Users/noel/code/github/noelrappin/nxns/app/javascript",
  "node_modules"
]

This is the complete list of where webpack should look to find modules, in our case, we look in our app files first, and then in the node_modules directory for third-party modules.

The resolveLoader section is only used to find webpack’s own loaders. In our case, the configuration is:

"modules": [
  "node_modules"
],

which means loaders are only searched for in the node_modules directory.

Plugins

Plugins are small bits of code that can have arbitrary effects on the webpack process. Webpack itself splits a lot of its features into plugins so they can be added only to projects that use them. You can see a list of official plugins online, and there are a lot of third-party ones as well.

As for what plugins our application uses, Rails Webpacker adds four of them to the default configuration, but our little pretty-printer for the environment fails on plugins because they are typically JavaScript objects in their own right and aren’t converted to JSON usefully. So this isn’t exactly what you see in the print-out, but it is what our configuration resolves to:

"plugins": [
  {
    "keys": [ // LOTS OF STUFF ],
    "defaultValues": { // LOTS OF STUFF }
 ]

What do all of these do?

The keys and defaultValues are courtesy of the EnvironmentPlugin, which is provided by webpack and makes your entire list of environment variables visible to webpack in case you want to refer to them in your code.

CaseSenstivePathsPlugin is a third party module that prevents conflicts between developers using MacOS and other operating systems. In MacOS files that have the same name but different case, like test.js and Test.js are considered the same, whereas other systems would consider them different files. Thus, a developer on a Mac might reference a file using the wrong case. It’ll work for the Mac but will produce an error for other machines. This plugin avoids that by causing a build error if an import statement doesn’t match the actual case of the file on disk.

The MiniCssExtractPlugin is the plugin that pulls CSS files that we import into a separate CSS file on disk, and the WebpackAssetsManifest is what generates the hash and chunkhash part of the output file so that saved files have a format like application-8f20d660421d3ae59057.js.

Dev server

Our development configuration has a lot of options for configuring the webpack-dev-server program. We can use webpack-dev-server to live reload webpack assets while in development. This is a great lead in to Rails Webpacker, and how to leverage it during development.

What’s next?

In this chapter, we saw how webpack lets us write files in the structure we want and still convert them to files that are manageable for a browser. Our Rails app uses webpack through the wrapper of Webpacker. In the next chapter, we’ll take a look at how Webpacker works to make webpack integrate more easily with Rails.