The Cost of Javascript Frameworks

🔊 Listen to this post Now in English

There is no faster (pun intended) way to slow down a site than to use a bunch of JavaScript.

The thing about JavaScript is you end up paying a performance tax no less than four times:

1. The cost of downloading the file on the network
2. The cost of parsing and compiling the uncompressed file once downloaded
3. The cost of executing the JavaScript
4. The memory cost

The combination is very expensive.

And we are shipping an increasingly high amount. We’re making the core functionality of our sites increasingly dependant on JavaScript as organizations move towards sites driven by frameworks like React, Vue.js, and friends.

I see a lot of very heavy sites using them, but then, my perspective is very biased as the companies that I work with work with me precisely because they are facing performance challenges. I was curious just how common the situation is and just how much of a penalty we’re paying when we make these frameworks the default starting point.

Thanks to HTTP Archive, we can figure that out.

The data

In total, HTTP Archive tracks 4,308,655 desktop URLs, and 5,484,239 mobile URLs. Among the many data points HTTP Archive reports for those URLs is a list of the detected technologies for a given site. That means we can pick out the thousands of sites that use various frameworks and see how much code they’re shipping, and what that costs the CPU.

I ran all the queries against March of 2020, the most recent run at the time.

I decided to compare the aggregate HTTP Archive data for all sites recorded against sites with React, Vue.js, and Angular detected¹.

For fun, I also added jQuery—it’s still massively popular, and it also represents a bit of a different approach to building with JavaScript than the single-page application (SPA) approach provided by React, Vue.js and Angular.

URLs in HTTP Archive with specific frameworks detected

FRAMEWORK	MOBILE URLS	DESKTOP URLS
jQuery	4,615,474	3,714,643
React	489,827	241,023
Vue.js	85,649	43,691
Angular	19,423	18,088

Hopes and dreams
Before we dig in, here’s what I would hope.

In an ideal world, I believe a framework should go beyond developer experience value and provide concrete value for the people using our sites. Performance is just one part of that—accessibility and security both come to mind as well—but it’s an essential part.

So in an ideal world, a framework makes it easier to perform well by either providing a better starting point or providing constraints and characteristics that make it hard to build something that doesn’t perform well.

The best of frameworks would do both: provide a better starting point and help to restrict how out of hands things can get.

Looking at the median for our data isn’t going to tell us that, and in fact leaves a ton of information out. Instead, for each stat, I pulled the following percentiles: the 10th, 25th, 50th (the median), 75th, and 90th.

The 10th and 90th percentiles are particularly interesting to me. The 10th percentile represents the best of class (or at least, reasonably close to the best of class) for a given framework. In other words, only 10% of all sites using a given framework reach that mark or better. The 90th percentile, on the other hand, is the opposite of the spectrum—it shows us how bad things can get. The 90th percentile represents the long-tail—that last 10% of sites with the highest number of bytes or largest amount of main thread time.

JavaScript Bytes
For the starting point, it makes sense to look at the amount of JavaScript passed over the network.

JavaScript Bytes

For the starting point, it makes sense to look at the amount of JavaScript passed over the network.

JavaScript Bytes Served to Mobile Devices, by Percentile

	10TH	25TH	50TH
All Sites	93.4kb	196.6kb	413.5kb
Sites with jQuery	110.3kb	219.8kb	430.4kb
Sites with Vue.js	244.7kb	409.3kb	692.1kb
Sites with Angular	445.1kb	675.6kb	1,066.4kb
Sites with React	345.8kb	441.6kb	690.3kb

JavaScript Bytes Served to Desktop Devices, by Percentile

 

For the sheer payload size, the 10th percentile turns out pretty much as you would expect: if one of these frameworks are in use, there’s more JavaScript even in the most ideal of situations. That’s not surprising—you can’t add a JavaScript framework as a default starting point and expect to ship less JavaScript out of the box.

What is notable is that some frameworks correlate to better starting points than others. Sites with jQuery are the best of the bunch, starting with about 15% more JavaScript on desktop devices and about 18% more on mobile. (There’s admittedly a little bit of bias here. jQuery is found on a lot of sites, so naturally, it’s going to have a tighter relationship to the overall numbers than others. Still, that doesn’t change the way the raw numbers appear for each framework.)

While even a 15-18% increase is notable, comparing that to the opposite end of the spectrum makes the jQuery tax feel very low. Sites with Angular ship 344% more JavaScript on desktop at the 10th percentile, and 377% more on mobile. Sites with React, the next heaviest, ship 193% more JavaScript on desktop and 270% more on mobile devices.

I mentioned earlier that even if the starting point is a little off, I would hope that a framework could still provide value by limiting the upper bound in some way.

Interestingly, jQuery driven sites follow this pattern. While they’re a bit heftier (15-18%) at the 10th percentile, they’re slightly smaller than the aggregate at the 90th percentile—about 3% on both desktop and mobile. Neither of those numbers is super significant, but at least sites with jQuery don’t seem to have a dramatically worse long-tail in terms of JavaScript bytes shipped.

The same can’t be said of the other frameworks.

Just as with the 10th percentile, Angular and React driven sites tend to distance themselves from others at the 90th percentile, and not in a very flattering way.

At the 90th percentile, Angular sites ship 141% more bytes on mobile and 159% more bytes on desktop. Sites with React ship 73% more bytes on desktop and 58% more on mobile. With a 90th percentile weight of 2,197.8kb, React sites ship 322.9kb more bytes of JavaScript to mobile users than Vue.js, the next closest. The desktop gap between Angular and React and the rest of the crowd is even higher—React-driven sites ship 554.7kb more JavaScript than Vue.js-driven sites.

JavaScript Main Thread Time

It’s clear from the data that sites with these frameworks in place tend to pay a large penalty in terms of bytes. But of course, that’s just one part of the equation.

Once that JavaScript arrives, it has to get to work. Any work that occurs on the main thread of the browser is particularly troubling. The main thread is responsible for handling user input, during style calculation, layout and painting. If we’re clogging it up with a lot of JavaScript work, the main thread has no chance to do those things in a timely manner, leading to lag and jank.

HTTP Archive records V8 main thread time, so we can query to see just how much time that main thread is working on all that JavaScript.

Scripting related CPU time (in milliseconds) for mobile devices, in percentiles

Scripting related CPU time (in milliseconds) for desktop devices, in percentiles

	10TH	25TH	50TH
All Sites	146.0ms	351.8ms	831.0ms
Sites with jQuery	199.6ms	399.2ms	877.5ms
Sites with Vue.js	350.4ms	650.8ms	1,280.7ms
Sites with Angular	482.2ms	777.9ms	1,365.5ms
Sites with React	508.0ms	1,045.6ms	2,121.1ms

 

There are some very familiar themes here.

First, sites with jQuery detected spend much less time on JavaScript work on the main thread than the other three analyzed. At the 10th percentile, there’s a 61% increase in JavaScript main thread work being done on mobile devices and 37% more on desktop. At the 90th percentile, jQuery sites are gain pretty darn close to the aggregate, spending 1.3% less time on the main thread for mobile devices and ..7% less time on desktop machines.

The opposite end—the frameworks that correlate to the most time spent on the main thread—is once again made up of Angular and React. The only difference is that while Angular sites shipped more JavaScript than React sites, they actually spend less time on the CPU—much less time.

At the 10th percentile, Angular sites spend 230% more time on the CPU for JavaScript related work on desktop devices, and 313% more on mobile devices. React sites bring up the tail end, spending 248% more time on desktop devices and 658% more time on mobile devices. No, 658% is not a typo. At the 10th percentile, sites with React spend 2.7s on the main thread dealing with all the JavaScript sent down.

Compared to those big numbers, the situation at the 90th percentile at least looks a little better. The main thread of Angular sites spends 29% more time on JavaScript for desktop devices and 27% more time on mobile devices. React sites spend 130% more time on desktop and 98% more time on mobile devices.

Those percentages look much better than at the 10th percentile, but keep in mind that the bulk numbers are pretty scary: that’s 20.8s of main thread work for sites built with React at the 90th percentile on mobile devices. (What, exactly, is happening during that time is a topic for a follow-up post, I think.)

There’s one potential gotcha (thanks Jeremy for making sure I double-checked the stats from this angle)—many sites will pull in multiple libraries. In particular, I see a lot of sites pulling jQuery in alongside React or Vue.js as they’re migrating to that architecture. So, I re-ran the queries, only this time I only included URLs that included only React, jQuery, Angular or Vue.js not some combination of them.