Some Notes on Optimizing a D3 Visualization

After noticing that a visualization was getting a burst of traffic the last week or so, I decided to make an optimization pass on it.

The visualization uses d3 and requires several large-ish data files. The updated site is viewable here or here. The pre-optimized site can be viewed here.

Details are further below (as well as a list of tools used), but here is a comparison of the "before" and "after" for page loading is shown in the short video below (it might take a moment for the iframe to load). This was built easily via the awesome http://www.webpagetest.org.

Comparison of Page Load Before and After Optimization Pass
You can also view this directly on webpagetest.org
(both tests done from Miami-FL with Chrome using http://www.webpagetest.org)

And, to put that in perspective, here's a comparison of page-loading for news.google.com to my updated site:

Comparison of Page Load of news.google.com to Updated Site
You can also view this directly on webpagetest.org
(both tests done from Miami -FL with Chrome using http://www.webpagetest.org)
And so we start...

The visualization requires that the user download several megabytes of data files, which are then processed a bit before starting the simulation. Much of the lag associated with this is unavoidable, but I was seeking ways to both improve this as I could with this pass, and to inform the user as soon as possible that stuff is actually happening as these steps are occurring. In the previous version of the visualization, it takes a while for a "Loading" message to appear, and the page can seem to just freeze.

One of the first things I used to help diagnose problems was google's PageSpeed Insight tool (there is also a chrome plugin that integrates with devtools). Seeing the initial results was a bit painful, but highly informative. I had heard of the recommendations it made before, but the immediate relevance meant that maybe this time these rules stuck a bit.

There were a number of howlers right away:

  • Having blocking javascript downloads, which delayed when rendering could begin
  • Several of the data files were not being compressed by the server (in this case, googledrive, where some of the sites are hosted)
  • Too many separate javascript files, which were not minified

Data-wise, there was almost 8MB of data files that had to be downloaded before the visualization could begin, and the existing "Loading blah..." messages took a while to appear. Further, as noted above, some of them were not getting compressed prior to being served.

Optimization Pass on the Data Files

The data files are either json, csv, or tab-delimited. Json can be a verbose format, and since one of the json files was for a flat structure, it was a candidate for conversion to csv. This was done with one of the node utilities available - json2csv (there are several others that do this as well).

A quirk of the server behind googledrive (which I have no control over) is that it seems to use the file extension to decide whether to gzip the content or not. In my case, there was at least one large json file that was not for a flat structure, and so I couldn't just convert it to csv. But duh - just renaming the json file to have a "csv" extension worked, and googledrive would compress it then.

This reduced the total size of the required data files, AND resulted in all of them being compressed by the googledrive web server.

Javascript

I had about a dozen separate javascript files being downloaded. While most were being done at the bottom of the body of the html document, in reviewing these I realized that I didn't even need four or five of them anymore at all - they had been used in initial versions of the visualization. One of them was almost 300kb.

The answer, of course, was to make sure all of the javascript was actually needed, and use grunt (which I hadn't done yet - shame on me) to combine and minify the javascript, resulting in a single javascript file to be downloaded.

CSS

I only had a few css files, but I had also peppered some <style> elements in the body of the page, which can impact performance. All of the css was moved to a single file, including the css that had been being downloaded from mailchimp, and this is now minified via the node module clean-css.

Showing Content as Soon as Possible

While it is somewhat unavoidable that some large files will have to be downloaded and processed, it's important to make sure the user knows the page is not stuck or something. For this reason, a high priority is given to showing the barebones html "Loading - please be patient", with an indeterminate progress bar. The progress bar is an animated gif create via the nice online tool at http://www.ajaxload.info/. Further, rather than require an additional trip back to the server to get the image file, it is done with inline base64, which was generated with another nice online tool at http://http://www.base64-image.de/.

With the changes above, the initial rendering of the page occurs fairly quickly now.

Mobile

While I did not make any specific changes for mobile here, I did notice that the iPad has a hard time - I don't know at this point know if it's the calculations or trying to render all of the svg. The interface for the iPhone doesn't show a map at all, and the lists (which are all that is shown) seem to update fine.

To IE or Not to IE

ugh.

IE9-11 supports svg, but its implementation of the spec can be spotty. I get the feeling that IE12 will be a significant improvement, but this might not be out for a while.

Anyway, there is obviously something going on with IE, in both the old and new versions of this, as the map extends on top of the header bar and over the list on the right. I am guessing that it is another unimplemented feature of the svg spec at play here. For a visualization someone might look at for just a minute or so to get the gist of the processes involved, it kind of still works for that. However, I need to fix it, and will leave addressing this to another day. Note that if you don't have a box with IE on it (as I don't anymore), I highly recommend BrowserStack for doing this - they make it so easy to run different browsers virtually in your own browser and step through the browser's debug tools. You can even get heck of a lot done with just the 30-minute free trial.

It is interesting to me that less than 10% of the users hitting this visualization are even using Internet Explorer at all (compared to IE's market share of about 20%, according to http://gs.statcounter.com/). At moments like these, all I can say is "thank goodness".

Further Optimization

There are a number of things I can still look at for improving the performance for this visualization:

  • Do more of the processing offline, and have the user download the generated processed datafiles - I can probably set up a grunt task for that. I think that this would primarily reduce the delay that occurs after all of the data files are downloaded.
  • When the map isn't even going to be used (such as for the iPhone interface), don't even download the map files - this could have a significant impact.
  • Reduce the size of the barebones html and css for the initial page load - there's a lot of stuff in there right now that isn't needed until the visualization is fully loaded.
Summary of Tools Mentioned/Used

  • webpagetest.org - I can't believe I hadn't heard of this. This is an open source project that is primarily being developed and supported by Google. It is an awesome free online tool for debugging/inspecting your site's loading performance.
    • Run tests against your site from locations around with the world, with various browsers and devices
    • Obtain detailed information on page load speed, how long it takes to load the various resources
    • Get specific tips on improving your page load
    • Save previous tests
    • Do visual/video comparisons across sites (including to any other publicly available site), and easily share these comparisons
    • View how your page loaded
    • And the list goes on...
  • Google's PageSpeed Insight tool - online and chrome plug-in. Provides tips on improving the performance of your page/site.
  • BrowserStack - "Live, Web-Based Browser Testing" - so easy to run different browsers virtually in your own browser and step through the browser's debug tools.
  • json2csv, a node module for converting flat json structures to csv
  • clean-css, a node module for minifying css
  • http://www.ajaxload.info/, an online tool for creating animated gif progress bars/spinners
  • http://http://www.base64-image.de/, an online tool for getting the base-64 encoding of png/jpg/gif files

Exploring the US Federal Budget - an Interactive Visualization (d3.js)

I am in the middle of working on a visualization for exploring the US Federal Budget, built with d3.js, bootstrap, and jQuery DataTables. The current version can be found at:

Please let me know if it looks weird on your browser or its use is non-intuitive. A summary is below of browsers I have checked so far.

Exploring the US Federal Budget
http://learnforeverlearn.com/usbudget/
Data: 1962 to 2019

The spending and receipts data files are from the OMB's Public Budget Database, available at http://www.whitehouse.gov/omb/budget/supplemental. Specifically, the "Outlays" csv file and the "Receipts" csv file. These contain both historical and projected data, going back to 1962 and up to 2019. I have spot-checked the rolled-up values against the corresponding values in the Historical Tables file from the gpo (which checked out), but this is still a work-in-progress.

Responsive Design - a Work-in-Progress

Perhaps obvious, but I have decided that every single possible screen size requires a potentially completely different interface design, with unique decision encumbrances for all of them. Each may present a nice little puzzle, but there are a wearying number of them. And it is difficult to predict ahead of time what the necessary adjust will be be: you have to see it.

So far, I have attempted to deal with a handful of cases, a few of which are listed below. I have tested only on Chrome on a Mac, an iPad, and an iPhone. It will be interesting to see how well the other browsers handle it.

Large Screen Desktop, iPad Landscape

In this case, all of the features are available, and you can explore the detail tables.

Large screens and iPad Landscape: you can drill down into detailed tables
Small, But Room at the Bottom

In this case, there are no detailed breakdowns available, but there's sufficient space for the horizontal scroll region at the bottom, where you can select different years.

Small Screen, but Room for the Horizontal Bar
Small, and No Room at the Bottom

In this case, there are no detailed breakdowns available, nor does the horizontal scroll area have enough room (per my current parameters), but you can change years and see the totals and how the spending breaks down in terms of mandatory, interest, and discretionary.

I have yet to deal with iPhone landscape.

Friends Don't Make Friends Read Vertical Text

In some cases, the red bar for the deficit is too small for a label. In these cases, the label dynamically slides out when you select one of these years, and tries to be as non-vertical as possible (there's a little leeway still to optimize, I think). An example is shown below. It's debatable if angled text is any easier to read than vertical text, but maybe it's the thought that counts.

Moving the Deficit label out when necessary,
but not going vertical, on a small screen
Browsers tested so far
  • Chrome on iMac - OK
  • Chrome on iPad - OK
  • Chrome on iPhone Portrait - OK (landscape needs some redesign)
  • (via Browserstack) Chrome on Windows 7 - OK
  • (via Browserstack) Chrome on Windows 8.1 - OK
  • (via Browserstack) Safari on Windows 7 - OK
  • (via Browserstack) Safari on Windows 8.1 - OK
  • (via Browserstack) IE9/10/11 on Windows 7/8.1 - should work now, but not yet confirmed - Something weird was happening with text that's supposed to be inside labels - turned out to be (I think) lack of support for "alignment-baseline" for svg text elements (see this msdn page)
  • Firefox 32 on Mountain Lion - like IE, Firefox does not support "alignment-baseline" - stopped using it and worked around it and things look ok again
  • (via Browserstack) Firefox 31 on Windows 7 - should be OK now - Same issue as IE
  • (via Browserstack) Firefox 31 on Windows 8.1 - should be OK now - Same issue as IE
  • (via Browserstack) Opera 23 on Mountain Lion - OK
  • (via Browserstack) Opera 23 on Windows 7 - OK
  • (via Browserstack) Opera 23 on Windows 8.1 - OK
  • (via Browserstack) Android on Nexus 7 - "Page has become unresponsive" - suggests optimization is getting past the premature stage
Next Steps

I have a healthy number of TODO's yet to get to - figuring out why IE and Firefox are misbehaving on Windows 7/8.1 is the current top priority item (the issue was that neither supports the alignment-baseline tag for the svg text element). There is also some usability considerations that I need to address based on some initial feedback.

A Short Note on Making Your d3.js Viz Responsive

OK - this seems to work, and I wanted to make a note of it. It's not a new thing, but might save you some time when you need to do the same thing.

The basic thing we want to do is re-render a d3.js graph after a user has resized their browser window.

With this approach, on a window resize we reconstruct everything svg-related from scratch. This might not be feasible in your case, but otherwise it can be more involved and it is easy to miss something.

We also use David Walsh's "debounce" plugin that can be used to throttle how often you try to rerender while the user is still resizing. There might be a better way to tell when a user is "done resizing" - please let me know how: it doesn't affect the core idea here, just how often you call the "rebuild from scratch" method.

So, so here is one way that works (even when you change your iPhone from portrait to landscape - sweet yay):

  • Set up debounce when the page loads, via
    
        window.addEventListener('resize', debouncedResize);
    
  • Set up your "debouncedResize" method (this is using a delay of 250ms before calling the rerender)
    
    var debouncedResize = debounce(function() {yourRerenderAfterResizeMethod();},
                                    250);
    
  • Set up your rerender method (it might be a method on another object or whatever)
    
    function yourRerenderAfterResizeMethod() {
    
    //destroy existing svg elements
    d3.select('#idOfYourSvgElementToReconstruct').selectAll('*').remove()
    
    // ... other svg elements you will rebuilding
    
    //make sure you don't have straggler object properties hanging around that are used
    //  when rendering (this bit me once - I was trying to add svg elements to an empty thing, and
    //  all actions were silently ignored)
    
    //Now go to town and build all of your svg stuff, 
    // taking into account the new screen size.
    //Note that I have found (so far) that it seems best 
    // to set the width/height attributes of the svg elements so that
    //  when the user is resizing (and before the rerender is called), 
    //  things don't scale crazy. YMMV.
    
    }
    

From my experience so far, you can prevent potential inadvertent transient wacky scaling while the user is resizing by explicitly setting the width/height of the relevant svg elements. This might only matter when the enclosing element has a width set to 100% of the parent, but the jankiness until the rerender is called can be distracting otherwise.

Popular Posts