Animating a temporal ton in a web map

Boss: "We need an animated map of this data set." "It's by zip code." "Over ten years." "By week."
You:

We faced a challenge along the above lines earlier this year when we set out to visualize usage of rotavirus vaccines produced by Merck. Simple-sounding on the surface, it involved some tricky design and back-end work, notably because weekly data by zip code over ten years means more than 17 million data points: a ton of data for a web map to be loading.

First, a brief overview of this map, which is at https://merck.axismaps.io/ or in a video demo below. It shows the percent of eligible children receiving a vaccine each week over approximately ten years at state, county, or zip code level. More detailed numbers are found in a chart at the bottom and by poking around the map. And that’s about it. Simple, right?

Prototyping

This project involved several prototypes to work through design decisions. Although in the end it became a fairly straightforward choropleth and point map, the client and we wanted to explore some map types that we thought might best show the spatial and temporal patterns. Early on we had a request for the map to appear such that the entire United States, even unpopulated areas, are covered, to avoid suggesting that there are areas that the vaccines hadn’t even reached. To this end we tried binning into grid cells, but that comes with a couple of problems.

There are places that zip code tabulation areas don’t touch—because nobody lives there—so ensuring no blank space means a certain minimum grid cell size, which may or may not be a good resolution for the more populated parts of the country.

Zip code centroids binned to various cell sizes

At one point we experimented with variable cell sizes, where each cell contained approximately the same number of zip codes. Big squares mean sparse populations, and small cells mean dense population, where there are a lot of zip codes in a small area. I’m still a little intrigued by this idea, but cartographically the effect is kind of opposite of the intended representation: all cells are meant to be “equal” in a sense, but the larger, sparser cells carry a lot more visual weight.

Variable grid cell sizes with approximately equal numbers of zip codes

A second problem with binning is that it requires aggregations that depend on actually having the necessary data. In this case, we had vaccination rates already aggregated to geographies like zip codes, but we did not have the actual number of vaccinations and the total number of eligible children. Without those, we weren’t able to display actual vaccination rates in a grid. Instead it was something like “percent of zip codes with rates above 50%,” so for example if a cell had 100 zip codes and 40 of them had vaccination rates above 50%, the map would show the cell as 40%. This is a bit too convoluted and may not do a great job at showing real spatial patterns anyway.

Data overload: time

As previously mentioned, weekly data for thousands of geographies over ten years is a boatload of data, way too much for a simple web map to load up front. The default county map would be well over a million values, and that’s one chunky CSV. A more efficient way to handle animated data is to deal with change in values, not values themselves. If a value for a county doesn’t change from one frame to the next, there’s no need to store data for that frame for that county. By pre-processing all the data with some fancy SQL to pull out changes, we can cut down significantly on the amount of data being sent to the map and improve rendering performance.

For states and counties, we use a 10-class equal interval classification, and for zip codes only two classes. Whenever a unit’s vaccination rate moves it from one class to another, we store the date (week), FIPS code, and new class number. If it changes to, say, class 8 and stays that way for six weeks, we don’t end up with six rows of data, but rather just one with the week when the class changed. A snippet of data looks something like this:

date,fips,newclass
...
200951,12015,1
201236,22107,6
201550,53031,3
...

Detailed data with actual vaccination rates is loaded on demand through a simple API to get values for a specific geographic entity.

Probe for actual data values

To further reduce file size and smooth out the animation, we mapped 12-week rolling averages instead of single-week snapshots. The data tends to be unstable and noisy when and where there were lower populations of eligible children, so our hope was that averaging values over time would present a better picture of trends, while also resulting in fewer rows of change in our final CSV.

Data overload: space

Besides the attribute data load, a national map at the level of something like zip codes means too much geographic data. For one, it’s another file size problem; for another, it’s a legibility problem.

Legibility concerns led us to the zip code point map. At a national scale, even county polygons are pushing it in terms of crowding, and most zip code polygons are definitely too small to be discernable. Thus we make you zoom in pretty far before zip codes resolve to polygon representations; before that they’re shown as centroid points, which are still crowded on a national map but are a bit easier to pick out.

Most of the map is drawn as SVG using standard D3 methods, but the zip code point layer is an exception. This many points, some 33,000, do not perform well as vector graphics and instead are drawn to a canvas element. It means some extra work to account for things like interactivity (we can’t just attach mouse handlers and have to search for nearby points on mouse move), but it’s worth it to avoid completely choking on rendering.

Zip code point probe

At the scale where we do show zip code polygons, the problem remains that this is a ton of geographic data. For this we built a simple Node vector tile server that sends the polygons in tile-sized chunks as topojson (and caches them to S3). We calculated and stored zip code centroids in a PostGIS database ahead of time, then can get the tiles by querying for centroids that fall within a tile’s bounds. We use centroids instead of polygon intersections so that each polygon is only drawn on one tile—it’s fine if it spills out into other tiles on the map as in the highlighted example below, but we don’t want it being drawn multiple times.

Zip code in multiple tiles

On the front end, when the user zooms past a scale threshold, the map switches to a standard web Mercator map (using d3-tile) onto which we can load zip code tiles as the map is panned. (As a bonus we can also easily load reference basemap tiles underneath to help with orientation.)

Zip code vector tiles

Recap

A few things we learned about animating a ton of data over time:

  • Animation can be hard to follow especially with so many data points. Explore ways to aggregate data (both spatially and temporally) that might be better than exact data values at showing trends. They may not work out, but it’s worth investigating.
  • Instead of loading all values, try loading only changes in values to cut down on file sizes; let exact values be retrieved in smaller doses on demand.
  • Generalize! Different scales call for different complexities of geometry, and this can go beyond polygon simplification to things like collapsing polygons to points.
  • Don’t be mystified by vector tiles! It’s not too difficult to make your own vector tiles for excessively detailed geodata.

Now hiring: Web Map Developer (Full Time)

Who:

At Axis Maps, our aim is to bring the traditions of cartography to the web. We design and build custom interactive web maps for a wide range of clients, focusing on clear design founded on good cartographic principles. Whether it’s a public health map, a tourist map, a live election map, an interactive museum exhibit, or an animated historical map, we strive to build the design best suited to our clients’ data and audiences.

What:

We are seeking a full-time web map developer to help bring our designs to life. As a developer, you will largely be responsible for writing code that powers our maps, but your cartographic and interactive design skills will also help us make the best maps possible.

We’re looking for these qualities:

  • Skill and comfort with front-end web technologies (JavaScript, HTML, CSS) and development processes. At a minimum you’ll need to be comfortable building and working with interactive web pages, familiar with tools and processes for organizing and building projects, and resourceful enough to find solutions when you’re stuck.
  • Experience with JS libraries commonly used for web mapping, such as Leaflet and D3. We don’t use the same platform for every project, but we do frequently use libaries such as Leaflet and D3, so experience with those and/or other mapping libraries is a big plus.
  • Familiarity with the basics of geographic data, databases, and GIS. There’s no escaping some data processing work for any project, so you should understand what geographic data looks like and how GIS and database operations might help get data into the right format.
  • A good eye for design and an understanding of cartographic principles. We pride ourselves on good cartographic and interactive design, and while design won’t be the main focus of your job, your design sense and cartographic knowledge will be a valuable contribution.
  • Ability to work remotely. Axis Maps is small team that is 100% remote, so the ability to communicate effectively online, keep projects organized, and put in a full day’s work outside an office setting is a must.
  • Willingness to learn. Web cartography is a quickly evolving field, so a willingness to keep pace with changing technology, and to fill in gaps in your knowledge when needed, is important.

Where:

Anywhere, but UK, Western Europe, and central or eastern US are preferred.

When:

Now! Please apply by Friday, 15 September 2017.

How:

Please email Andy Woodruff at andy@axismaps.com and include:

  • A link to a portfolio of your work. (Important! We cannot consider applications without examples of your work.)
  • A brief summary of your background, relevant experience, and interest in this kind of work.
  • Anything else you want to share about your motivations and inspirations related to interactive cartography.

Updated Map Server Instructions

About 4 years ago we wrote a post about setting up a map server with Mapnik and PostGIS. It’s still one of the most popular posts on the site but it’s VERY OLD. I wanted to update it with a slightly easier install method and some newer software. What’s in the stack? I’m glad you asked!

The pancakes again

Unlike the previous guide, this one won’t cover basics of Linux and the command line. It’s also written for a Red Hat Enterprise Linux (RHEL) 7.2 server instead of Ubuntu. Let’s do it.

Read more...

Updates to the San Francisco Typographic Map

San Francisco Poster

Ever since the San Francisco map sold out over the holidays we’ve been eager to get it reprinted and back up for sale. Of course, before doing so, we couldn’t resist making a few changes to refresh and update the design. The new version, pictured above, is the third in six years. Read down the page for a quick rundown of what’s new, or skip it and go straight to the typographic maps store where you can check out the map of San Francisco and our collection of other typographic cities.

Read more...

Probing on a Tiled Map

For the past few weeks, we’ve been working through the soft launch of imagineRio, a project we’ve been working on for a couple of years with Rice University. Fun fact: The Portuguese translation of imagineRio is imagináRio which directly translates to imaginary. There’s more background information about the project on the Rice Humanities Research Center website, but in short, the goal of the project was to create a platform to display spatially and temporally accurate reference of Rio de Janeiro from 1500 to the present day. The current front-end for the project uses these maps to display a range of iconography, including maps, plans, urban projects and images of the city (with viewsheds).

The project has numerous technical challenges (which of course pale in comparison to the challenge of digitizing all that historical data), but I just wanted to focus on one of them for this post: data probing and feature identification on a raster map. I’ve always considered data probing in the browser to be something that is exclusive to vector maps. Raster maps are just a collection of pixels. We don’t know the features that are there so we can’t interact with them. Usually that’s OK. Interactive maps are vector thematic data on top of raster base tiles, right? Not always (and yes, we’ll talk about vector tiles another time, this project started 2 years ago):

  • What if the thing your map is about is the type of thing usually reserved for basemaps (roads, buildings, natural features, etc)?
  • What if you need more map rendering oomph (compositing, labels, etc) than the browser can provide?
  • What if your dataset is just too big for the browser to handle as vectors?
Read more...

Little Design Details in a Simple Map

I wanted to title this post: You Won’t Believe This Cartographer’s 4 Weird Tricks for a Nicer Map. That seemed like a bit much (plus the length of this post got away from me so it’s now more Longreads than Upworthy), but the sentiment isn’t entirely untrue. Design (big-D Design—I would’ve capitalized it even if it didn’t start the sentence) is an intimidating and amorphous topic. Academic cartography provides good guidelines for thematic cartography, but interactivity and user-interface design are often “I know it when I see it” type of things. What follows are 4 quick design concepts and techniques that can be applied in many situations to improve the look and feel of an interactive map.

These concepts were taken from a map we made for the Eshhad project tracking sectarian violence in Egypt. It’s a relatively straightforward map with:

  1. A point dataset with a handful of attributes of various types (date, categories, short / long text, URLs)
  2. A Leaflet implementation with basemap tiles
  3. A responsive design for mobile

These are 3 very common circumstances for an interactive map, which should make these tips transferrable to a wide variety of projects.

Read more...

SVG Effects in Leaflet

We recently finished work on a live election map as part of The Tahrir Institue for Middle Eastern Policy’s parliamentary election coverage. Egypt’s complex (and ever-changing) election laws made this an interesting and challenging project, one that required novel mapping techniques to represent the data.

Stripes

The overview map uses value-by-alpha to display the results. Each district is colored according to the party that won the most seats. Transparency is controlled by the number of seats won in that district (not the number of seats available). Because Egypt uses a proportional system representation for each district, a party wins seats in proportion to how much of the vote they won. This leads to lots of ties, especially in the individual results list where the districts are very small with only 2 - 4 seats up for grabs, and many candidates running unaffiliated with any political party.

Stripes!

Read more...

Zooming In On History

Last weekend saw the close of a six-month exhibition by the Norman B. Leventhal Map Center at the Boston Public Library titled “We Are One: Mapping America’s Road from Revolution to Independence,” a piece of which we at Axis Maps had the pleasure of creating. A commemoration of the 250th anniversary of the Stamp Act, the exhibition featured an impressive collection of contemporary maps from the years prior to the American Revolution through the early years of the new nation, documenting and providing context for the events leading to American independence.

A few maps from 'We Are One'

Read more...

Quick tip about Javascript architecture

Most of the work we do here at Axis involves coding Javascript interactive maps. Frequently, the map will have elements controlled by UI components and/or charts and/or timelines. Calling each of the map functions after a UI component gets interacted with (or chart/timeline/etc…), can get exponentially convoluted if you aren’t careful.

If you are fairly new to coding you may not be familiar that code can have design patterns. Just like there are cartographic principles that help with everything from layout to typography to color, there are coding principles that help with code clarity, maintainability, and efficiency.

Read more...

D3 web maps for static cartography production

We’re a mapping company that’s most at home on the web. So when a more traditional static cartography job came along, we even took that to the web.

The ongoing project involves producing a set of four maps for each of thirty-three different countries, showing GDP along with flood and earthquake risk. To make 132 maps, clearly we need to set up a good, easily-repeatable workflow. Because the map themes and symbologies are the same for each country, essentially we need a set of templates into which we can throw some data and get an appropriately styled map that we can export for refinement in Illustrator, and finally layout in InDesign.

Ordinarily that’s where GIS software like QGIS or ArcGIS comes in. But using GIS can be a slow, repetitive task. In the interest of creating a faster process suited to our purpose specifically, we thought it would be easier to develop our own web-based tool to handle the parts of the workflow normally handled by GIS. Fortunately, it’s easy to do such a thing with D3.

Read more...