Blindfolded Cartography

This is an adaptation of a talk I gave at the 2015 OpenVis Conference in Boston, expanded in a few spots and abbreviated in others. You can see slides and, ugh, video from the talk.

At Axis Maps, a rough napkin sketch of our projects often looks like this:

Map viewer sketch

We’re tasked with designing and building interactive maps of data that, for one reason or another, we can’t yet see. Sometimes the client doesn’t have data ready soon enough. Sometimes we expect data to change or be added after our work is done. Sometimes it’s just too vast for us to lay eyes on all of it. Whatever the reason, this reality of web mapping is a significant departure from what we knew back in cartography school.

Traditional cartography allows—encourages, perhaps—perfectionism. We were taught attention to detail, and to find and tell the story in our data. Crafting beautiful, effective maps and telling stories with them is one thing when you can obsess over every detail; it’s quite another when you have to do it as though blindfolded. (And I don’t mean sea monsters and mapping unknowns; I mean mapping real things that are known but somehow inaccessible to the cartographer.)

Compromise

As always in cartography, this is all about making compromises. We make design choices that we know are not are not ideal for most cases, but are at least acceptable for all cases. What follows is a list of some areas where we’ve found the “blindfold” particularly challenging, and some compromises that we or others have found helpful. And an occasional shrug where we haven’t found good answers yet. Much of this is about design, some of it is about code (JavaScript), some of it isn’t even about maps, and none of it is perfect.

Data classification

John Nelson has a succinctly handy explanation and demonstration of why data classification decisions are important on choropleth maps. One way of dividing data into bins can produce a starkly different map from another way, potentially in a functionally and aesthetically bad way.

Data classification example by John Nelson
Data classification example by John Nelson

We want to devise a classification scheme around several questions: Will the map be useful and look good? Are the class breaks meaningful? Are they understandable? Are they pretty numbers? As usual, this isn’t too difficult when dealing with a single data distribution. But we need satisfactory answers to those questions for any data set, without our manual intervention. And real data… well, real data are always messy.

Data in designs vs data in reality

Two common solutions are Jenks optimal breaks, where class breaks are calculated to optimize both grouping and spacing, and quantiles, in which each bin contains the same number of values. Both offer some assurances of visible spatial patterns, but both have drawbacks. For example, optimal breaks can be hard to understand, and quantiles can inappropriately group or split values.

Optimal breaks and quartiles

One compromise we’ve used can be described in two parts:

1. Unevenly distributed percentile breaks. Quantiles are friendly, readable, and somewhat reliable, so we often err toward them. Instead of dividing data into evenly sized classes, though, we might break down some of the classes further to separate or highlight extremes or important breakpoints, for example the top 5% in the histogram below.

A compromise classification

2. Use percentiles of unique values, not all values. Datasets often contain many duplicate values; when quantiles are calculated properly, a single value could span several classes. (A common example is having a whole bunch of zero values.) To avoid weird map legends and to better bring out geographic patterns, we might base our classification on the distribution of only unique values, avoiding duplicates.

// basic example
function getValueAtPercentile( data, percentile ){
  return data[ parseInt( (data.length - 1) * percentile ) ];
}
// using underscore.js
var dataArray = _.sortBy( [ 0, 0, 12, 23, 2, 5, 0, 5, 19, 0, 0, 0, 33, 9, 25, 0 ], Number );
var uniques = _.uniq( dataArray, true );
getValueAtPercentile( dataArray, .25 ); // = 0
getValueAtPercentile( uniques, .25 ); //  = 5

null

Expect holes in the data. Assume that sometimes data will be bad or missing. Catch these missing values in code, and, importantly, design for them. “No data” is data too.

Andy Kirk gave a talk called “The Design of Nothing: Null, Zero, Blank” at last OpenVis in 2014, covering (among other things) some no-data scenarios. It’s worth a look:

There are two sides to handling missing data. On one side is code that won’t break if a null value is thrown at it. Basically this means a lot of error handling and bailing out of functions if they’re passed null values. But it also requires some understanding of the data format. A missing value could come through in a variety of ways.

// no data might be...
null
undefined
NaN
""
"NULL"
-9999
// &c. &c.

 
Be careful to avoid the JavaScript gotcha of equating zero with “no data”—usually not the same thing in reality.

// a tempting way to catch "no data" in javascript
if ( data ){
  // yay, we have data!
} 

// but watch out!
var data = 0; // zero is real data
if ( data ){
  // zero won't get us in here :(
}

 
The other side is design. One common scenario is missing data in a choropleth map. We like using texture to indicate missing data. It’s distinct, not easily confused with colored values; and it’s explicit, keeping “no data” as part of the story instead of hiding it. If you’re working with D3, check out Textures.js for easy texturing.

choropleth textures

Another common case is gaps in time series data. Again, it’s good to be explicit. Interpolation can be misleading. For example, I like the dashed line in the iOS Health app. It explicitly indicates a gap in data without losing continuity.

iOS health - gaps in data

On maps, one thing we’ve tried is using a special symbol to indicate a gap in data. Below is a frame from an interactive animated map where proportional symbols remain on the map even when there’s no data for the current year, showing the most recent available data but with a different, hollow symbol.

Gaps in time series data on a map

Text

Text in an interactive map setting is more than just labels. Marty Elmer has written nicely about prose on maps, and how it’s ubiquitous yet overlooked. Text, too, can be an unknown part of map data—one that can ruin otherwise beautiful layouts.

The big lesson from experience here is to restrict the space allotted to text. If you’ve designed for a short sentence, assume you’ll be fed a novel, so make sure text doesn’t overflow all over everything. CSS properties like max-height and overflow:auto can help. For shorter labels, truncation and abbreviation (such as via text-overflow:ellipsis may be useful, but also take advantage of what you do know about the data to be clever about this. For example, in the chart below we knew that some labels would be for congressional districts. If we simply chopped off those labels after a few characters, they’d all be the same, so instead we put the ellipsis in the middle and retained the district number at the end.

Label abbreviations

One last thing: mind your number formatting! It’s easy to forget to design label formats for things like singular versus plural or different orders of magnitude, in which case you end up with some funny-looking labels.

Bad number formatting

The Lorem Ipsum Map

All of this, broadly, is about the challenge of designing around missing or fake data. The “lorem ipsum map” is a phrase used by Rob Roth and Mark Harrower (2008), as a caution against designing user interfaces around a placeholder map. In studying the usability of a map we made for the UW-Madison Lakeshore Nature Preserve, they found some negative reactions to the cold, metallic UI (which we designed first), in contrast to the green and fuzzy map (which we didn’t add until later). It’s hard to offer solid advice about this, other than to suggest trying to understand at least the basic nature of the unknown data and what it might look like. ¯\_(ツ)_/¯

Instead of designing around a blank spot, however, we often generate fake data. That of course can be a challenge itself, because how do we design somewhat realistic fake data when we don’t know what the real data look like? Carolyn Fish has a good term for this goal—”smart dummy data“—and an interesting anecdote about challenges she faced trying to make OpenStreetMap data stand in for totally inaccessible (to her) classified data.

All that said, fake data can be an excellent test of code. In some ways, the more unrealistic the better. If the code can handle ridiculous data scenarios, it can probably handle the real data.

The Human Touch

Big picture time. The ultimate goal, really, is to approximate good, human design when we’re forced to do it through rules and algorithms applied to unknown data. I’m going to keep pointing to Daniel Huffman to explain why this matters, at least until he makes good on his promises to follow up on four-year-old articles.

To illustrate how enormous a task handmade design can be in the web cartography age, consider The Essential Geography of the United States of America by David Imus. You probably saw this map a few years ago when a Slate writer declared it “The Greatest Paper Map of the United States You’ll Ever See.” Indeed, it’s a lovely map with tons of manual details.

The Essential Geography of the United States of America

The map is 1:4,000,000 scale, and according to the Slate article, it took nearly 6,000 hours to complete. 1:4,000,000 is approximately zoom level 7 in the standard slippy map scheme. Just for kicks, let’s extrapolate and say David Imus wanted to design a fully multiscale web map (zoom levels 0 to 18) with the same attention to detail. How long would it take him? I’ll spare you the details of my calculation: it’s 2,803 years and 222 days. And that’s only the United States. Im-freaking-possible, in other words. That’s why we need algorithms.

Big Design

This gets into what I want to call “big design.” So-called “big data” isn’t just a computational challenge; it’s a design problem too. How do we accomplish good human design when there’s simply too much data for a human to see?

The problem is perhaps unique to cartography, where at some level we want to see almost every piece of data we have, not aggregates and distillations. For most of us that means a map based on OpenStreetMap data down to the highest detail levels. With 2 million users mapping all over the world, quality and consistency can’t be 100% guaranteed. Our challenge is to come up with design rules that ensure the map looks pretty good everywhere, at all scales. Yikes.

Nicki Dlugash of Mapbox spoke to this very topic at NACIS 2014. To summarize her advice:

  1. Prioritize areas you care about for your specific use case.
  2. Prioritize the most typical or common examples of a feature.
  3. Look for the most atypical examples of a feature to develop a good compromise.

To help you along the way, there are things taginfo, where you can explore OSM tags and get a sense of things like geographic variation; or things like this view in Mapbox Studio where you can inspect many places at once.

Mapbox Studio "places" view

And of course the nice thing about OpenStreetMap is that you can always fix the data if you find problems!

Good Luck

Well, I hope that assortment of tips and problems has been useful. Good luck as you stumble through the dark!

Blindfolded Mercator

Three mobile map design workarounds

We recently built a map for Napa Valley Vintners, a nonprofit trade association with 500+ members located in one of the most well known wine growing regions in the world. The map is meant to help people locate wineries, plan a visit to the region, and then send directions and a trip itinerary to a phone for easy navigation while driving. Two versions of the Web map were developed, one for the desktop and another for mobile phones. We ran into a couple of issues while working on the mobile version, in particular, that were solved with a design workaround or three. They’re relatively small things, but should have a positive impact on how users experience the map.

Browser chrome – minimize before entry

The browser chrome is all the stuff that appears around a window, taking away screen space from what you’re trying to look at. On a phone, when maximized, it usually appears as an address bar at the top and tool bar at the bottom for various things like page navigation, sharing and bookmarking. The bottom bar can obscure content, which isn’t great if you want to use that space for other things, like parts of a map interface, or if you simply want more map area showing on the screen. Typically, the browser chrome minimizes automatically when scrolling down a normal web page via tap+drag down. However, because our map is fixed to the viewport, scrolling down pans to the south instead, and the chrome doesn’t minimize.

What’s the workaround? How do we minimize the chrome so that when users get to the map they as much of it as possible? By forcing people to scroll down a separate introduction page. The page doubles as a loading screen and also includes some branding, a background image, and an attribution line. When the map finishes loading, a big up arrow with the words “Pull up to view map” appears. Pulling up, which scrolls down, sends people to the map with a minimized chrome.

chrome

The HTML is structured with a few special elements:

<body>
   <div id="loading">
      // 100% width / height
      // loading screen content
   </div>
   <div id="treadmill">
      // absolutely positioned div with height of 100000%
      // no matter how much you scroll, you'll never reach the end
   </div>
   <div id="wrapper">
      // div containing all the map content
      <div class="pane">
         // each UI is given its own div
      </div>
   </div>
</body>

There’s only two bits of javascript you need to get it to work:

$( window ).scroll( function( e ) {
   // wait until the map is loaded to allow scroll
   if( load === false ) return false;

   // check for scroll distance
   if( $( window ).scrollTop() &gt; 100 ) {
      // transition the wrapper into view with CSS
      $( '#wrapper' ).addClass( "visible" );
   }
});

$( ".pane" ).scroll( function() {
  // prevent scrolling on the UI frames and scroll their contents instead 
  return false;
});

Of course, it can come back later! For example on our map, entering a search term will maximize it again. Complicated matters, Safari on iOS doesn’t report it’s window height differently when the state of the chrome changes so it’s not something we can detect in code. For this reason, we decided to position key map ui elements, like the winery “list” button, higher in the window so as not to be obscured. Sure, a little scroll from the header would shrink it again, but it’s not obvious.

Geolocation – ask if the browser can ask

Geolocation is great for plotting your location along a route, fixing your location to the center of the map while driving, or plotting a route from your location to a winery. But none of this works unless you first grant the browser permission to track the phone’s physical location. The problem is that the browser only asks once. Furthermore, that request doesn’t tell people the advantages of accepting. If the one browser request is denied, you get none of the good stuff… ever. Or, at least not until you reset your security preferences, which isn’t obvious and is too complicated to explain in a short help message.

What’s the workaround here? How can we 1) encourage people to allow their location to be tracked, and 2) if it’s denied, still ask for it again later to see if they change their minds? Essentially, we did it by asking if the browser can ask for permission in the first place. In other words, we used a custom notification that both explains the advantages of allowing your location to be tracked and asks if you would like to enable GPS. If the custom notification is denied, we can ask again next time… the browser hasn’t blocked anything because it hasn’t asked anything and the user is sent to the map. If the custom notification is accepted, the user is asked again by the browser. Yes, it is an additional click, but we see this as better than the alternative of a user potentially not having geolocation enabled at all.

geolocation

Data Probe – animation indicates more

Every time a winery point is tapped on the mobile map we display a set of attributes, such as name, address, phone, hours, and appointment details. These show up in a small data probe panel fixed to the bottom of the screen. In addition, the panel includes a button that generates a mapped route and step-by-step directions to the selected winery. Together, the details and button take up all the space available in the data probe panel. A lengthier text description and logo also exist, but flow off the bottom of the screen. This information is accessible by swiping up from the small data probe. The problem was that in user testing no one knew to swipe up and this extra information was rarely even discovered.

We needed a visual cue that would make it clear that more information existed without using up any space in the data probe itself. There was no room for an obvious up arrow here. However, we did find a nice bounce animation at animate.css that served as a good workaround. With each tap of a point, the probe panel bounces up and down a couple times from the bottom of the screen, then stops. It’s a subtle way of showing that more information can be accessed by swiping up.

probe

Swiping up yields the full details for the winery:

probe_full

Lessons Learned

With a huge push towards responsive web design, this is something we’re going to be doing more and more as users expect to have content tailored for whatever screen they’re using. Mobile maps present a lot more challenges than a standard webpage of long text content. Elements can’t reflow and reposition with a grid scaffolding and some basic CSS width queries and you often feel like you’re fighting against the tendencies of the browser. However, by making mobile design a key part of the total map design process, you can force yourself to think about how mobile users use the map differently; which features are important to them and which should be left for the desktop only. It may feel like building two separate products but the results mean that everyone gets what they need wherever they are.

Device testing

PS – Get ready to buy a bunch of extra phones off eBay because each one is a unique little snowflake which makes testing a nightmare.

Mass Observation Basemap

We recently worked on a project with Adam Matthew Digital mapping British diarists in the UK who wrote during the first half of the 20th century. Most of the 500+ entries deal with observations about the impact of WWII on everyday life.  The map allows users to search and filter diarists by gender and date of birth or view them in smaller groupings by theme (e.g., “Political Opinions”, Women in Wartime”, “London During the Blitz”). Brief bios and excerpts from the diaries themselves are also viewable, as shown in the screenshot of the interface below:

Mass_Observation

One of the fun challenges of the project was designing a tiled basemap that resembles other maps from the time period in terms of look and feel. We needed something that would help set the tone and mood for the project without using an actual historical map made in the 1940s. Although we’ve used actual historical maps for other projects, in this case doing so would have caused too many accuracy issues around the locations of the diarists. Instead we turned to TileMill, where we brought together modern-day data from OpenStreetMap, the Ordnance Survey, and Natural Earth and applied our own design.

mo_animation

After spending some time online browsing maps made in 1930s-40s England and the UK, such as those out of the British Ordnance Survey, we narrowed down a few of the salient design elements we found. Of course, we saw a lot of variability, but typography seemed to have one of the greatest impacts in terms of evoking the look and feel we wanted. Serif fonts with good contrast between thick and thin strokes were common, as was the liberal use of italic for map labels. Map symbol colors were similarly impactful. They ranged across the spectrum but were generally desaturated and flat on light-colored backgrounds (very few gradients, glows, drop-shadows or other effects). We took note of a few other elements too, like line weights, which tended to be thin (not overly chunky), and line styles, which varied greatly–we found all kinds of dashes, dots and dash-dot combinations.

One other element having a big impact on look and feel was the paper the maps were printed on. Textures in the maps gave the impression of something old and used. They also added a tangible and personal quality, which seemed a perfect fit for our map of diary entries. Of course, tiled basemaps with textures that look worn or folded are not new  (e.g.,  Geography Class map) but we couldn’t resist making a version of our own since it seemed so applicable for this project. Here is a closer look at the paper texture at zoom levels 9-11:

zoom9zoom10zoom11
We tried a bunch of different fonts for the map (e.g., Cochin, Magneta, Playfair, Essay) and settled on Latienne Pro in the end. It’s not quite as high contrast as some of the others, but thicker strokes help it hold up well at smaller sizes on screen. And its available on TypeKit!

latiennePro

The color palette we arrived at is made up of mostly desaturated colors that become even more so when made semi-transparent on our light-colored background, as is the case with the national parks (green) and urban areas (yellow) on the map.

colors

Finally, our background texture was made using scanned images of folded paper and cardboard. After a good bit of trial and error, we worked up a seamless version of the image in Photoshop that could be repeated throughout the map.texture

FatFonts World Population Map

We’re excited to announce that we’re bringing the work of some of our friends to the Axis Maps Store. We’ve always been interested in new ways to encode and represent data. Also, typography. We ♡ typography big time. It was both of these interests that lead us to the work of Miguel NacentaUta Hinrichs, and Sheelagh Carpendale and the FatFonts project. From their site:

Fatfonts are designed so that the amount of dark pixels in a numeral character is proportional to the number it represents. For example, “2″ has twice the ink than “1″, “8″ has two times the amount of dark ink than “4″ etc. You can see this easily in the set of characters below:

Blockfont onelevel cut1

This proportionality of ink is the main property of FatFonts. It allows us to create images of data where you can read the numbers, and represent tables that can be read as images (like the example below).

Blockfontexample1

It’s a fantastic technique for cartographers (especially when making static maps) because the ink density allows you to see geographic patterns while the digits give you the exact values for each grid cells. This bi-visual technique essentially replicates an interactive data probe on a static map.

If you’re interested in using FatFonts in your own maps, you can download the font faces from the project home page.

The FatFonts World Map

If you want to get your hands on your very own FatFonts map, the World Population map is now available at the Axis Maps store. They were a natural fit and quantitative cousin to our typographic maps. Like the typographic maps, the FatFonts maps show the big picture from afar, but reveal their extensive detail as you move closer. From Miguel’s blog:

The poster is made using an equal area projection of the world, and it represents data collected by CIESIN and others. Each grid in the main map, which represents an area equivalent to 200 by 200 km has a 2-level digit FatFont digit in it. That way we can know, with a precision of 100,000 people, how many humans live there.

Since the number of dark pixels of a FatFont digit is proportional to the number that we are representing, we can calculate how many people each black pixel represents. For an A1 poster in the main area of the map at 600 pixels per inch, each pixel represents approximately 1880 people! FatFonts with the orange background are up by an order of magnitude, so there the ink of a pixel represents approx 18,000 people.

It’s a fascinating map that clearly illustrates how unevenly population is distributed across the globe. Particular areas of interest are highlighted and given higher-resolution treatment:

detail_1_600

The FatFonts World Population map is now available from the Axis Maps store. All profits will be re-invested to fund more FatFonts research.

2nd edition of the Boston typographic map

It’s been five years since we first introduced our map of Boston. The popularity of that map started us down a course of typographic mapping that has since expanded to 11 cities. Given the special place that Boston has in our hearts, we felt it was well deserving of a redesign. And to prove just how much we love this city, we started from scratch. In other words, we started at the beginning with a blank canvas, imported a fresh copy of Open Street Map geography and place names, determined a new map extent and layout, and chose new fonts, colors, and character styles. The new Boston, 2nd edition poster is still 24 inches wide by 36 inches tall, but looks and feels dramatically different.

boston_full
Boston 2nd ed. typographic map

Here are some of the most notable changes:

Extent and Map Scale

The 2nd edition covers a much larger area than the original version. We now include the entire city of Boston, plus a number of surrounding towns. Cambridge and Somerville are still present, of course, but we now extend farther north into Medford and Malden. The extension southward is even greater, covering the area all the way down to Dedham, Milton, and the Blue Hills Reservation.

Fitting more territory on the same-sized poster meant a smaller scale map, which in turn called for some design modifications. For example, because streets were crunched closer together at the new scale we reduced font sizes. The smallest text on the map is that of residential streets — now just 6pt in size. It’s still readable yet small enough to prevent excessive overlap in the denser parts of the city. We also removed the underlying neighborhoods layer that existed in the original version. Again, due to the smaller scale, it cluttered the map and was mostly buried by streets anyway.

Area Features

One of the most visually striking design changes was to the area features on the map, including water, parks and “institutions” (a catch all for universities, airports, or other important not-green areas). Solid color fills behind reversed-out white text gives them a bolder look, compared to the original map. Also, the high contrast between these areas and the white map background gives the map more definition overall and makes some features like small parks, narrow rivers, and jetties, easier to see. Achieving strong figure-ground contrast between land and other area features has always been a challenge with typographic mapping because there is only so much ink that a letter can hold. This method of treating areas with solid background fills seems to help the map visually and functionally while staying true to our aim of covering everything with text.

There are a couple of smaller changes worth noting, too:

Wavy water text

We’ve liked the wavy water text style ever since we first applied it to the Chicago map. Variation in text size, placement along curving paths that parallel one another, and the use of opacity masking to give the appearance of overlapping waves all come together to represent the flow and movement of water better than any other technique we know. We hope the redesigned water text in the 2nd edition map does the same.

Tunnels

Boston is famous for its major underground and underwater tunnels. The 2nd edition Boston map is the first to employ a text style specifically for tunnels. Where highway text goes subsurface, we simply reverse its fill and stroke color. For example, Interstate 90 is represented by purple text with a white stroke but where it becomes Ted Williams Tunnel it is reversed and becomes white text with a purple stroke. I can’t help but compare the bright white tunnel text to the bright lights of cars being turned on as they go beneath the surface.

Below are images of the two maps side by side, each covering an area that is approximately 13 by 13 inches on the printed poster. The 2nd ed. covers a lot more ground in the same amount of space, has a bolder look due in part to its color-filled areas, has wavy water text, and a couple of new headlight-style tunnels. Let’s say goodbye to the original Boston map and hello to the 2nd edition!

Boston 2nd ed. typographic map detail
Boston 2nd ed. typographic map detail
Original Boston typographic map detail
Original Boston typographic map detail

AskCHIS NE map: All D3 Everything

Last week saw the launch of AskCHIS Neighborhood Edition, a product of The California Health Interview Survey and the UCLA Center for Health Policy Research, with whom we worked to develop map and chart components for this new interactive tool. The short story of AskCHIS NE is that it is a tool for searching, displaying, and comparing various California health estimates at local levels such as zip codes and legislative districts. Take a look at it if you feel like setting up an account, or you can watch a demo at its launch event (demo begins at 14:00).

AskCHIS Neighborhood Edition

The long story is interesting too, as this is fairly sophisticated for a public-facing tool, so we’d like to share a few details of how the map and charts were made.

We built that big component with the map, bar chart, and histogram. It lives in an iframe, and as a user makes selections in the table above, various URL hash parameters are sent to the iframe telling it what data to load. The map and bar chart then make requests to a data API and draw themselves upon receiving data. Almost everything here uses D3, either for its common data-driven graphic purposes or simply for DOM manipulation. As usual, this much D3 work made it a fun learning experience. And it once again expanded my awe for Mike Bostock and Jason Davies and their giant brains: more than once during this project, I asked about apparent TopoJSON bugs far beyond my comprehension, and each time they fixed the bug almost immediately.

Davies/Bostock mega-brain

Tiled maps in D3

The map shows one of six vector layers on top of a tiled basemap that is a variation on Stamen’s Toner map, derived from Aaron Lidman’s Toner for TileMill project. Normally we use Leaflet for such tiled maps, but our needs for the vector overlays were complex enough that it made more sense to adapt the basemap to the vector layers’ framework (D3) rather than the other way around. Tiled maps in D3 are pretty painless if you follow the examples of Mike Bostock and Tom MacWright. The six vector layers are loaded from TopoJSON files and drawn in the usual fashion in the same Mercator projection as the basemap.

Dynamic simplifcation

The most interesting technical detail of the map (to me, anyway) is that it uses dynamic scale-dependent simplification. This is a nice design touch, but more importantly it ensures that the map performs well. Zip codes need to have reasonable detail at city scale, but keeping that detail at state level would slow panning, etc. tremendously. We took (once again) one of Mike Bostock’s examples and, after some trial and error, got it to work with the tiled map. Here’s a stripped-down example. Simplification is based on the standard discrete web map zoom levels. As the user zooms, the vector layers redraw with appropriate simplification when those thresholds are crossed, with basic SVG transformations for scaling in between those levels. Keep your eye on the black-outlined county in the gif below, and you should be able to see detail increasing with zoom.

Dynamic simplification

Pooled entities

One of the more sophisticated capabilities of this tool is combining geographies for pooled estimates. For example, maybe you want to look at Riverside and San Bernardino Counties together as a single unit:

Pooled entities

The API does the heavy lifting and delivers pooled data; our challenge was to display pooled geographies as a single entity on the map. Although TopoJSON seems to support combining entities (after all, it does know topology), I had no success, apparently because of a problem with the order of points. Instead we use some trickery of masking, essentially duplicating the features and using the duplicates to block out interior borders, as outlined below. If you wonder why we couldn’t just skip step 2, it’s because our polygons are somewhat transparent, so the interior strokes would still be visible if we simply overlaid solid non-stroked polygons.

Combining SVG shapes while preserving transparency

Image export

The biggest source of headaches and discovery was the image export function. Users can export their current map or bar chart view to a PNG image, for example the map below. (Click it.)

Map export

I learned that converting an HTML canvas element to PNG data is built-in functionality, but getting this to work in a user-friendly way was not so simple. For one thing, our map is rendered as SVG, not canvas. Luckily there is the excellent canvg library to draw SVG to canvas, so we build the entire layout above in SVG, then convert the whole thing to canvas (except the basemap tiles, which are drawn directly to the canvas). Pretty cool. Thanks, canvg!

The nightmares begin when trying to trigger a download, not just display an image on screen. Firefox and Chrome support a magic “download” attribute on <a> elements, which does exactly what we want, downloading the linked content (image data, in this case) instead of opening it in the browser. If only everyone used those browsers! Cutting to the chase, we couldn’t find any way to ensure the correct download behavior across browsers without sending the image data to a server-side script that returns it with headers telling the browser to download. The final task was showing a notification while the image is being processed both client-side and server-side, which can take long enough to confuse the user if there is no indication that something is happening. Failing to detect the download start through things like D3’s XHR methods, we ended up using a trick involving browser cookies.

COOKIE?!?!?!?!?!?!?!

Phew! Turns out making a static graphic from an interactive is complicated! But it was a valuable experience—subsequently I’ve even used JavaScript and D3 deliberately to produce a static map. It works, so why not? (Most of this dot map was exported from the browser after drawing it using this method.)

ALL the D3

For all of the above, it’s hard to post code snippets that would make sense out of context, but we hope that the links and explanations are helpful, and are happy to talk in more detail if you are working on similar tasks. There’s a lot more going on than what I’ve described here, too. Suffice it to say, D3 is a pretty amazing library for web mapping!

Offline Web-Maps

Recently I said something silly to a client:

…and because we’re not using a database, you’ll be able to run this map offline on your local computer.

Great! They give lots of presentations where they want to show off the map and this means they wouldn’t have to rely on conference wifi, the most overstretched resource on the planet. We delivered the completed map to be deployed on their site along with the instructions:

Just open index.html in your web-browser and you’ll be up and running.

Nope.

Did you try Chrome instead of IE?

That ol’ chestnut? Still nope.

Why was it working for me but not for them? We developed this map running the files locally so what was the difference between our machines and theirs?

The Development Webserver

Everyone doesn’t use one of these? If you develop stuff for the web, you most likely have some kind of webserver running on your computer (MAMP, WAMP, LAMP, et al) so when you view a project in the browser, the URL looks like:

http://localhost/...

The page is loaded over the same protocol used when it is on the web, it’s just not looking outside your actual computer to do so. However, if you just open any old HTML file in the browser, you’d get something like:

file:///Users/...

This local file protocol means your browser is loading your file without it being initially processed by a webserver, which opens it up to some potential security issues that make the browser say “nah… not loading that guy”. If it’s a really simple page (static HTML, CSS), no problem. It is a problem, however, if you’re running any XMLHttpRequests to asynchronously load data. In D3, these requests look something like:

d3.json()

or any of these. In jQuery, this is the guy you’re looking for:

$.ajax()

The Fix

Once I’ve located all the XMLHttpRequests on the page, it’s a relatively simple job to replace them. In this map, I have this block of code that loads a topojson file:

d3.json("json/states.json", function(error, us) {
   map.selectAll("path")
      .data(topojson.feature(us, us.objects.states).features)
      .enter()
      .append("path")

This loads the states.json file and as soon as it is received, runs the function that takes an error and a variable called us. The second variable contains the contents of the loaded file and that’s the important one for what we’re doing.

There’s two changes to make. The first is to open states.json and at the beginning of the file, add:

var us =

This takes the proceeding JSON object and declares it as the variable us, the same variable used in the success function for the d3.json request. Now, delete the first line of the request (and the closure at the end) so you’re code becomes:

map.selectAll("path")
   .data(topojson.feature(us, us.objects.states).features)
   .enter()
   .append("path")

The new code is still looking for the us variable, but now it isn’t wrapped in the d3.json function that waits for the XMLHttpRequest to finish before executing. It will just run as soon as it gets to that line in the flow of the code. Because of that, you need to make sure that data is available when it gets there.

In your index.html file add the modified states.json file like it was a regular javascript file:

&lt;script src="json/states.json"&gt;&lt;/script&gt;

before the javascript file that executes the code. Just put it at the top of the <head>, below your CSS files. You don’t need to worry about it slowing down the load time of the page because none of these files are being transmitted over the web.

You now have a set of files that will work wherever you send them. Anyone can open your index.html file in a browser and see your map as you intended.

Some Caveats

  1. Watch out for convenience functions like d3.csv() that have conversions built into them. This method only works for files that are already Javascript objects and can be declared as variables. If you must have your data stored as CSV, you’ll need to alter the file to become one long string and parse it yourself.
  2. The variable you use in the external file will be global. Be careful with that. You can’t reuse any of those variables in your code so it’s worth a quick search to make sure the names don’t come up again.
  3. There’s a reason we don’t load our data like this all the time. It will be slow and gross.
  4. The latest version of Firefox appears to be cool with XMLHttpRequests over file:/// so if you’re lazy, just tell your clients to use that instead.

Geography of Jobs: animated mapping with D3

One of our recently completed projects is a new Geography of Jobs map for TIP Strategies. Have a look at it, and read what they have to say about it.

Geography of Jobs

It’s a month-by-month map of job gains or losses over the prior twelve months for most (or all?) of the metropolitan areas of the United States, from 1999 to present. Proportional circles are colored to indicate gain or loss, and the map can either be animated or controlled by moving a slider.

This is a very straightforward project, and we like it as an example of animated proportional symbol mapping using the D3 library. Data are loaded from a CSV file and plotted on a map drawn with D3’s convenient Albers USA projection, which automatically handles Alaska and Hawaii insets. Then we use basic transitions and timers to create animation. The timeline slider uses Bjørn Sandvik’s d3.slider plugin.

Smooth transitions between frames are achieved by tweens (created with D3’s transition method) that are the same length as the frames. Although this may obfuscate specific point values, tweening improves change blindness problems, and is just plain nice to watch. Compare the smooth transitions:

Tween

With an abrupt version:

No tween

Perhaps the latter would be easier to interpret numerically, but in general we expect people to retrieve exact values by poking around the map, rather than watching the animation.

In any case, view source on the map if you’re looking for code examples of animated D3 maps!

Geography of Jobs

In Defense of Bad Maps

Last month at the annual NACIS conference, I gave a presentation called “In Defense of Bad Maps” — an attempt to demonstrate the value of prolific, popular, yet supposedly “bad” cartography on the web; and to propose a small bit of advice on how to approach cartographic sacrifices in the real, professional world.

This was a last-minute submission, not without occasional regret, and as such there are no especially strong arguments here, but my hope the is that the talk provided one or two things to think about as web cartography continues to grow and evolve. Presentation slides have been posted, but as usual they don’t mean much without the spoken component. Here’s a summarized version.

F tha map police!

Now, I was going for something a little trollish, at least in the title, but I’m too mild-mannered (by day) to go on a true rant, so the provocation—while evident, I hope—is minimal. This is the tl;dr synopsis:

  1. There are “bad” maps on the web, they sometimes become very popular in spite of (or even because of) being bad, and they bother cartographers.
  2. “Bad” does not mean they fail utterly and are without use, however.
  3. All of us commit cartographic crimes, as tradeoffs and sacrifices are part of any real-world job.
  4. We can be wise and responsible in practicing “bad” cartography to good effect.
  5. We should reconsider how we conceive of web cartography, and perhaps break away from the rules of old.

Bad maps

A “bad map” could be a lot of things, but here I’m focusing on four categories, a few of which I suspect are related the potential of a map to become popular among a web audience.

  • Breakin’ the law: Rule-breaking maps are easy to identify and pick on. Mercator projection? More like jerk, hate yer projection, amirite??
  • Lack of design: It’s the dreaded pushpin map, which is the web equivalent of the godawful GIS map with a north arrow the size of a baby. It’s easy to accept defaults and crank out a web map in seconds, but defaults are rarely good across the board.
  • Form over function: Some maps pretend to say something but really don’t—they’re eye candy with no real value added by the cartographer.
  • Code over content: Similarly, there is, uh, the code candy map. A map is presented as something useful and important, but in fact exists mainly as a demonstration of its underlying technology. (This “crime” is perhaps more often committed by the people who promote the map, not the cartographer.)

Who cares?

We care about this because, well, it’s what we do. It’s what we’ve done for hundreds of years. Good cartography has been figured out, and we don’t want to take steps backward. Furthermore, perhaps, we’ve so bought into the idea of maps being powerful that we feel bad cartography may even be dangerous in the long run—or else we just want to assert how important and powerful we are as cartographers.

And, in a point to which I’ll later return, I think we look for bad maps because we don’t really embrace the web; we treat it as an obstacle to overcome. Try to find literature on web cartography that doesn’t give substantial treatment to limitations: screen resolution, colors, projection, and so on. Why did we cheer when D3 came along? We cheered because it allowed us to overcome the Mercator projection, not because it opened up so many new possibilities.

Why we do it

When it comes to deliberate carto-crimes, we make bad maps because of money. A client or boss asks us to do something stupid, and we do it because we like having a job. Or we do something bad but eye-catching, and the attention leads to work. Or we don’t have the budget or time to do a good job, so we do something quick and not so good.

Or, besides money, we make bad maps as part of a process of exploration. We’ve always done this. It’s just that it used to mean printing something out that never left the confines of our cubicles, except in the trash. The web is a much more public and open environment in which we’re encouraged to publish and share our processes, which may include bad cartography.

Bad maps are no big deal

My conjecture is simply that most “bad” maps under-inform rather than misinform. That is, they don’t entirely succeed in their purpose, but they do offer some small takeaway. And, if the sacrifices in cartographic integrity mean that the small message is more likely to reach a wide audience, the map is valuable.

Map message

The open, public exploration process is important, too. Maps needn’t always “say something” but rather can be used to prompt questions and discussion. Having more maps out there, even if many of them are technically bad, inspires us to think about a variety of issues, and additionally drives a sort of design competition that will lead to all-around better cartography.

Wise uses of bad cartography

We can, then, judiciously and wisely practice bad cartography for good reasons. Here are some ways:

  • Know the basics. Yes, you should learn basic cartographic rules and best practices. Ignorance is no reason to do anything. And, in our experience, cartographic expertise is respected. Our clients trust our decisions because of it.
  • Design in proportion to meaning. If your map is meant for a silly fluff piece that intends to do no more than raise a smile for two seconds, then sure, go ahead and make it a dumb non-normalized Mercator choropleth map. If it’s meant to explore a complex and important topic, then give it the proper thought toward good, sound cartographic design.
  • Be clear in purpose. If your map is for exploration or experimentation, let that be known. If it’s a code demonstration, bill it as such. Don’t pretend that a map is saying important things about its subject if you didn’t design it to do so. On the web, we can’t control every context in which our maps appear, but we can give them the right start and help prevent them from being carried away for the wrong reasons.
  • Accept feedback. Nothing on the web is permanent or final, and especially if we are trying new things, we should be willing to change them based on feedback from others. You don’t always have the time or resources to change something after you’ve published it, and yes this is the internet so people are probably going to be awfully rude in their comments, but you can still learn something for your next project.
  • Be smart in responding to client/boss requests. If they request something bad, first try to talk them out of it or propose better alternatives. Try some smart business maneuvers to discourage them or shift the responsibility for the decisions. When those things fail, then give in and do the bad thing. It’s not the end of the world.
  • Choose your battles. Continuing on that last point, prioritize the elements of your design. Not everything is worth defending when a client asks for a change. For example, we’ll give up on the look of UI controls most easily, then non-spatial content, then basemap symbology, and then we’ll most strongly defend thematic symbology. We’d rather have a map be not that good than be wrong.

Moving forward

What I mean to say is that bad maps are nothing to worry about, and are not a plague to fight. But we have a chance for things to be different and better, and to escape the same old types of critique.

Let’s study and understand the use and impacts of web maps specifically on massive, public audiences. What is the actual impact of a “bad” but popular map? What makes a map popular in the first place? I’m looking forward to continued attention on web map use from the academic side of cartography. There’s work from Ian Muehlenhaus on online persuasive cartography, Sarah Battersby on Web Mercator, and even more casual folks like Marty Elmer on viral maps.

Let’s continue to educate users on map reading. I feel confident in my use of a bad map, but maybe that’s because I know maps well. That should not be a privilege. More map literacy means more trust of users, which means less dumbing down of maps because we fear users won’t understand them, but also less fear that “bad” maps will be misused or dangerous.

And, importantly, let’s rethink what web cartography is. We cartographers are in a print mindset, no matter how much we think we are web mappers. The rules of web cartography are a mashing together of a bunch of different things: the principles of print cartography, web design, computer science, statistics and “data science,” and whatever else. Crucially, it’s all founded on cartography rules that were developed for static maps. We don’t need to start over, but let’s at least have a thought exercise: what would web cartography look like if we didn’t base its rules and principles on print cartography? The web is different in so many crazy ways from paper; should the rules really be the same?

web cartography

I used to be in that camp that always says, “there isn’t a new cartography; only new underlying technology.” But I’ve been backing away from that lately. True, at the moment the bulk of web cartography isn’t anything truly revolutionary—it’s only modern means of producing, presenting, and interacting with the same types of displays we’ve always had. (Think choropleth, proportional symbols, etc. We jazz these up considerably over paper versions, but fundamentally the cartography is the same.) But maybe there is a new cartography out there. Maybe our fidelity to the rules of paper prevents us from discovering it. And maybe that’s why born-and-bred cartographers are not the ones leading the charge these days.

Above map by Marty Elmer

Yakkin’ ‘Bout Mappin’

Last week, we made an election map that shows how counties voted in relationship to several different demographic variables. It gave us a chance to take value-by-alpha (VBA) mapping one step further than we did after the 2008 election. Back then, we produced a nice little static map. Our new, interactive map is a bit more substantial, having a user interface, loading data, including a charting component, and displaying a data probe with details on mouseover.

Unlike our typical interactive mapping project, this one was rather small in scope. We wanted to make something that could come together quickly and easily and be seen before people stopped caring about the election. There was also no client, so we were free to work however we pleased in order to get done fast. In other words, no one was telling us that we had to make this work in IE7! All said and done, we devoted twenty-eight hours to the map before sharing it on Twitter.

Because the project was short and received a sustained, concentrated effort from each of us, a behind-the-scenes look at its development seems like it might be of interest to other mapmakers. If nothing else, it serves as an example of how three people, working in different parts of the world, interact together online to get work done. Something for the human geographers out there, at least, if not for the cartographers.

What follows is our Campfire transcript covering the duration of the project. Outside of this transcript, there was no video, voice or other written communication between us. The language here has been smoothed out and edited somewhat in order to reduce each thought, question, or decision to its essence, although there are some direct quotations thrown into the mix.

We think about every project, large or small, slow or fast, client or not, in terms of three primary components: data, design, and code. They are essential ingredients of web cartography and what any aspiring cartographer should learn. To that end, the transcript below has been tagged with colored dots that represent the predominant component in play at any given moment in time, plus a yellow dot for instances when our thoughts were mostly on project planning or management issues.

As you scan through, some patterns to note are:

  1. Entries pertaining to all three components, as well as a basic project plan, are found in the first 30 minutes.
  2. The number of entries about data start out heavy and all but disappear on Day 2.
  3. Entries about code pick up steam toward the middle and end of the project.
  4. Entries about design appear rather consistently throughout the project, with a run of back and forth data-design entries in the middle of Day 1 and a similar back and forth run of code-design entries at the end of Day 2. Interesting!

Planning = Planning
= Data
Design = Design
Code = Code

Day 1 – November 7, 2012

8:15 AM Planning “Want to make a map?” -Dave
8:15 AM Planning “Maybe just this one last time. Then I’m retiring.” -Andy
8:20 AM Design How about an election-by-demographics map using the value-by-alpha (VBA) technique?
8:25 AM Code What’s the best technology setup for this? (Polymaps? CSS?).
8:25 AM Planning Let’s get started this way:

  • Andy: prepare the data
  • Dave: get interactive setup going
  • Ben: put together an interface design
8:30 AM Data I’m exploring election data from The Guardian in Excel.
9:05 AM Data What kinds of demographic data would be worth mapping? Which are effected by geography?
9:05 AM Design What kind of chart should accompany the map?
9:15 AM Design Margin of victory versus demographic variable by county sounds like a decent chart.
9:25 AM Data Here’s a Shapefile with geographic and election+demographic data.
9:25 AM Planning We need to share this file with the world.
9:25 AM Data Let’s explore the Shapefile in indiemapper.
9:30 AM Design We should wait and share this data once it’s all cleaned up.
9:40 AM Data Here’s a second version of the Shapefile with our map data.
10:00 AM Data What’s the best way to store the data? A series of JSON files?.
10:05 AM Data What scale is appropriate for the county boundary data? Will we need to zoom in?
10:10 AM Data There’s no need for detailed, large-scale county data.
10:10 AM Data Let’s store data and geography separately.
10:10 AM Data I’m preparing the data in Google Refine.
10:20 AM Design Anyone have good red/blue color specs?
10:25 AM Design Here are two 5-class sequential color schemes, one for red and one for blue.
10:30 AM Data Looks like there are problems with this data.
“It has Obama winning most of Wyoming handily.” -Andy
10:40 AM Data “In Colorado, that’s showing the Constitution Party candidate!” -Andy
10:55 AM Data I’m fighting with pivot tables in Excel. Trying to get sums into columns instead of grouped in rows.
11:00 AM Planning “Back in a moment… need to go do something to my car before it starts raining.” -Andy
11:45 AM Data Here’s a third version of the Shapefile with our map data.
11:50 AM Design Here’s a VBA map showing county margin of victory by population.
11:50 AM Design Switch to black background.
11:55 AM Design Let’s go with 2-class winners. It’s too hard to see both change in color and alpha.
12:05 PM Design Here are new red and blue color specs.
12:10 PM Design Here’s a new VBA map showing county winner (2-class) by population.
12:10 PM Design “That’s looking pretty okay.” -Dave
12:10 PM Design How is population classified? The map looks kinda bright, making it hard to see population differences.
12:10 PM Design Should we go with equal-interval or unclassed population instead of quantiles?
12:20 PM Data “Argh, this stupid data. Chicago is red!” -Andy
12:25 PM Data For unclassed data, opacity should be a percentage of the max, right?
12:25 PM Design “Los Angeles is ruining it for everybody.” -Dave
12:30 PM Data How about capping the population values at a certain threshold?
12:50 PM Design Here’s a new VBA map showing winner by population, capped at the 90th percentile.
12:55 PM Data What about going with population density instead, in order to control for county size?
12:55 PM Design Here’s a new VBA map showing winner by population density.
1:20 PM Data Here’s the fourth version of a Shapefile with our map data.
“Gave up on Excel, used a Python. Er, Python, not a Python. No snakes involved.” -Andy
1:25 PM Design The user interface mockups are done.
2:00 PM Code Maybe we ought to go with D3 so we can use a map projection.
2:05 PM Design Does using VBA even make sense for mapping demographic variables? What if there is no real correlation between the demographic variable and county winner? VBA might be best for things that are magnitudes or certainty.
2:15 PM Design It’s worth a try. In the end, it’s just a bivariate color scheme.
2:25 PM Data Here’s the final version of the Shapefile with our map data.
2:25 PM Planning Let’s tweet that data.
2:25 PM Data I’m still not sure if population or population density makes more sense here. If we think of VBA as a cartogram, shouldn’t we just map totals?
3:05 PM Design Does this map need a legend?
3:10 PM Planning How should we proceed, now that we’ve got data and an interface design?
3:15 PM Planning What exactly are we trying to make here? A static map viewer? A basic interactive map with data probe and linked chart?
3:20 PM Code If it’s a static map viewer, how about using indiemapper + SVG?
3:25 PM Code There are problems getting a good data probe and chart that way, plus the SVG is too huge.
3:30 PM Code D3 and JSON is sounding like the best approach.
3:45 PM Code The D3 shell is made.
“It was an election day sweep for President Robert Smith of the Goth Party.” -Dave
4:10 PM Code Here’s what it looks like with colored data.
4:20 PM Data The JSON files are ready.
4:30 PM Planning Let’s divide up the remaining work (data probe, data loading, chart, and ui).
4:40 PM Design Let’s put the project data on SVN so it’s easier to work together.
4:55 PM Data Any other interesting demographic variables we should be mapping?
5:15 PM Data Here is poverty:
5:30 PM Data Here is uninsured:
6:10 PM Data Data loading is complete for five demographic variables (birth rate, medicare, poverty, uninsured, wealthy, and non-white).
9:05 PM Design We should go back to quantiles for the demographic data so the maps look balanced with respect to alpha.
9:10 PM Design We probably need non-linear alpha steps to make them look right. Differences should be optical, not mathematical.
9:40 PM Data Can we get data indexed by FIPS code in those JSON files?

Day 2 – November 8, 2012

8:30 AM Design Yeah, let’s do 10-class quantiles for the demographic variables.
8:45 AM Data Get rid of Alaska and Hawaii. There’s no data for those in the source.
8:50 AM Code The 10-class quantile maps are online.
8:50 AM Design They still look too bright overall.
8:55 AM Data Let’s problem solve any data issues. Class breaks and data distribution look okay, at least.
9:05 AM Design Let’s try that non-linear alpha scale to even these out visually.
9:10 AM Design “That’s hella-nicer.” -Dave
9:15 AM Design What kind of instructions will people need to understand these maps? E.g., Brightness is NOT margin of victory.
9:25 AM Code A basic chart is working.
9:25 AM Code How can we easily link the map and chart on mouseover?
9:30 AM Code Try looping through counties to find matches.
9:30 AM Design Here’s a new, cleaner, down-pointing triangle PNG for the user interface.
9:30 AM Code How’s that chart looking?
9:35 AM Code Are the fonts in the mockup on TypeKit?
9:40 AM Design Yep, that’s Ubuntu Regular and Condensed.
9:55 AM Code We need a red and blue PNG for the bars in the chart.
10:10 AM Code The new fonts are in.
10:25 AM Design We need a better way to do the x-axis labels on the chart. They get buried and probably aren’t very clear.
10:25 AM Code Let’s drop in an Axis Maps Logo.
10:30 AM Design Does the page feel too tall? Can we fix that without shrinking the map?
10:40 AM Design Here’s a mockup showing new chart labels.
10:55 AM Code Implemented the new chart label design.
11:00 AM Design Let’s put in an active state for the selected variable text in the UI.
11:05 AM Code Implemented the new active state.
11:05 AM Design There’s a problem with positioning the divider in the new chart labels design. How about a text pipe instead?
11:15 AM Code Implemented chart label tweak.
11:20 AM Design Horizontal page scrolling won’t go away.
11:25 AM Code The scrolling problem is fixed.
11:25 AM Design Let’s give the map a final look over.
11:30 AM Code Let’s stick in a new map title and add some instructions for users.
11:30 AM Design Dang. The page gets cut-off in Firefox.
11:45 AM Code The FF page problem is fixed.
11:50 AM Design Dang. The chart isn’t showing up in Safari.
12:05 PM Code The Safari chart problem is fixed.
12:10 PM Planning Let’s tweet this map to the world!