Updated Map Server Instructions

About 4 years ago we wrote a post about setting up a map server with Mapnik and PostGIS. It’s still one of the most popular posts on the site but it’s VERY OLD. I wanted to update it with a slightly easier install method and some newer software. What’s in the stack? I’m glad you asked!

The pancakes again

Unlike the previous guide, this one won’t cover basics of Linux and the command line. It’s also written for a Red Hat Enterprise Linux (RHEL) 7.2 server instead of Ubuntu. Let’s do it.

Provision the server

The first thing we need to do is provision a Red Hat Enterprise Linux server. Amazon EC2 is as good a place as any to do this, but feel free to use any server you’d like. A few things to keep in mind:

  1. You need at least 2GB of memory. On EC2, that’s a medium instance
  2. Make sure you open up at least port 22 so you can access the server via SSH. If you’d like, you can also open up port 5432 for remote Postgres access and 8000 (or anything you’d like) to access services you create via the web.
  3. If you are running this on EC2 and you create your PEM key to log into the server, don’t forget to chmod 400 to allow you to use it to login. Also, your username on EC2 is going to be ec2-user.

Login to the server via SSH

Once you’re all logged in, run a quick software update to make sure everything is up to date:

sudo yum update

Great. We’re like 20% of the way there now. I told you this was going to be easy. Next step is to install Git and clone the install script:

sudo yum install git
git clone https://gist.github.com/5417b515b421a99360ca.git

Now change the permissions on the install script and execute it:

sudo chmod +x 5417b515b421a99360ca/install_mapnik_rhel.sh
./5417b515b421a99360ca/install_mapnik_rhel.sh

That’s it! Sit back and relax and pretend you’re Neo watching the matrix.

The install script

This script is pretty much a copy of Dane Springmeyer’s script for installing on a AWS instance. It sets up a few extra sources and installs all the necessary software. It also includes a few fixes specific to RHEL.

Testing the installation

While the install script was running, I hope you walked away to get a coffee and a snack. If so, you missed any error messages that might have appeared. Just in case there were some problems, let’s run a quick test.

If you did stick around, you may have noticed that the vector tile tests for node-mapnik failed so this installation won’t support vector tiles generation from Mapnik. If you’d like to investigate / fix the problem, please fork the Gist and I’ll update the post to point to yours. If you’d like to see the results of that test again run npm-test from the node-mapnik directory.

For the tests, we’ll use the node-mapnik-sample-code. There are lots of tests and sample code, so it’s a good place to poke around once you’re up and running. First, clone the code onto the server:

git clone https://github.com/mapnik/node-mapnik-sample-code.git

Now install node-mapnik in that directory:

cd node-mapnik-sample-code/
npm install mapnik

Now run the basic rendering code to use mapnik to generate a very simple map:

node render/app.js stylesheet.xml test.png

When you view the file (log onto your server using SFTP and download it), it should look something like this:

It's a test!

Success! You are now the proud owner of a map server. Go get another snack. You’ve earned it.


Updates to the San Francisco Typographic Map

San Francisco Poster

Ever since the San Francisco map sold out over the holidays we’ve been eager to get it reprinted and back up for sale. Of course, before doing so, we couldn’t resist making a few changes to refresh and update the design. The new version, pictured above, is the third in six years. Read down the page for a quick rundown of what’s new, or skip it and go straight to the typographic maps store where you can check out the map of San Francisco and our collection of other typographic cities.

Blue Water

To further accentuate the peninsula, city, and coastline, the San Francisco Bay and Pacific Ocean were filled with a solid blue background color. Blue text in these areas now recedes into the background while the peninsula and city rise to the foreground.

Blue Water

Stronger Parks

Large parks, such as Golden Gate and the Presidio, that had once been light green, were darkened so they would stand out better against the page background. These areas are important landmarks and by increasing the contrast, readers are more quickly able to get oriented.

Parks

Beaches

Ocean, China, Baker, Marshall, and East beaches were added to the west and north sides of the peninsula. So as not to get lost among all the other text, they were given a unique font (Museo 500 italic) and color (beach-brown). The text is oriented to flow with the contours of the coastline.

Beaches

Simplified Title

This isn’t really a neighborhood map, at least not in the way of some of our past efforts, where we’ve shown neighborhood names as a background layer (e.g., Chicago and Madison). So, gone is the long list of San Francisco neighborhoods that once hovered over the Pacific Ocean in favor of a cleaner space reserved for the city name alone.

We hope you enjoy the new typographic map of San Francisco as much as we do!


Probing on a Tiled Map

For the past few weeks, we’ve been working through the soft launch of imagineRio, a project we’ve been working on for a couple of years with Rice University. Fun fact: The Portuguese translation of imagineRio is imagináRio which directly translates to imaginary. There’s more background information about the project on the Rice Humanities Research Center website, but in short, the goal of the project was to create a platform to display spatially and temporally accurate reference of Rio de Janeiro from 1500 to the present day. The current front-end for the project uses these maps to display a range of iconography, including maps, plans, urban projects and images of the city (with viewsheds).

The project has numerous technical challenges (which of course pale in comparison to the challenge of digitizing all that historical data), but I just wanted to focus on one of them for this post: data probing and feature identification on a raster map. I’ve always considered data probing in the browser to be something that is exclusive to vector maps. Raster maps are just a collection of pixels. We don’t know the features that are there so we can’t interact with them. Usually that’s OK. Interactive maps are vector thematic data on top of raster base tiles, right? Not always (and yes, we’ll talk about vector tiles another time, this project started 2 years ago):

  • What if the thing your map is about is the type of thing usually reserved for basemaps (roads, buildings, natural features, etc)?
  • What if you need more map rendering oomph (compositing, labels, etc) than the browser can provide?
  • What if your dataset is just too big for the browser to handle as vectors?

These are all cases where you might choose to render your maps as rasters, but still want to give your users the ability to identify features through data probing and get information on-demand. First, a little background on the tools (or stack) being used for this project. Here they are as a sandwich:

Stack sandwich

Delicious! And here they are as a literal stack of pancakes:

Stack of pancakes

Tasty! All of geographic data is stored in the PostGIS database. Each feature is tagged with a start date and end date, base on its first and last appearance (in that particular form) in the primary source documents. Map tiles are rendered using Mapnik (through Tilelive) based on:

  1. The layers requested by the user (all is the default)
  2. The features available for those layers at the requested year

Once delivered to the browser, the tiles are cached on AWS S3 so they won’t be rendered again (unless the data in the database changes). The API (outside of the tile requests) is handled through ExpressJS.

Hopefully that provides enough context for the technical side of this post. I imagine it’s a stack that’s pretty familiar to lots of you. Personally, I prefer it in sandwich form. The basic flow of data probing on a raster map involves 4 separate functions:

Full probe workflow

  1. The user clicks the map, requesting features at the lat / lon coordinates under their mouse
  2. The API uses PostGIS to identify which features exist at that given location and returns those features back to the browser
  3. The user selects the specific feature they’re interested in and requests details by the feature’s ID
  4. The API returns the outline of the feature to the browser so it can be highlighted

Requesting from the client

In order for the client to request intersecting features from the database, we need to know 2 things:

  1. The coordinates the user clicked
  2. The radius to search for features

First, setup a function that runs every time the map is clicked. The event object that gets passed to that function contains the coordinates we need:

map.on( "click", probe );

function probe( e ){
  var lng = e.latlng.lng;
  var lat = e.latlng.lat;
}

Because this probing will operate on a multi-zoom map, we need to account for difference in zoom levels while the user is probing. At lower zoom levels (more zoomed-out), we need to search a larger radius because the tiny pixel at the very tip of the mouse pointer literally takes up more geographic space than it does at higher zoom levels. Furthermore, because this isn’t as seamless as vector zooming where we can instantly highlight features on mouseover and there is a small amount of waiting involved, we want to cast the widest net so users get the features they’re looking for. We set the search radius on a zoomend event like so:

map.on( "zoomend", function(){
  var zoom = map.getZoom();
  switch ( zoom ){
    case 15:
      probeZoom = 0.0005;
      break;
    case 16:
      probeZoom = 0.00035;
      break;
    case 17:
      probeZoom = 0.0002;
      break;
    default:
      probeZoom = 0.0006;
      break;
  }
});

The units assigned to probeZoom are decimal degrees (which is why they are so small). This was determined mostly by trial and error and you may want to go with smaller numbers depending on the density of your features.

The last thing we should do on the client side is provide a little bit of feedback to the user. Since the request to the server may take a small amount of time, we can prevent duplicate requests and frustrations by letting the user know their request has been received (and we’re working on it, OK). We display a very small animation where the user clicks that runs until the response is received.

Animated user feedback

It’s built using pure CSS so it loads much faster than an animated GIF. It can be places at the mouse cursor if it is appended to the #map div using the x and y properties of the event object passed to the click function.

Finding intersecting features in PostGIS

At this point, we know the geographic coordinates and the search radius for our query. Now, it’s just a matter of asking the database what exists at that location. We’re using ExpressJS to setup the framework for the API. This makes it easy for us to structure our API URLs using a single line of code:

app.get( '/probe/:year/:radius/:coords/:layers?', geo.probe );

This defines the URL pattern, where each variable preceded by a : is a variable that will be available to us in the request object in the geo.probe function. The actual request made by the client is a jQuery $.getJSON() request to http://imaginerio.rice.edu:3000/probe/2013/0.0005/-43.1941,-22.9286/.

With all the parameters delivered to the probe function, we can use the node-postgres client to run our PostGIS query:

SELECT id, name, layer
  FROM basepoly
  WHERE ST_DWithin(geom, ST_SetSRID( ST_MakePoint( -43.1941, -22.9286 ), 4326 ), 0.0005 )
  ORDER BY layer

There are a few PostGIS functions at work here:

  1. ST_MakePoint() creates a new point geometry that the given lon, lat coordinates
  2. ST_SetSRID() defines the spatial reference system for the coordinates passed to the point
  3. ST_DWithin() searches the table for all features with geometry (geom) that is within 0.0005 of the point we created in ST_MakePoint()

Once the query has ended, it’s just a matter of packaging the data up into an object the client can work with and sending it back as a JSON response.

Letting the user choose their own feature

We’ve returned all of the matching features to the browser, with each feature’s name, layer, and unique id. This allows us to present them in an organized way to the user like this:

List of features

It’s important to organize them by layer if the type of feature they are isn’t immediately apparent by its name.

It’s good to note that this step isn’t 100% necessary for all datasets. Our data is dense enough and diverse enough that immediately drawing the vectors for each of the 7 features would be overwhelming and visually messy. Furthermore, it wouldn’t accomplish the user’s task of giving them information on the 1 feature they clicked on (it’s not their fault a lot of stuff occupies the same geographic space). Instead, we’ve given them the tools to browse through the list of matching results which supports the tasks of those who want information on a specific feature, and those who want details on everything nearby.

Pick from the list

This type of probing is also really good for displaying features that otherwise wouldn’t draw on the map because of their size and potential visual dominance. Check out the final feature Centro chosen from the list. It’s a neighborhood / area so it is labeled on the map, but not drawn because the complex borders would be distracting. By adding it to the list of probe-able features, we’re given users a way to see it’s exact boundaries.

Drawing a feature on the map

The final step of the probing process is to highlight the selected feature on the map. This tiny bit of user feedback is really important. It connects the information displayed in the window to the feature it represents on the map. It also makes the user feel like they’re actually interacting with the features on the tiles. To “highlight” a feature on the map, we request a vector data (GeoJSON from the server) and draw it on top of the tiles.

The request from the server uses postgeo (it’s since been updated to dbgeo), to package the geometries returned from the server as nice GeoJSON that can be read into Leaflet using omnivore. The server-side code is very simply:

exports.draw = function( req, res ){
  postgeo.connect( conn );
  var id = req.params.id;
  postgeo.query( "SELECT ST_AsGeoJSON( geom ) AS geometry FROM basepoly WHERE id = '" + id, "geojson", function( data ){
    res.send( data );
  });
}

The ID parameter is passed to the API using a similar URL structure as we setup with ExpressJS before. Outside of its use here, this is an excellent function to have as a part of your API to render GeoJSON for features on-demand. Once it’s drawn in Leaflet, the highlight looks like:

Highlighted

We’ve added a small pseudo-halo around the polygon by drawing the vector twice. It gives it a little more depth and makes it seem less out-of-place when drawn on the pseudo-3D elements on the map. The styling object we use is:

var topStyle = { 
      color: color,
      fillColor: color,
      fillOpacity : 0.2,
      weight : 2,
      radius : 4
    },
    bottomStyle = { 
      color: color,
      fillColor: color,
      fillOpacity : 0,
      opacity : 0.2,
      weight : 6,
      radius : 4
    };

If you use this, make sure to put your mouse interactions on topStyle since that’s the one with the fill.

Wrapping up

There’s a few more steps involved here than the simpler layer.on( "mouseover", showProbe ) that we usually do. However, none of the steps taken on their own are that complicated. If you’re building a medium to large-scale application (at least one big enough to justify PostGIS and Mapnik), you probably have a lot of these functions built into your API already. In fact, this is much more acutely a design and UX problem. How do we deliver the functionality that the user is expecting, without getting them bogged down in the different data formats we’re using to display the data? How do we design an experience that gives them access to the information without needing to understand the minutiae of mapping?

…and those vector tiles I didn’t want to talk about? This is all going to change in 6 months, tops.


Little Design Details in a Simple Map

I wanted to title this post: You Won’t Believe This Cartographer’s 4 Weird Tricks for a Nicer Map. That seemed like a bit much (plus the length of this post got away from me so it’s now more Longreads than Upworthy), but the sentiment isn’t entirely untrue. Design (big-D Design—I would’ve capitalized it even if it didn’t start the sentence) is an intimidating and amorphous topic. Academic cartography provides good guidelines for thematic cartography, but interactivity and user-interface design are often “I know it when I see it” type of things. What follows are 4 quick design concepts and techniques that can be applied in many situations to improve the look and feel of an interactive map.

These concepts were taken from a map we made for the Eshhad project tracking sectarian violence in Egypt. It’s a relatively straightforward map with:

  1. A point dataset with a handful of attributes of various types (date, categories, short / long text, URLs)
  2. A Leaflet implementation with basemap tiles
  3. A responsive design for mobile

These are 3 very common circumstances for an interactive map, which should make these tips transferrable to a wide variety of projects.

Use color sparingly

This is a concept that I first encountered with Edward Tufte, but I think has been best put into practice time and time again by NYT Graphics. This map is hugely effective in its judicious use of color because:

  1. Elements with color draw more attention than elements without.
  2. The most important things on the map should attract most of the reader’s attention.
  3. Therefore, the most important things on the map should have color, while less important elements should not.

What’s the most important thing on the map? It’s the thing that the map is about. For this map, it was the point dataset of sectarian attacks.

Colored points on a subtle map

While the UI elements are certainly necessary for using the map, their importance is communicated through their size and position. The Positron basemap from CartoDB), though necessary to provide geographic context for these points, fades into the background and does not compete for visual attention. It’s great to have so many nice basemaps to choose from!

Color variation

Not only are the points one of the few elements on the map that are colored, they also use a categorical color scheme to display the target type. We decided to map this particular variable for the sole reason of bringing color variation to the map. In doing so, we greatly reduce visual intimidation for the users of the map as lots of 1 kind of point on the map becomes a few groups of slightly less points. Egypt’s geography, clustered around the Nile River and delta, make this even more important. The color variation allows users to more easily pick out single points for probing when they are tightly clustered together.

Color echoes

Color is also an extremely useful tool for reiterating connections between different features in the map. When color is used sparingly, you can easily communicate to the user that things that are the same color are related in some way. We’ve used a colored heading in the fixed data probe, not only to give it some visual weight, but also connect it to the point and tell the user this text here is about that point there.

Color in the fixed data probe

Also—and there’s more on this below—because the fixed data probe is so heavy with text and categorical attributes, we’ve used a subtle color variation to break up the text box from the other attributes in the fixed probe. This is partly to highlight the longer text, but mostly to create visual differentiation to the probe and reduce the text intimidation.

Data probes

Data probes provide information to the user on-demand. For a more in depth look at their design, this post by Ben is still fantastic (though the examples are showing their age). This map uses a 2-stage data probe technique that we’ve been using more and more in out maps that have a similar level of attribute detail.

Floating probe

The floating probe appears over the map. It is visually connected to the corresponding point through its position and a small change in state (larger, highlighted) for the selected point. It appears on an investigative action (mouseover) and shows just a few details, enough for the reader to decide if they’d like to get information about the point. We often include a call-to-action here (click to read more), but given the simplified nature of the map, didn’t feel like us was necessary.

Floating data probe

Fixed probe

The fixed probe appears on the left side of the screen. Its large height allows it to contain scrolling text, which removes the limits on the length of content it can contain. It’s activated through a deliberate action (click) and the corresponding point increases in size again, changes state, and remains persistently active.

Fixed data probe

To avoid text intimidation (in addition to the small color variation), we’ve used headings and changes in font to break the text up into smaller visual chunks and organize it for easier reading.

Mobile probing

For the mobile version of the map, we had to make a few simple changes to the probe:

  1. The floating data probe is removed. More accurately, it’s still there, but you can’t activate it because you can’t hover on a touchscreen. Fortunately, the data is replicated entirely on the fixed probe.
  2. The fixed probe now covers nearly all of the map. It’s not fully modal because we wanted to keep the reader still somewhat connected to the map while they were viewing the details on the probe.

Mobile data probe        Mobile data probe

One more data probe

One more thing on data probes! It’s not just the thematic features on the map that should be probe-able. Any time you use a visual variable to encode data, a data probe can quickly answer the first question about the graphic: How much is that exactly? We’ve added a small little probe to the histogram to quickly clarify both the quantity and the bin (time period) for the selected bar.

Chart data probe

Integrated legends

Not every map needs a legend. This map doesn’t need a persistent legend because:

  1. The categorical color scheme was done for its own sake, to create color variation amongst the points
  2. All of the attribute data (including the categories used to color the map) is available in the data probe

However, this doesn’t mean we should throw the legend out entirely, just be clever about how much space we give it. For this map, we present the legend to the user when they first load the map. After that, it is available integrated into the filter menu for the attribute that we’re mapping.

Integrated legend

By integrating it into an existing control, not only do we save valuable map / UI space, we reinforce the connection between the controls and the map.

Non-map controls

The final thing I wanted to point out is the prev / next buttons at the top of the data probe. These buttons are used to move between points in the map. Clicking this button will activate the next / prev point in time, mirroring a click event that populates the fixed data probe and changes the point to the active marker.

We like including these alternative browsing methods in our maps because—as much as we hate to admit it—not everyone wants to access this information using a map. Some users are interested in a narrative that is entirely non-spatial (and since our data has a temporal element, that’s even more likely). Furthermore, not everyone is comfortable with the basic mechanics of interactive maps (panning / zooming / probing), and we want to make sure the content is still accessible to them.


Eshhad Map - TIMEP
Mapping religious conflict

SVG Effects in Leaflet

We recently finished work on a live election map as part of The Tahrir Institue for Middle Eastern Policy’s parliamentary election coverage. Egypt’s complex (and ever-changing) election laws made this an interesting and challenging project, one that required novel mapping techniques to represent the data.

Stripes

The overview map uses value-by-alpha to display the results. Each district is colored according to the party that won the most seats. Transparency is controlled by the number of seats won in that district (not the number of seats available). Because Egypt uses a proportional system representation for each district, a party wins seats in proportion to how much of the vote they won. This leads to lots of ties, especially in the individual results list where the districts are very small with only 2 - 4 seats up for grabs, and many candidates running unaffiliated with any political party.

Stripes!

Our original solution to this issue was to treat ties as no data. This ended up not working because:

  1. It failed at giving an overall impression of the geographic distribution of the election results by party
  2. There were lots of ties

This was aside from the other challenges of no data on this map, including:

  1. Unaffiliated candidates could also be considered no data because they don’t have a party, but make up a huge section of the candidates running
  2. The Al Nour party is associated with the color black. With the varying transparency of a value-by-alpha map, this reserved every shade of gray (usually used for no data) for this party’s results
  3. Because this was a live map, with updating election results, the real no data was results that hadn’t been released yet

We settled on using a striped pattern to display both parties’ colors when the results are a tie. Fortunately—and as often is the case—there was a fantastic Leaflet plugin that did nearly all of the hard work for us:

//check to see if there are multiple parties stored as an array
if( _.isArray( party ) ) {
  //name the pattern after both parties
  var patternName = party.join( "-" );
  //create the pattern if it doesn't exist
  if( polygonPatterns[ patternName ] === undefined ){
    polygonPatterns[ patternName ] = new L.StripePattern({
      //2 stripes defined as color + space
      color: parties[ party[ 0 ] ].color,
      spaceColor: parties[ party[ 1 ] ].color,
      spaceOpacity: 1,
      angle: -45
    });
    //patterns must be added to the map before use
    polygonPatterns[ patternName ].addTo( map );
  }
  //create a new style object using the pattern
  style = {
    fillPattern: polygonPatterns[ patternName ],
    color: "#ccc",
    //number of seats won controls fill opacity
    fillOpacity: alphaScale( seats, maxSeats ),
    weight: 1
  }
}
//apply the style
layer.setStyle( style );

Shadows

The most challenging part of the brief for this map was in dealing with displaying the changes to the electoral system between the 2011 and 2015 elections. Outside of changes to the law, the most visible change was to the electoral districts themselves, and this was something we needed to show on an already visually complicated map.

Shadows!

To display the former boundaries, we decided to use a shadowy / gradienty type line style. We wanted something that was heavy enough to be visible on top of the complex polygon fills, but transparent and fuzzy enough to suggest that it is no longer present on the map. To achieve this style, we used a simple SVG filter.

To start, create the filter as a standalone SVG in the HTML file that contains your map:

<!-- Be sure to set the width / height to 0 -->
<svg xmlns="w3.org/2000/svg" version="1.1" style="width:0;height:0">
  <defs>
    <!-- Reference this filter in the code using the id -->
    <filter id='dropshadow'>
      <feGaussianBlur in='SourceAlpha' stdDeviation='4' />
    </filter>
  </defs>
</svg>

Apply the filter to each _container in the layer group. You may need to include a nested group to get to all levels:

formerGroup.eachLayer( function( l ){
  //if the layer is a LayerGroup, loop through and apply filter to each _container
  if( l._layers ){
    _.each( l._layers, function( m ){
      $( m._container ).css({ filter: "url(#dropshadow)" });
    });
  }
  else{
    $( l._container ).css({ filter: "url(#dropshadow)" })
  }
});

American Election maps are so ubiquitous and it was nice to have a change to embrace the challenges of a new election with new geography and new data. You can view the full map to see more of the data and design challenges and how we worked around them. The exciting thing was being able to solve these problems from a design perspective, and be completely confident we could easily implement whatever solution was deemed best.


Egyptian Parliamentary Election Map - TIMEP
Live parliamentary election map