Growing pains & getting data out of OpenStreetMap

One of the killer features of OpenStreetMap, which makes it completely different to Google Maps and the rest, is that we provide totally free geodata. In fact it’s really the primary purpose of OpenStreetMap – the various maps shown on the homepage are just a tasty preview.

For those of us lacking the time, money and skills enjoyed by some of the cooler data users (i.e. unable to run a dedicated server with a PostgreSQL database and all the programs and storage space needed to maintain an up-to-date clone of the OpenStreetMap database) there are two main ways to grab OpenStreetMap data.

One is to simply download a defined area using the API. The “export” tab on the web site and editors like JOSM make this easy. The disadvantage is that you get everything in that area. If you wanted to get, say, all the power generators in the UK you would need to download and process gigabytes of data for the whole of the UK to pull them out.

Thankfully we have the wonderful eXtended API (XAPI, or “zappy”). This lets you define the features you want to download, and the area you want them for. So I can easily download a small file just containing power generators in the UK.

I have used XAPI for my OpenEcoMaps project. It’s great. It lets me build a web site on cheap shared hosting without needing ninja data processing skills. I love it.

Unfortunately, for the past few months XAPI has gradually become so overloaded that it is useless. I don’t think I have been able to make a batch of five calls for data from XAPI for weeks. OpenEcoMaps tries to download fresh data every hour, to no avail. Downloads either time out, return empty files, or return partial files that stop half-way through.

As a result I’ve not been able to launch OpenEcoMaps, and I’ve had to slow down conversations with local community groups, charities, councils and regional government about various exciting collaboration ideas.

I don’t begrudge hard-pressed volunteers who set XAPI up, especially when so much good work has gone into making the main web site, the default map, the wiki and the underlying database run so reliably.

But I wish and cross my fingers for the day when there is a resource – either from the community or a company – to run a reliable XAPI service. I’d even be willing to pay a small annual fee for access. In the meantime, sadly for me, the work I’ve been building up over the past few years networking with those other groups has ground to a halt because of my inability to get data out of OpenStreetMap. Suggestions are welcome.

11 Comments

  1. Phil! said:

    On the dev list there was a discussion about XAPI recently. One person pointed to OSM3S ( http://wiki.openstreetmap.org/wiki/OSM3S ) as a XAPI alternative that seems pretty easy to set up and run yourself, especially as compared to XAPI or the rail database.

    9th March 2011
    • Tom Chance said:

      Looks interesting, thanks.

      10th March 2011
  2. Chris Hill said:

    Hove you looked at the regional downloads from Geofabrik? They create national or, in the UK, some county updates each night. If you need an extra county or city they might well add it if you ask. The newer .pbf format they provide is much more compact to download too and runs through osmosis very quickly.

    9th March 2011
    • Indeed. I’ve been doing postcode analysis over the great_britain.osm.pbf file they provide and Osmium runs my full analysis over it in about 1 minute. The file itself is about 250MB so it’s easily manageable on any local machine.

      9th March 2011
    • Tom Chance said:

      I have, and Frederik kindly got in touch to discuss options. But what I need is something I can use on a cheap web host without access to PostgreSQL and without the ability to install Osmosis. I want it to be automated.

      For other people less technically capable than me (so 99.9% of the population) who might want to grab data, it’s no use.

      XAPI is by far the easiest way to get data out, it would be great if we could find a way to offer it as a reliable service.

      10th March 2011
      • Halcyon (P2’s rendering engine) will do that, to a certain extent. It can read tiled .osm files from your cheap-and-cheerful webhost and render them.

        The support’s a bit rudimentary and I need to spend some time on it; and, for ease of deployment, someone would need to provide tiled .osm files for download (so you don’t have to split country extracts yourself). But none of that is too difficult.

        10th March 2011
  3. booo said:

    Hi,

    we are working on a new and reliable implementation of the xapi. We will start a code sprint in the beginning of April. Request for nodes should work now. If you like to participate in coding visit https://github.com/slomo/osm-spline-xapi

    Perhaps we will submit an application for the google summer of code.

    There is currently also a java implementation on the way:

    https://github.com/iandees/xapi-servlet

    10th March 2011
    • Tom Chance said:

      Sounds good, I tried out the Java XAPI that Ian Dees is working on which is also very welcome!

      I look forward to watching your progress.

      Given how server intensive I imagine XAPI is, I wonder about limiting the service to users with keys rather than letting it simply melt down?

      10th March 2011
  4. booo said:

    Hopefully we don’t melt the server. We could try to implement an optional key feature. I will add this to the todo list 😉

    10th March 2011
  5. Separate to my xapi needs, I’ve set up a pgsql osm server(desktop PC) for playing around with mapnik. For me I could avoid the xapi need by having a script that pulls data I want out of the local pgsql database and throws it up to my live mysql database in the form I want.
    My problem is that I don’t know the osm2pgsql scheme, and I probably wouldn’t know what to write a script in.

    12th March 2011

Comments are closed.