As you may or may not have noticed, hbus.ca has been down for the past few weeks. Halifax updated the data but I didn’t have a chance to update it. Well, I finally did, and hbus is now up in its former glory (minus a small issue with stops named ‘6016_merged_3300509′: thanks Metro Transit, time to update that script to massage your data again!) . I still wonder a bit about why I keep putting time into a site oriented around a city I haven’t lived in for over 2 years now (especially when Google Maps arguably does a better job at most things than I do), but there you go!
NIXI is still up and being useful to me, though I’m a little bit disappointed by its uptake from the rest of the world (the site gets like 5–12 unique views a day). What do I need to do for you guys? A mobile version? French localization? I just added support for Washington (Capital Bikeshare) and Boston (Hubway), maybe it’ll get some uptake there.
Not really my project, but Stephane Guidoin put up an awesome site called zonecone.ca which helps you find out about traffic obstacles that might delay your journey. It even has a nifty feature where you can create an account, specify a route you take regularly (say, your commute), and it’ll automatically notify you if something pops up. As you may have guessed, I’m not the biggest fan of automobiles, but this is still very cool. 😉 This site was originally based on the map layout template that I announced a few months ago.
I have a lettuce plant growing on my new balcony. It’s doing ok, though it will probably have to be brought in soon due to frost. Will it get enough sun? Can I prevent my cat from eating it? Will it make a delicious salad? Stay tuned!
Just a quick note to say that I just opensourced the software behind hbus.ca, nicknamed “Routez” under the Affero GPL. You can get the code on github.
For those new to the project, hbus.ca is a generic trip planning / transit information site for Halifax, Nova Scotia written using the Django web framework. It currently has two main features:
A trip planning front-end much like Google Transit (built from the ground up using the libroutez library).
A “nearby” routes feature which gives you all the bus departures near a particular location.
On the backend, both of these features are accessible via JSON APIs, for use in transit apps, etc. Transit to Go uses these to great effect.
There is nothing particularly Halifax-specific to the underlying Routez software, aside from various references in the web front end to Halifax and hbus.ca. In fact, we use Routez to provide information for Transit to Go Edmonton right now, with no modification.
Originally my plan was to release something that was completely generic out of the box so that anyone could trivially make up a version of this site for their favourite city. I’ve made some headway towards that goal over the last week or so, but there’s still some ways to go. There’s basically two major issues:
The geocoder depends on information gleaned from the geobase road network dataset. The intent behind this is noble (provide an end-to-end solution that doesn’t depend on third parties) but in practice this limits the software’s usefulness. It would be better to optionally allow a Routez-based site to use Google’s geocoder on the front-end. Unfortunately, to comply with Google’s terms of service, we’d also need to use their Maps API for the base map as well. Perhaps the best option here would be to use something like Mapstraction to allow users to select their preferred mapping provider.
The trip planning software used in the backend, libroutez, is getting a bit long in the tooth and is quite finicky about what kind of data it will accept. I think the long-term solution to this is to switch to Graphserver (which is more mature and better supported), but some features would have to be added to it to support the kind of things that Routez needs (like a list of upcoming departures at a particular transit stop).
Even with these problems, I figured it would be better to open up what I have for people to check out and play with. Have a look and let me know what you think!
Due to some schedule adjustments, it appears as if Metro Transit’s Google Transit is temporarily out of service while they wait for Google to process their new transit feed.
However, fear not! Unlike Google, obsessive compulsive computer programmer William Lachance works on weekends for free (or the low, low price of $1.99 in the case of iPhone applications) so your thirst for updated transit information can be satisfied. I link to these things often enough on this blog, but here they are again, for the benefit of first time viewers:
Transit to Go In other news, it appears as if Metro Transit has finally seen the light and made their transit information available to anyone who wants it. I would like to think that I might have had something to do with this, but I honestly have no idea. Anyway, if you want to check it out, it’s available here.
After a month long hiatus, I finally had a chance to get hbus.ca back into a working state. The big difference is that we’re now using the official HRM transit data. Lots more remains to be done to make this site competitive for 2010 (most of the recent work has been back-end infrastructure stuff), but at least it’s usable again.
A productive day on the transit development front. Finished up a few big features related to hbus.ca and Transit to Go:
Sped up the graph and database generation by an order of magnitude. Not too exciting from a user perspective, but I should now be able to iterate the codebase much faster.
Better transit stop / street graph linking: No more does libroutez simply try to find the closest street level vertice to link to when merging transit stops with street information: we now actually create _new_ street level vertices as needed and link to those. Upshot? Slightly better directions and prettier polylines. When I first thought up the algorithm a month ago, I thought I was totally brilliant, only to later find out that Andrew Byrd had done something almost identical a few months earlier for graphserver. Ah well, at least it’s implemented.
I coded up a script to automatically generate synthetic headsigns for GTFS feeds which don’t have them. This was needed to provide a sensible view for the Edmonton version of Transit to Go. All the props in the world for opening up your data guys, but can’t you do better than saying that all your buses go in the “1” direction? There’s a reason why it’s a required field you know. Not only would it help me, but Google Transit would give better results for your city as well.
One new years resolution I have set for myself is to blog more about what I’m working on. I’ve learned over the last year that the audience of people who care about your projects in development is vanishingly small. Thus, the need for secrecy in order to make a “PR splash” is rather small: announce far and wide when you have something that people can use by all means, but don’t worry too much about talking about what you’re working on with the internets.
In this spirit, some projects I’m 99% certain I’ll be releasing publicly in 2010:
neocoder A lightweight geocoding library, with wrapper libraries for your language of choice. Written in C++ using SQLite and boost regular expressions. Will support both OpenStreetMap and GeoBase GML as input. Currently in development on github, hoping to release with routez (as its geocoding component).
routez A generic travel planning web service, written in python using the django framework and the libroutez libraries. This is basically the software behind hbus.ca the goal for 2010 is to clean it up and make it generic by clearing out the Halifax-specific stuff (mostly just the geocoding and site theme stuff at this point), then release it to the public under the Affero GPL License (was originally going to with GPL, but Simon Law convinced me otherwise: more on that in a future post).
Transit To Go A dedicated iPhone client for the routez software, developed in collaboration with Dmitri Dolguikh and Bill Wilson, two talent developers from Halifax. Has some innovative (in my opinion, anyway) details on how things will be viewed. This one’s going to be proprietary, but will be affordable and awesome.
Besides this, I have a few more irons in the fire; however, I’m hesistant to talk about them just yet. Just getting the above done in the midst of my work with Navarra (to say nothing of having a life in there somewhere) is going to be challenging.
Thoughts? Would be particularly interested in hearing from people working on similar projects to neocoder and routez. Despite how it may some times appear, I don’t have a NotInventedHere mentality: I’ve done as an exhaustive survey of the field as I could before deciding to work on my own projects, and what I’ve found just hasn’t been the right fit for what I’m trying to accomplish. However, the world’s gotten so damn big that I’m not sure if I’m missing something.
People frequently ask me how I manage to collect and input the data that is used by hbus.ca, given Metro Transit’s intransigence. The “bike and GPS” angle is well known by now, but what about the rest of the process? How do I get the data into a format that hbus.ca can consume?
The defacto standard for the interchange of transit information is Google Transit Feed (GTFS). This exceedingly simple comma seperated value format is now supported by a plethora of software, including Google Transit, graphserver, as well as my very own libroutez (used by hbus.ca). It was obvious to me right from the beginning that the first step to building hbus.ca would be to create one of these feeds.
Manipulating a GTFS by hand is probably not a great idea. It’s basically a dump of a relational database, and is pretty inscrutable from the point of view of a human being. What I really want to be able to do is be able
to manipulate things on the level of stops, service periods, and routes: and let some kind of abstraction layer take care of the low-level details. Fortunately, the awesome engineers at google created a python library called Google Transit Data Feed, which can help with creating one of these things by providing abstractions of the key elements of a google transit feed (stops, service periods, etc.). You can then write a program which uses these abstractions to create and save a GTFS.
Of course, providing the library appropriate information is easier said than done. Metro Transit’s PDF schedules are not readily computer parsable (being designed to be printed out, after all). I needed some kind of semi-automated way of converting a Metro Transit schedule into GTFS, or this whole project was
going nowhere fast.
As an initial step, it turns out that it’s quite possible to extract textual information from a PDF using the open source popplar library. From there, it’s possible to extract the stopping times for an individual bus route. Let’s give an example. For example, let’s take the case of adding the 60 (Portland Hill’s route), something I’m currently working on. All I had to do was download the PDF file from Metro Transit’s site and then run the following on the command line:
<br /> pdftotext -raw route60.pdf<br /> The raw option basically makes sure the raw strings are dumped to disk, and that no attempt is made to preserve formatting. The result is a text file with content like this in it:
<br /> 842a 847a 855a 858a 903a 906a 912a -<br /> 857a 902a 910a 913a 918a 921a - 925a<br /> 910a 915a 923a 926a 931a 934a 940a -<br /> 940a 945a 953a - 1000a 1003a 1009a -<br /> ...and every 30 minutes until<br /> 210p 215p 223p - 230p 233p 239p -<br /> This type of format can be parsed easily enough. To create a proper transit feed though, schedule information isn’t enough: you also need to know the locations of the stops, names of routes, etc. After some deliberation, I came to the determination that I needed some kind of intermediate format to store the above schedule information and this additional information. It would be readable both by humans (to ease its creation) and machines.
The obvious markup for something like this is YAML (if you’re still using XML to store structured information, run, don’t walk, and look at YAML: you can thank me later). Simple, clean, effective. GTFS is still the better choice for using the information in another application as its representation is much more amenable to being stored in a graph. Here’s a few examples of my YAML format in action:
Besides the scheduling information, the other main interesting component of a GTFS is the location of the stops. As anyone who’s used a Metro Transit schedule has noticed, only major timepoints are covered in the PDF schedules. What of all the stops in between? This is where the bike and GPS come in.
What I did was take a standard GPS from Mountain Equipment Co-op (The Garmin GPSMap 60x), get on my bike, take the readings of individual gotime numbers and positioning information, of the individual stops between the major timepoints. I then took this device back to my computer and, using a utility called GPSBabel, dumped out the stop information in a format called “comma seperated value”. It looks like this:
<br /> 44.65825, -63.59252, 6785-21-31-33-34-35-3-7<br /> 44.65982, -63.59452, 6768-21-31-33-35-86-3-7<br /> 44.66113, -63.59659, 6782-21-31-33-34-35-3-7<br /> The first two items are latitude and longitude, providing the positioning of the stop. The last item is a gotime number, followed by the set of buses which pass by the stop. Turning this into YAML is a matter of applying
the following regular expression to the input:
<br /> \([0-9]+.[0-9]+\), \(-63.[0-9]+\), \([0-9]+\)- -> - { name: xxx, stop_code: \3, lat: \1, lng: \2 }<br /> To get an actual name for the stop (i.e.: “Gottingen and Young”), I wrote a simple script which finds the nearest intersection close to the stop in the GeoBase dataset. I then (at my discretion) corrected it based on my on-the-street knowledge of the layout of Halifax as well as adding certain details to help the user (e.g. bus stops on the way to the south end of Halifax are marked “south bound”).
With these two elements in place (a format for creating human-readable transit information and a library for creating GTFS), the only thing left to do is create a program which bridges the gap. Behold, the magic of
createfeed.py. With all of this in place, creating a google transit feed for Halifax is a simple matter of typing “make”.
Is this a ridiculous amount of work? I wouldn’t say so. The vast, vast majority of my work on hbus.ca has been in creating the pathfinding code and geocoding functionality. This is work that can be translated to many different municipalities, and can easily be extended and made more useful in a myriad of ways.
What does seem a little intimidating to me is completing what I started. Capturing bus stop information for the Halifax peninsula is one thing, but covering the outlying areas (Bayer’s Lake, Sackville, etc.) is quite
another. There’s a lot of biking involved there, more perhaps than what one person can reasonably be expected to do. It was my hope that the initial release of hbus would validate the model of community-developed transit software to Metro Transit and they would see the benefit of releasing their internal copy of this data to the public, but unfortunately that doesn’t seem to have happened.
Getting that problem solved seems to be more a political problem than a technical one, and it’s not my specialty. It really does make me wonder if I shouldn’t reconsider the option of crowd sourcing, which I had
rejected earlier.
So I opened up my baby, hbus.ca, to the public last week (though traffic only really started to pick up yesterday, after a positive article in the daily news). This site, a trip planner for the Halifax Regional Municipality, was the cumulation of about 6 months of part time work on my part, between contracts for my awesome company, Navarra.
I’m debating on whether or not to start a seperate blog for hbus. At the moment I’m leaning towards no: my thinking is that most people don’t care about the inner workings of a site like hbus. They just want to figure out how to get from point A to B. Those who do care can read the rest of what I (and my part time co-conspirator, Peter McCurdy) have to say.
The most glaring limitation in hbus right now is that its route coverage is woefully limited. Trips on the main Halifax peninsula are generally planned pretty effectively. If you’re travelling to a suburban area like Bayer’s Lake or Burnside, not so much (unless you’re lucky enough to be starting/ending near a bus timepoint). What is to be done?
A frequent suggestion I get from more technically minded folks is that I should “crowd source” the missing information. This basically implies creating a wikipedia-like architecture such that people could contribute their favourite stops, routes, etc.
It’s a tempting idea. Such sites as OpenStreetMap show that this approach can be very effective for gathering large amount of geographical data. I’m frankly not convinced it’s the right approach here though. The fact is that Metro Transit MUST have a complete set of stops, route schedules, and route plans internally. There’s no way they could plan their operations halfway effectively otherwise. Why should I burden the public with the task of recreating something which has already been done?
I may be crazy, but I think the best avenue for the moment is to try to convince Metro Transit that it would be worthwhile to make this information public. I paid for the generation of the information with my tax dollars, why shouldn’t I be able to make use of it? The preferred format for this information is Google Transit Data Feed, but I could make use of information in just about any representation (Arc GIS, etc.). Just give me what you have, and I’ll take of the rest. Over 20 of the most successful transit agencies in North America (many of them much bigger than Metro Transit) have opened up their information to the public, with only positive results.
The most obvious use of this information is a trip planner and, yes, I know every agency and their dog has (or will have) one of these. But maybe someone has a cool idea on how to make a trip planner easier to use (compare hbus.ca with Tous Azimuts). Or what about transit maps that help people figure out where to live? Or iPhone and Blackberry applications? Or cool screensavers? Or or or. The possibilities are truly endless once the data is out there. Come on Metro Transit, you have nothing to lose and the eternal love of your ridership to gain.
