Data pipeline for GTFS transit arrival and departure information

Abstract

Cities generate large volumes of data daily through digital services and smart city applications, these include Public Transport Authorities which generate big data as part of their daily operations, such as vehicle positions, counts of passengers and user travel patterns. The General Transit Feed Specification (GTFS) is a data format that allows public transport data to be consumed by a wide variety of software applications. This paper presents a data pipeline developed to manipulate the GTFS feeds into a general and flexible dataset of realtime transit arrivals. There are three barriers to widespread access to the information addressed by creating a one-size-fits-all data pipeline for realtime operations from GTFS. First, the protocol buffer format is not human readable and requires processing before use in most transport applications. Secondly, the general specification does vary place-to-place and the conditionally required and optional fields are inconsistent between locations. Thirdly, the raw data may contain errors including missing stop sequence or a reverse direction bus being detected in the bus stop area. The pipeline is constructed of set of data cleaning and transformation steps to address these challenges. The paper briefly presents a potential use cases of the processed data to illustrate its relevance to researchers and practitioners.