Expand partial APC messages with service journey metadata and combine them by stops to create full APC messages.

Expand all of the partial APC messages from vehicles with service journey metadata from HFP messages. Use vehicle IDs to map between the partial APC messages and the HFP messages. Combine the expanded messages by stops so that the combined messages look like full APC messages.

This project depends indirectly on transitdata-common project for the Protobuf proto definition files though the files have been slightly modified.

This service hides fairly complex details in it. It has to deal with streaming merging requirements, acknowledge read messages at the right time and deal with bugs and oddities in the HFP data and human driver behavior.

Some details:

• In general, if several partial APC messages are received per stop, their passenger counts are aggregated.

• We merge messages based on the server-side ingest timestamps instead of the reported time in the vehicles as experience has shown that the clocks in the vehicles cannot be trusted.

• The metro is not currently supported by this service as the current logic for detecting departures from stops depends on HFP event types PDE and DEP that are not currently available for the metro.

• There are some current limitations in the Node.js client for Apache Pulsar that are either bypassed or tolerated. In general, these limitations are commented in the code.

The following situations are considered in the code:

1. The vehicle departs from a stop in the middle of a service journey.

   This is the most basic case. In this and other cases we allow for partial APC messages to arrive a bit later than the message informing about the departure from the stop. That way network delays produce fewer errors of matching passenger counts to wrong stops. However, it's a delicate balance between the latency of the results, acceptance of network delays for the source data and avoiding overlapping time windows for consecutive stops.

2. The passengers alight from the previous service journey and board the next service journey at roughly the same time.

   This might happen, for example, if the final stop of the previous service journey and the first stop of the next service journey are the same. In this case we match alighting passengers to the previous service journey and the boarding passengers to the next service journey.

3. The driver signs from a dead run onto the service journey only after already allowing passengers to board.

   In this case we accept passenger counts from just before the start of the service journey but throw away old counts that might for example originate from the cleaner crew in the depot.

4. The driver signs out of the service journey and starts a dead run before letting all passengers out.

   In this case we wait for some time when a dead run starts before sending APC results for the last stop.

5. The driver initially signs onto the wrong service journey from a dead run.

   In this case we wait until the first departure from a stop on the new service journey before trusting the service journey choice.

6. The vehicle unexpectedly starts a dead run in the middle of a service journey and returns to the same service journey shortly after.

   In this case we try to figure out if we missed stops while the dead run was active.

7. The HFP messages for some vehicles do not ever get interpreted as departures from a stop.

   Possibly the messages are from broken HFP implementations or from test vehicles. In this case we forcibly acknowledge the read HFP messages when they become too old.

8. Some vehicles get partial APC messages but never HFP messages.

   Possibly the partial APC messages are from forgotten test systems. In this case we forcibly acknowledge the read partial APC messages when they become too old.

1. Install the build dependencies for the Apache Pulsar C++ client.

2. Create a suitable .env file for configuration. Check below for the configuration reference.

3. Install dependencies:

   npm install

4. Run linters and tests and build:

   npm run check-and-build

5. Load the environment variables:

   set -a
   source .env
   set +a

6. Run the application:

   npm start

There is a Docker Compose setup for running the partial APC pipeline locally. First cd ./docker-compose and then run either:

./run-everything-in-docker-compose.sh

to run every service inside Docker Compose, including this service, or run:

./run-everything-but-this-service-in-docker-compose.sh

to run every other piece of the pipeline except this service in Docker Compose. This latter method probably enables you to develop and debug this service easier.

There are unit tests, property-based tests and integration tests. npm run check-and-build runs all tests.

The integration tests are based on anonymized MQTT data dumps and expected results. Each integration test case has its own subdirectory under ./tests/testData/. The data is collected with ./scripts/collect-mqtt-data.template.sh.

As there is some delay-based business logic in use, integration tests cannot practically be run so that the code being tested would run in Docker while the test logic would run outside of Docker. Otherwise it would take e.g. 15 minutes for one integration test to finish while most of the execution time is spent waiting a timer to go off. Instead we mock the timers in the integration tests and use Testcontainers for the services we depend on.

You can use the Docker image hsldevcom/transitdata-partial-apc-expander-combiner:edge. Check out the available tags.