About OSPA/GOSPA

[Po-Jen]

Might not be directly related to this repo, but I think the users here are more likely to be able to answer this question. Please let me know if it's not adequate to post it here.

For MOT benchmarks, CLEAR MOT/HOTA metrics are prevalent, so I am wondering why OSPA/GOSPA are not included (not even discussed in many papers such as HOTA paper). Also, along the direction of OSPA/GOSPA metrics, discussions about CLEAR MOT/HOTA seems rare and unimportant (See Performance Evaluation Metrics and Approaches for Target Tracking: A Survey).

Is this because OSPA/GOSPA mixes localization error with detection/association error? Or is there any other reason I might be missing?

For those not familiar with OSPA/GOSPA, here're some reference:
Stonesoup OSPA Metric
Generalized optimal sub-pattern assignment metric

Thanks in advance 🙏

[cheind]

@Po-Jen py-motmetrics, despite its name, is not limited to a particular set of metrics per se. py-motmetrics is a community driven FOSS project, hence it requires two ingredients for metrics to be added: a) someone how is willing to implement it via a PR b) someone how is committed to maintain it and answer community questions related to the particular set of metrics.

I can't judge whether the OSPA metrics are useful, but given that they haven't been requested yet, I can assume that either there is already a working software solution that people are happy with, or that there is a lack of interest in this metric.

In case you want to hop in, please feel free to start a PR that integrates the OSPA metric.

[Po-Jen]

Gotcha, thank you. Yes OSPA/GOSPA is available in stonesoup, that might be the reason why it's not requested here.

I'll probably leave this issue open for a while to see if there are more comments related to why OSPA/GOSPA are not used in most MOT benchmarks.