The ISMRM 2015 Tractography Challenge

The evaluation process

1. Segmentation of the bundles

See the Tractometer’s description of the bundle segmentation process. The results have originally been published using Recobundles, but an updated version using ROIs is in preparation.

  • Using Recobundles: The process has been published as a standalone Github version, which can be used with the ground truth data to run, in one single command line, both the segmentation using all the appropriate Recobundles parameters and the computation of the metrics. Results are saved as a json file. Open this file in your favorite text editor to visualize the results. This code has been updated to python 3 as of 2022.
  • Using ROIs: information will come.
History: why we made two versions

The process of segmenting bundles from a tractogram is not an easy task!

The original goal imagined by the leading team of the Challenge was to evaluate all submissions using ROI segmentation. However, during the initial evaluation phase, it became evident that the ROIs would be very difficult to create in this dataset simulating real-world conditions. For example, the classical approach uses masks of the endpoints of ground truth bundles to determine if streamlines are valid connections. However, with that data, endpoints masks being only 1 voxel thick and did not suffice to recover streamlines in most cases. It was evident that the manual creation of ROIs that would work well, even considering the big variability amongst submission, would be very time-consuming. The team had a close deadline to be able to present their result at the Diffusion Study Group Worshop, and the choice that was made was to use Recobundles, a novel bundle segmentation tool at the time. That choice offered a quick way to obtain acceptable results, enough to lead to a good analysis of submissions, providing insightful conclusions of the challenge, published in the 2017 paper.

In 2022 however, it became clear that the ISMRM 2015 Challenge was still very much used by the tractography community. A careful examination of the Recobundles results on the initial submissions led our team to decide to tackle the difficult task of manual creating ROIs that would allow a more precise and refined segmentation of the bundles.

2. Metrics

See the Tractometer’s description of the metrics.