The ISMRM 2015 Tractography Challenge

Score your own tractogram (link to the tools)

The code below allows anyone interested to score their own tractogram.

History: why we made two versions

Together with the updated ground truth tractogram (see description in the phantom creation process page), we have also updated the bundle segmentation technique. Why?

The original goal imagined by the leading team of the Challenge was to evaluate all submissions using ROI segmentation. However, during the initial evaluation phase, it became evident that the ROIs would be very difficult to create in this dataset simulating real-world conditions. For example, the classical approach uses masks of the endpoints of ground truth bundles to determine if streamlines are valid connections. However, with that data, endpoints masks being only 1 voxel thick and did not suffice to recover streamlines in most cases. It was evident that the manual creation of ROIs that would work well, even considering the big variability amongst submission, would be very time-consuming. The team had a close deadline to be able to present their result at the Diffusion Study Group Worshop, and the choice that was made was to use Recobundles, a novel bundle segmentation tool at the time. That choice offered a quick way to obtain acceptable results, enough to lead to a good analysis of submissions, providing insightful conclusions of the challenge, published in the 2017 paper.

In 2022 however, it became clear that the ISMRM 2015 Challenge was still very much used by the tractography community. A careful examination of the Recobundles results on the initial submissions led our team to decide to tackle the difficult task of manual creating ROIs that would allow a more precise and refined segmentation of the bundles.


Preparing your tractogram

** AN IMPORTANT NOTE ON FILE FORMAT: Data format managing was not very well defined in 2015. A lot of effort was made to ensure that data would be readable by any software. Yet, things have evolved. As of 08.2022, the standalone tool has been updated. The code is now in python3, and tractograms are loaded through Dipy’s Stateful Tractogram tools. Previous version of the code applied authomatic 0.5 shifts when loading files as trk, this is NOT done anymore. We have removed the .tck and .trk versions. Please use the safer .fib version (VTK files). These files are easily loadable through Dipy / Scilpy, the two libraries we use. The .fib files are also the ones included in the whole scoring archive in the Tools page.

Please be careful: verify that final segmented bundles are well aligned with your initial tractogram, showing that space attributes were correctly interpre>

Ground truth data + Code

2022 version: To be used with ROI-based segmentation

  • Scripts are offered in scilpy. In can be used in a terminal with a command line such as:


           python scil_score_tractogram.py [more information to come]

  • Should be used with [this scoring_data]. It contains the bundles, the bundles masks and the configuration file to be used with the mentionned script.

2015 version: Initial version, to be used with Recobundles.

  • Link to the Github repository containing the code . When used to score the 2015 submissions, it generates the exact same results as in the paper. Compared to the initial version, the only changes concern processing speed. Any remaining issue was already present at the time. An example of command line is given in the README file.


           ./scripts/score_tractogram.py YOUR_TRACTOGRAM_FILE scoring_data/ results/

  • The version v1.0.1 released in 2015, in python2 and deprecated, is still available. DOI

  • Should be used with this scoring_data , which contains the bundles, the bundle masks and a configuration file with all parameters necessary for Recobundles.