The code below allows anyone interested to score their own tractogram.
Preparing your tractogram
** AN IMPORTANT NOTE ON FILE FORMAT: Data format managing was not very well defined in 2015. A lot of effort was made to ensure that data would be readable by any software. Yet, things have evolved. As of 08.2022, in both versions of the scoring (Recobundles version through the initial standalone tool, ROIs version using scilpy scripts), tractograms are loaded through Dipy’s Stateful Tractogram functions. This is particularly important if you already used the standalone tool before. Previous version of the code applied authomatic 0.5 shifts when loading files as trk. In the new python3 version, this is NOT done anymore. Please be careful: verify that final segmented bundles are well aligned with your initial tractogram, showing that space attributes were correctly interpreted.
Ground truth data + Code
| 2023 version: ROI-based segmentation
| 2015 version: Recobundles-based segmentation
History: why we made two versions
The original goal imagined by the leading team of the Challenge was to evaluate all submissions using ROI segmentation. However, during the initial evaluation phase, it was decided that the manual creation of acceptable ROIs (even considering the large variability amongst submissions), would be very time-consuming. The team had a close deadline to be able to present their result at the Diffusion Study Group Worshop, and the choice was made to use Recobundles, a novel bundle segmentation tool at the time. That choice offered a quick way to obtain results good enough to lead to a correct analysis of submissions, providing insightful conclusions of the challenge. Results of the scoring of bundles segmented with Recobundles are published in Maier-Hein 2017.
Years later, it became clear that the ISMRM 2015 Challenge was still very much used by the tractography community. A careful examination of the Recobundles results on the initial submissions revealed that some bundles were recurrently poorly segmented. Recobundles has limitations: it depends on thresholds and on the order of the bundles during scoring. We decided to tackle the difficult task of manually creating ROIs that would allow a more precise and stable segmentation of the bundles. With this second technique, scores are similiar overall, but some bundles show major changes in scoring (see the Renauld 2023). This new segmentation choice is more stable and should now be preferred.