The code below allows anyone interested to score their own tractogram.
Preparing your tractogram test
** AN IMPORTANT NOTE ON FILE FORMAT: Data format managing was not very well-defined in 2015. A lot of effort was made to ensure that data would be readable by any software. Yet, things have evolved. As of 08.2022, in both versions of the scoring (Recobundles version through the initial standalone tool, ROIs version using scilpy scripts), tractograms are loaded through Dipy’s Stateful Tractogram functions. This is particularly important if you already used the standalone tool before. Previous version of the code applied authomatic 0.5 shifts when loading files as trk. In the new python3 version, this is NOT done anymore. Please be careful: verify that final segmented bundles are well aligned with your initial tractogram, showing that space attributes were correctly interpreted.
Ground truth data + Code
1) 2023 version: ROI-based segmentation
-
Scoring data: here . It contains the bundles, the bundle masks and the configuration files to be used with the mentionned script. - These ground truth bundles were modified compared to the initial version in order to have bundles that allowed creation of ROIs. Modifications are detailed in the Renauld 2023. In short, looping and broken streamlines were discarded, and the CSF / FPT / POPT were merged as one bundle called Brainstem Projection System (BPS). This scoring technique thus offers scores for 21 bundles instead of 25.
-
Scripts: Here is the link to the bash script . The process is divided into the following steps: 1) Segmentation of the bundles and sub-bundles. 2) Merging back sub-bundles of the CC and ICP. 3) Scoring the final bundles. Results are then stored in the results.json file. The python scripts used are from scilpy. They run with version 1.5.0, which uses python 3.7 and can be installed by downloading the release here and using “
pip install .” from inside the folder. For help adapting these lines to later versions, please contact us or check here . Note that if you are a user of DWI_ML , this script is installed as part of the package.
2) 2015 version: Recobundles-based segmentation
-
Link to the GitHub repository containing the code . Code was updated to python3 as of 08.2022. The only changes concern processing speed, and tractogram format management, as described above. When used to score the 2015 submissions, it generates the exact same results as in the paper. Any remaining issue was already present at the time. The version v1.0.1 released in 2015, in python2 and deprecated, is still available.
-
An example of command line is given in the README file.
-
Scoring data: Should be used with this scoring_data , which contains the bundles, the bundle masks and a configuration file with all parameters necessary for Recobundles.
History: why we made two versions
The original goal imagined by the leading team of the Challenge was to evaluate all submissions using ROI segmentation. However, during the initial evaluation phase, it was decided that the manual creation of acceptable ROIs (even considering the large variability amongst submissions), would be very time-consuming. The team had a close deadline to be able to present their result at the Diffusion Study Group Worshop, and the choice was made to use Recobundles, a novel bundle segmentation tool at the time. That choice offered a quick way to obtain results good enough to lead to a correct analysis of submissions, providing insightful conclusions of the challenge. Results of the scoring of bundles segmented with Recobundles are published in Maier-Hein 2017.
Years later, it became clear that the ISMRM 2015 Challenge was still very much used by the tractography community. A careful examination of the Recobundles results on the initial submissions revealed that some bundles were recurrently poorly segmented. Recobundles has limitations: it depends on thresholds and on the order of the bundles during scoring. We decided to tackle the difficult task of manually creating ROIs that would allow a more precise and stable segmentation of the bundles. With this second technique, scores are similiar overall, but some bundles show major changes in scoring (see the Renauld 2023). This new segmentation choice is more stable and should now be preferred.