Leakage-Free Evaluation of Capsule Network Heads for Brain Tumor MRI Classification

Maulana Kanigara; Guruh Fajar Shidik

doi:10.59261/jequi.v8i5.478

Authors

Maulana Kanigara Universitas Dian Nuswantoro
Guruh Fajar Shidik Universitas Dian Nuswantoro

DOI:

https://doi.org/10.59261/jequi.v8i5.478

Keywords:

Brain Tumor Classification, Capsule Network, Data Leakage, Magnetic Resonance Imaging, Reproducible Evaluation

Abstract

Background: Deep learning studies using the widely used Brain Tumor MRI Dataset routinely report accuracies above 95%. However, the dataset combines overlapping public collections, and capsule networks are often promoted for such data without controlled comparisons.

Objective: This study quantified duplicate-induced data leakage in the dataset and investigated whether capsule heads improve classification under leakage-free evaluation.

Methods: Exact and near-duplicate slices were identified using file hashing and correlation analysis of head-cropped thumbnails, with the correlation threshold calibrated against label-conflict rates. Ten models using three backbones were trained under a unified protocol with five random seeds on a consumer-grade GPU. Capsule, dense, and pooling heads were compared. The models were also evaluated using the original, potentially leaky split, and their rotation robustness and inference costs were assessed.

Results: Near-duplicates linked 1,707 of 5,712 training images (29.9%) to images in the test set, and data leakage inflated test accuracy by 3.22–6.33 percentage points through both memorization and the larger effective training set. With a lightweight backbone trained from scratch, the capsule head improved macro-F1 by 7.16 percentage points over the dense head while using 91.3% fewer parameters, and by 6.63 percentage points over the pooling head. These improvements were consistent across all five random seeds. However, the capsule head provided no improvement with ImageNet-pretrained backbones and did not consistently reduce the loss in accuracy under rotation.

Conclusion: Capsule heads appear to offer the greatest benefit when features are learned from limited data. Leakage-free evaluation across multiple random seeds is essential for credible brain tumor MRI benchmarks. These conclusions apply to the tested public dataset and experimental protocol and do not constitute clinical validation.

Downloads

Download data is not yet available.

References

Afshar, P., Mohammadi, A., & Plataniotis, K. N. (2018). Brain Tumor Type Classification via Capsule Networks. 2018 25th IEEE International Conference on Image Processing (ICIP), 3129–3133. https://doi.org/10.1109/ICIP.2018.8451379

Afshar, P., Plataniotis, K. N., & Mohammadi, A. (2019). Capsule Networks for Brain Tumor Classification Based on MRI Images and Coarse Tumor Boundaries. ICASSP 2019 - 2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 1368–1372. https://doi.org/10.1109/ICASSP.2019.8683759

Barz, B., & Denzler, J. (2020). Do We Train on Test Data? Purging CIFAR of Near-Duplicates. Journal of Imaging, 6(6), 41. https://doi.org/10.3390/jimaging6060041

Bhuvaji, S. (2020). Brain tumor classification (MRI)[Data set]. Kaggle.

Bouthillier, X. (2021). Accounting for variance in machine learning benchmarks . 747–769.

Cheng, J., Huang, W., Cao, S., Yang, R., Yang, W., Yun, Z., Wang, Z., & Feng, Q. (2015). Enhanced Performance of Brain Tumor Classification via Tumor Region Augmentation and Partition. PLOS ONE, 10(10), e0140381. https://doi.org/10.1371/journal.pone.0140381

Deepak, S., & Ameer, P. M. (2019). Brain tumor classification using deep CNN features via transfer learning. Computers in Biology and Medicine, 111, 103345. https://doi.org/10.1016/j.compbiomed.2019.103345

Deng, J., Dong, W., Socher, R., Li, L.-J., Kai Li, & Li Fei-Fei. (2009). ImageNet: A large-scale hierarchical image database. 2009 IEEE Conference on Computer Vision and Pattern Recognition, 248–255. https://doi.org/10.1109/CVPR.2009.5206848

Dietterich, T. G. (1998). Approximate Statistical Tests for Comparing Supervised Classification Learning Algorithms. Neural Computation, 10(7), 1895–1923. https://doi.org/10.1162/089976698300017197

Fateh, A., Rezvani, Y., Moayedi, S., Rezvani, S., Fateh, F., Fateh, M., & Abolghasemi, V. (2026). BRISC: Annotated Dataset for Brain Tumor Segmentation and Classification. Scientific Data, 13(1), 361. https://doi.org/10.1038/s41597-026-06753-y

Geirhos, R., Jacobsen, J.-H., Michaelis, C., Zemel, R., Brendel, W., Bethge, M., & Wichmann, F. A. (2020). Shortcut learning in deep neural networks. Nature Machine Intelligence, 2(11), 665–673. https://doi.org/10.1038/s42256-020-00257-z

Gu, J., Tresp, V., & Hu, H. (2021). Capsule Network is Not More Robust than Convolutional Network. 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 14304–14312. https://doi.org/10.1109/CVPR46437.2021.01408

He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 770–778. https://doi.org/10.1109/CVPR.2016.90

Holm, S. (1979). A simple sequentially rejective multiple test procedure. Scandinavian Journal of Statistics, 6(2).

Jabbar, A., Naseem, S., Mahmood, T., Saba, T., Alamri, F. S., & Rehman, A. (2023). Brain Tumor Detection and Multi-Grade Segmentation Through Hybrid Caps-VGGNet Model. IEEE Access, 11. https://doi.org/10.1109/ACCESS.2023.3289224

Kapoor, S., & Narayanan, A. (2023). Leakage and the reproducibility crisis in machine-learning-based science. Patterns, 4(9), 100804. https://doi.org/10.1016/j.patter.2023.100804

Kingma, D. P., & Ba, J. L. (2015). Adam: A method for stochastic optimization. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.

Litjens, G., Kooi, T., Bejnordi, B. E., Setio, A. A. A., Ciompi, F., Ghafoorian, M., van der Laak, J. A. W. M., van Ginneken, B., & Sánchez, C. I. (2017). A survey on deep learning in medical image analysis. Medical Image Analysis, 42, 60–88. https://doi.org/10.1016/j.media.2017.07.005

Louis, D. N., Perry, A., Wesseling, P., Brat, D. J., Cree, I. A., Figarella-Branger, D., Hawkins, C., Ng, H. K., Pfister, S. M., Reifenberger, G., Soffietti, R., von Deimling, A., & Ellison, D. W. (2021). The 2021 WHO Classification of Tumors of the Central Nervous System: a summary. Neuro-Oncology, 23(8), 1231–1251. https://doi.org/10.1093/neuonc/noab106

Lu, D., Zhang, Y., & Chaudhary, D. (2026). MTA-Swin: A Multi-Token Attention Swin Transformer for Brain Tumor Classification with Leakage-Free MRI Benchmarking. Journal of Medical Systems, 50(1), 100. https://doi.org/10.1007/s10916-026-02418-w

Ma, N., Zhang, X., Zheng, H. T., & Sun, J. (2018). Shufflenet V2: Practical guidelines for efficient cnn architecture design. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 11218 LNCS. https://doi.org/10.1007/978-3-030-01264-9_8

Nickparvar, M. (2021). Brain tumor MRI dataset. Kaggle.

Ostrom, Q. T., Cioffi, G., Waite, K., Kruchko, C., & Barnholtz-Sloan, J. S. (2021). CBTRUS Statistical Report: Primary Brain and Other Central Nervous System Tumors Diagnosed in the United States in 2014-2018. Neuro-Oncology, 23. https://doi.org/10.1093/neuonc/noab200

Raghu, M., Zhang, C., Kleinberg, J., & Bengio, S. (2019). Transfusion: Understanding transfer learning for medical imaging. Advances in Neural Information Processing Systems, 32.

Sabour, S., Frosst, N., & Hinton, G. E. (2017). Dynamic routing between capsules. Advances in Neural Information Processing Systems, 2017-December.

Schwartz, R., Dodge, J., Smith, N. A., & Etzioni, O. (2020). Green AI. Communications of the ACM, 63(12), 54–63. https://doi.org/10.1145/3381831

Simonyan, K., & Zisserman, A. (2015). Very deep convolutional networks for large-scale image recognition. 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings.

Swati, Z. N. K., Zhao, Q., Kabir, M., Ali, F., Ali, Z., Ahmed, S., & Lu, J. (2019). Brain tumor classification for MR images using transfer learning and fine-tuning. Computerized Medical Imaging and Graphics, 75, 34–46. https://doi.org/10.1016/j.compmedimag.2019.05.001

Tan, M., & Le, Q. V. (2019). EfficientNet: Rethinking model scaling for convolutional neural networks. 36th International Conference on Machine Learning, ICML 2019, 2019-June.

Varoquaux, G., & Cheplygina, V. (2022). Machine learning for medical imaging: methodological failures and recommendations for the future. Npj Digital Medicine, 5(1), 48. https://doi.org/10.1038/s41746-022-00592-y

Yagis, E., Atnafu, S. W., García Seco de Herrera, A., Marzi, C., Scheda, R., Giannelli, M., Tessa, C., Citi, L., & Diciotti, S. (2021). Effect of data leakage in brain MRI classification using 2D convolutional neural networks. Scientific Reports, 11(1), 22544. https://doi.org/10.1038/s41598-021-01681-w

Yunus, R. E. (2024). Radiology Loading and Coverage Hours in Indonesia. Korean Journal of Radiology, 25(7), 597. https://doi.org/10.3348/kjr.2024.0267