A Leakage-Aware Evaluation Framework for Class Imbalance Handling in CNN-BiLSTM Network Intrusion Detection on UNSW-NB15

Wahyu Riza Pangestu; Aditya Kurniawan

doi:10.59261/jequi.v8i4.434

Authors

Wahyu Riza Pangestu Universitas Bina Nusantara
Aditya Kurniawan Universitas Bina Nusantara

DOI:

https://doi.org/10.59261/jequi.v8i4.434

Keywords:

Network Intrusion Detection, Class Imbalance, Data Leakage, Evaluation Protocol, UNSW-NB15, CNN-BiLSTM, SMOTE, ADASYN, CTGAN, Leakage-Aware Evaluation

Abstract

Background: Reported performance of deep learning-based network intrusion detection systems can be sensitive to evaluation design, particularly whether preprocessing statistics are estimated using only training data or information that also includes evaluation data.

Objective: This study examines a leakage-aware CNN-BiLSTM evaluation framework on UNSW-NB15 for binary and multiclass intrusion detection. It compares a pooled-fit configuration with an official train-only-fit configuration while explicitly recognizing that the protocols also use different test sets.

Methods: The study used the UNSW-NB15 training and testing files, a common CNN-BiLSTM architecture, repeated random seeds, and four training-only imbalance treatments: class weighting, SMOTE, ADASYN, and CTGAN-based oversampling. Performance was assessed with binary and multiclass metrics, and imbalance methods were compared with the baseline using paired tests across matched seeds.

Results: The pooled-fit and official configurations produced binary F1 scores of 96.30% and 88.95%, respectively, and multiclass macro F1 scores of 54.87% and 44.76%. These are differences between protocols rather than estimates attributable solely to preprocessing fit-scope. Under the official configuration, ADASYN increased binary F1 from 88.95% to 90.67% and reduced the false positive rate to 19.27% (p<0.001). No imbalance method improved multiclass weighted F1; class weighting descriptively increased balanced accuracy from 48.65% to 58.69%.

Conclusion: Evaluation design materially affects reported CNN-BiLSTM performance on UNSW-NB15. Under the conservative official configuration, imbalance handling provided modest binary gains, while multiclass results showed a trade-off between weighted performance and minority-class sensitivity.

Downloads

Download data is not yet available.

References

Abdelkhalek, A., & Mashaly, M. (2023). Addressing the class imbalance problem in network intrusion detection systems using data resampling and deep learning. The Journal of Supercomputing, 79(10), 10611–10644. https://doi.org/10.1007/s11227-023-05073-x

Bouke, M. A., & Abdullah, A. (2023). An empirical study of pattern leakage impact during data preprocessing on machine learning-based intrusion detection models reliability. Expert Systems with Applications, 230. https://doi.org/10.1016/j.eswa.2023.120715

Cao, B., Li, C., Song, Y., Qin, Y., & Chen, C. (2022). Network intrusion detection model based on CNN and GRU. Applied Sciences, 12(9). https://doi.org/10.3390/app12094184

Carrasco, J., García, S., Rueda, M. M., Das, S., & Herrera, F. (2020). Recent trends in the use of statistical tests for comparing swarm and evolutionary computing algorithms: Practical guidelines and a critical review. Swarm and Evolutionary Computation, 54. https://doi.org/10.1016/j.swevo.2020.100665

Fu, X., Wang, J., & Su, H. (2022). Hybrid CNN-BiLSTM model with attention mechanism for imbalance network intrusion detection. IEEE Transactions on Network and Service Management, 19(3), 2515–2527. https://scholar.google.com/scholar?q=%22Hybrid%20CNN-BiLSTM%20model%20with%20attention%20mechanism%20for%20imbalance%20network%20intrusion%20detection%22

Grinsztajn, L., Oyallon, E., & Varoquaux, G. (2022). Why do tree-based models still outperform deep learning on typical tabular data? Advances in Neural Information Processing Systems, 35. https://doi.org/10.52202/068431-0037

Jihado, M., & Girsang, A. S. (2024). CNN-BiLSTM hybrid model for network intrusion detection on CICIDS2017 and UNSW-NB15 datasets. Journal of Advances in Information Technology, 15, 219–232. https://scholar.google.com/scholar?q=%22CNN-BiLSTM%20hybrid%20model%20for%20network%20intrusion%20detection%20on%20CICIDS2017%20and%20UNSW-NB15%20datasets%22

Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., & Liu, T.-Y. (2017). LightGBM: A highly efficient gradient boosting decision tree. Advances in Neural Information Processing Systems, 30, 3146–3154. https://scholar.google.com/scholar?q=%22LightGBM%3A%20A%20highly%20efficient%20gradient%20boosting%20decision%20tree%22

Khraisat, A., Gondal, I., Vamplew, P., & Kamruzzaman, J. (2019). Survey of intrusion detection systems: Techniques, datasets and challenges. Cybersecurity, 2. https://doi.org/10.1186/s42400-019-0038-7

Luque, A., Carrasco, A., Martín, A., & de las Heras, A. (2019). The impact of class imbalance in classification performance metrics based on the binary confusion matrix. Pattern Recognition, 91, 216–231. https://doi.org/10.1016/j.patcog.2019.02.023

Maoudj, S. E., & Belghiat, A. (2025). A deep learning-based approach with two-step minority classes prediction for intrusion detection in Internet of Things networks. Knowledge-Based Systems, 312. https://doi.org/10.1016/j.knosys.2025.113143

Mohammadpour, L., Ling, T. C., Liew, C. S., & Aryanfar, A. (2022). A survey of CNN-based network intrusion detection. Applied Sciences, 12(16). https://scholar.google.com/scholar?q=%22A%20survey%20of%20CNN-based%20network%20intrusion%20detection%22

More, S., Idrissi, M., Mahmoud, H., & Asyhari, A. T. (2024). Enhanced intrusion detection systems performance with UNSW-NB15 data analysis. Algorithms, 17(2). https://doi.org/10.3390/a17020064

Rahman, M. M., Shakil, S. Al, & Mustakim, M. R. (2025). A survey on intrusion detection system in IoT networks. In Cyber Security and Applications (Vol. 3). https://doi.org/10.1016/j.csa.2024.100082

Ring, M., Wunderlich, S., Scheuring, D., Landes, D., & Hotho, A. (2019). A survey of network-based intrusion detection data sets. Computers & Security, 86, 147–167. https://doi.org/10.1016/j.cose.2019.06.005

Sathyanarayanan, S. (2024). Confusion Matrix-Based Performance Evaluation Metrics. African Journal of Biomedical Research. https://doi.org/10.53555/ajbr.v27i4s.4345

Ur Rehman, J., Ulum, M. S., Shaffar, A. W., Hakim, A. A., Mujirin, Abdullah, Z., Al-Hraishawi, H., Chatzinotas, S., & Shin, H. (2025). Evolutionary Algorithms and Quantum Computing: Recent Advances, Opportunities, and Challenges. IEEE Access, 13. https://doi.org/10.1109/ACCESS.2025.3530952

Verizon. (2025). 2025 Data Breach Investigations Report. Verizon Business. https://www.verizon.com/business/resources/reports/dbir/

Xu, L., Skoularidou, M., Cuesta-Infante, A., & Veeramachaneni, K. (2019). Modeling tabular data using conditional GAN. Advances in Neural Information Processing Systems, 32. https://scholar.google.com/scholar?q=%22Modeling%20tabular%20data%20using%20conditional%20GAN%22

Zhang, Z., Hao, J., Chen, L., Hou, T., & Liu, Y. (2026). A dual-attention CNN-BiLSTM model for network intrusion detection. Computers, Materials & Continua, 86(1), 1–22. https://doi.org/10.32604/cmc.2025.068372

Zhao, Z., Kunar, A., Birke, R., Van der Scheer, H., & Chen, L. Y. (2023). CTAB-GAN+: enhancing tabular data synthesis. Frontiers in Big Data, 6. https://doi.org/10.3389/fdata.2023.1296508

Zhou, W., Xia, C., Wang, T., Liang, X., Lin, W., Li, X., & Zhang, S. (2025). HIDIM: A novel framework of network intrusion detection for hierarchical dependency and class imbalance. Computers & Security, 148. https://doi.org/10.1016/j.cose.2024.104155