Hybrid Threat Detection Using Wazuh, DBSCAN, Random Forest, and MISP Threat Intelligence

Miko Dewi Hatmanti; Rojali Rojali

doi:10.59261/jequi.v8i4.419

Authors

Miko Dewi Hatmanti Universitas Bina Nusantara
Rojali Rojali Universitas Bina Nusantara

DOI:

https://doi.org/10.59261/jequi.v8i4.419

Keywords:

DBSCAN, Cyber Threat Intelligence, MISP, Random Forest, SIEM, Wazuh

Abstract

Background: Cyber-attacks continue to grow in volume and sophistication, and small-to-medium organizations are increasingly exposed because of limited budgets and security personnel. Open-source Security Information and Event Management (SIEM) platforms such as Wazuh offer a low-cost alternative to commercial solutions, yet their reliance on static, rule-based correlation logic limits their ability to detect previously unseen or evolving threats and tends to generate a high rate of false positives.

Objective: This study evaluates a layered Wazuh pipeline in which DBSCAN supplies density-context features to a Random Forest classifier and MISP provides post-classification IOC enrichment within a controlled AIT-ADS laboratory setting.

Methods: The experimental evaluation is conducted using the AIT-ADS dataset, which consists of 423,499 alerts. A within-class blocked temporal split produced 338,799 train set and 84,700 test set. Four strictly causal temporal features were combined with ten row-local features; DBSCAN was fitted only on the train set (ε = 0.466428, min_samples = 5) and added four density-context features, yielding 18 classifier inputs before being cross-checked against MISP indicators of compromise.

Results: Experimental results show DBSCAN+Random Forest achieved macro-F1 = 0.2976, balanced accuracy = 0.3747, and MCC = 0.4940. Its binary matrix changed from TP = 84,266, TN = 63, FP = 241, FN = 130 to TP = 84,284, TN = 63, FP = 241, FN = 112 after MISP, a 13.85% FN reduction with no induced FP. XGBoost FN decreased from 72 to 54 (25.00%), but FP increased from 270 to 272.

Conclusion: The proposed hybrid pipeline demonstrates that machine-learning-based alert classification can reduce false positives in Wazuh while preserving high threat detection sensitivity. However, the results also show that performance remains uneven across minority attack classes, and the benefits of DBSCAN and MISP depend on data characteristics and IOC coverage. Overall, Random Forest provides a more suitable operational trade-off for reducing alert burden in the evaluated scenario.

Downloads

Download data is not yet available.

References

Akinrinade, O., & Du, C. (2025). Skin cancer detection using deep machine learning techniques. Intelligence-Based Medicine, 11. https://doi.org/10.1016/j.ibmed.2024.100191

Ali, M. A., & Al-Sharafi, S. A. H. (2025). Intrusion detection in IoT networks using machine learning and deep learning approaches for MitM attack mitigation. Discover Internet of Things, 5(1). https://doi.org/10.1007/s43926-025-00104-w

Ali, M. L., Thakur, K., Schmeelk, S., Debello, J., & Dragos, D. (2025). Deep Learning vs. Machine Learning for Intrusion Detection in Computer Networks: A Comparative Study. Applied Sciences (Switzerland), 15(4). https://doi.org/10.3390/app15041903

Alzahrani, I. Y., Lee, S., & Kim, K. (2024). Enhancing cyber-threat intelligence in the Arab world: Leveraging IoC and MISP integration. Electronics, 13(13). https://doi.org/10.3390/electronics13132526

Ariyanto, Y., Syaifudin, Y. W., Ratsanjani, M. H., Muladawila, A. R., Fatmawati, T., Saputra, P. Y., & Setiadi, C. (2025). Cyber Threat Detection and Automated Response Using Wazuh and Telegram API. MATRIK : Jurnal Manajemen, Teknik Informatika Dan Rekayasa Komputer, 25(1). https://doi.org/10.30812/matrik.v25i1.5610

Bin Sarhan, B., & Altwaijry, N. (2023). Insider Threat Detection Using Machine Learning Approach. Applied Sciences (Switzerland), 13(1). https://doi.org/10.3390/app13010259

Chamkar, S. A., Zaydi, M., Maleh, Y., & Gherabi, N. (2025). Improving threat detection in Wazuh using machine learning techniques. Journal of Cybersecurity and Privacy, 5(2), 1–25. https://doi.org/10.3390/jcp5020034

Chidukwani, A., Zander, S., & Koutsakis, P. (2024). Cybersecurity preparedness of small-to-medium businesses: A Western Australia study with broader implications. Computers & Security, 145, 104026. https://doi.org/10.1016/j.cose.2024.104026

Chicco, D., & Jurman, G. (2023). The Matthews correlation coefficient (MCC) should replace the ROC AUC as the standard metric for assessing binary classification. BioData Mining 2023 16:1, 16(1), 4-. https://doi.org/10.1186/S13040-023-00322-4

Dhanya, K. A., Vajipayajula, S., Srinivasan, K., Tibrewal, A., Kumar, T. S., & Kumar, T. G. (2023). Detection of network attacks using machine learning and deep learning models. Procedia Computer Science, 218, 57–66. https://doi.org/10.1016/j.procs.2022.12.401

Deng, D. (2020). DBSCAN clustering algorithm based on density. Proceedings of the 2020 7th International Forum on Electrical Engineering and Automation, 949–953. https://doi.org/10.1109/IFEEA51475.2020.00199

ENISA. (2025). ENISA, “Threat Landscape for Financial Sector,” European Union Agency for Cybersecurity. ENISA Threat Landscape: Finance Sector.

Ester, M., Kriegel, H. P., Sander, J., & Xu, X. (1996, August). A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd (Vol. 96, No. 34, pp. 226-231).

Hidayat, M. R. T., Widiyasono, N., & Gunawan, R. (2025). Optimasi Deteksi Malware Pada Siem Wazuh Melalui Integrasi Cyber Threat Intelligence Dengan Misp Dan Dfir-Iris. Jurnal Informatika Dan Teknik Elektro Terapan, 13(1). https://doi.org/10.23960/jitet.v13i1.5686

Jakub Ziolek, D. B. J. L. I. B. I. C. G. F. K. L. ‎Jussi O. (2025). ENISA Threat Landscape 2025 — Part 2. In European Union Agency for Cybersecurity, ENISA (Number October).

Lee, S., Mujammami, A. A. H., & Kim, K. (2025). Leveraging Social Networks for Cyber Threat Intelligence: Analyzing Attack Trends and TTPs in the Arab World. IEEE Access, 13. https://doi.org/10.1109/ACCESS.2024.3508025

Lella, I. (2024). ENISA threat landscape 2024. European Union Agency for Cybersecurity.

Lones, M. A. (2024). Avoiding common machine learning pitfalls. Patterns, 5(10). https://doi.org/10.1016/j.patter.2024.101046

Luay, M., Layeghy, S., Noorbin, N., Sarhan, M., Kulatilleke, G., Moustafa, N., & Portmann, M. (2026). Time Matters: Temporal NetFlow Features for ML-Based Network Intrusion Detection. IEEE Access, 14, 66899–66913. https://doi.org/10.1109/ACCESS.2026.3688204

Manzoor, J., Waleed, A., Jamali, A. F., & Masood, A. (2024). Cybersecurity on a budget: Evaluating security and performance of open-source SIEM solutions for SMEs. PLOS ONE, 19(3), e0301183. https://doi.org/10.1371/journal.pone.0301183

Nalliah, R. P., Praveen, S., Allareddy, T. V., Gajendrareddy, P., Lee, M. K., Oubaidin, M., & Allareddy, V. (2025). Cybersecurity threats and preparedness: Implications for dental schools. Journal of Dental Education, 89(4). https://doi.org/10.1002/jdd.13758

Negara, B. S. dan S. (2025). Threat Intelligence Report: Indonesian Critical Infrastructure Cybersecurity Status. BSSN.

Sahin, O., & Uludag, S. (2025). Leveraging inter-arrival time for efficient threat filtering: A parsimonious approach. Computers & Security, 154, 104471. https://doi.org/10.1016/J.COSE.2025.104471

Shakir, Y. H., Abdelhamied, M. M., Al Mandhari, E. A. A., Alkhazraji, A., & Reda, N. M. (2026). An efficient approach for cyber-attack detection by using machine learning and deep learning algorithms. IAES International Journal of Artificial Intelligence, 15(2). https://doi.org/10.11591/ijai.v15.i2.pp1219-1235

Tariq, A., Manzoor, J., Aziz, M. A., Tariq, Z. U. A., & Masood, A. (2023). Open source SIEM solutions for an enterprise. Information and Computer Security, 31(1). https://doi.org/10.1108/ICS-09-2021-0146

Uccello, F., Pawlicki, M., D’Antonio, S., Kozik, R., & Choraś, M. (2024). Towards Hybrid NIDS: Combining Rule-Based SIEM with AI-Based Intrusion Detectors. Lecture Notes in Networks and Systems, 956 LNNS, 244–255. https://doi.org/10.1007/978-3-031-56950-0_21

Vazão, A. P., Santos, L., Costa, R. L. de C., & Rabadão, C. (2023). Implementing and evaluating a GDPR-compliant open-source SIEM solution. Journal of Information Security and Applications, 75. https://doi.org/10.1016/j.jisa.2023.103509

Wagner, C., Dulaunoy, A., Wagener, G., & Iklody, A. (2016). Misp: The design and implementation of a collaborative threat intelligence sharing platform. In Proceedings of the 2016 ACM on workshop on information sharing and collaborative security (pp. 49-56).

Yin, L., Hu, H., Li, K., Zheng, G., Qu, Y., & Chen, H. (2023). Improvement of DBSCAN Algorithm Based on K-Dist Graph for Adaptive Determining Parameters. Electronics (Switzerland), 12(15). https://doi.org/10.3390/electronics12153213