Kybernetika 62 no. 3, 456-480, 2026

From API calls to behavioral graphs: state-clustering and Markov transition features for classical and quantum malware detection

Aktham Youssef, Ivan Zelinka and Eslam AmerDOI: 10.14736/kyb-2026-3-0456

Abstract:

This paper presents a malware-detection pipeline for Windows API-call sequences that converts variable-length traces into fixed-length behavioral features. API tokens are embedded with Word2Vec and clustered with $K$-means to form a shared behavioral state space, and each trace is summarized by first-order Markov transition frequencies ($S^2$ features). Experiments on 150{,}656 traces with a family-aware split (20\% of families held out for testing) show that linear baselines trained on the full transition representation provide strong and scalable performance. Quantum models are evaluated under explicit computational limits by applying TruncatedSVD to low-dimensional subsets and comparing a fidelity-kernel QSVM and a variational quantum classifier against a matched classical SVC-RBF on identical stratified subsets. Beyond ROC/PR curves, security-oriented operating points are reported by selecting thresholds on validation to enforce high malware recall ($R_{\mathrm{mal}}\ge 0.90$), highlighting the recall trade-off and the scaling cost of quantum kernel evaluation as qubit count increases.

Keywords:

malware detection, API call sequences, Word2Vec, $K$-means, Markov transitions, quantum machine learning, QSVM, VQC

Classification:

81P68, 68M25, 68T05, 68T10

paper.pdf

References:

  1. E. Amer, S. El-Sappagh, T. Abuhamad, B. Ali Saleh Al-Rimy and A. Mohasseb: Graphshield: Advanced dynamic graph-based malware detection using graph neural networks. Expert Systems Appl. (2025).   CrossRef
  2. E. Amer and I. Zelinka: A dynamic windows malware detection and prediction method based on contextual understanding of api call sequence. Computers Security 92 (2020), 101760.   DOI:10.1016/j.cose.2020.101760
  3. E. Amer, I. Zelinka and S. El-Sappagh: A multi-perspective malware detection approach through behavioral fusion of api call sequence. Computers Security 110 (2021), 102449.   DOI:10.1016/j.cose.2021.102449
  4. G. Barrué and T. Quertier: Quantum machine learning for malware classification. In: Joint European Conference on Machine Learning and Knowledge Discovery in Databases (ECML PKDD), Springer 2023, pp. 245-260.   CrossRef
  5. A. Bellante, T. Fioravanti, M. Carminati, S- Zanero and A. Luongo: Evaluating the potential of quantum machine learning in cybersecurity: A case-study on pca-based intrusion detection systems. Computers Security 154 (2025), 104341.   DOI:10.1016/j.cose.2025.104341
  6. M. Cerezo, A. Sone, T. Volkoff, L. Cincio and P. J. Coles: Variational quantum algorithms. Nature Rev. Phys. 3 (2021), 625-644.   DOI:10.1038/s42254-021-00348-9
  7. L- Cui, J. Cui, Y. Ji, Z. Hao, L. Li and Z. Ding: Api2vec: Learning representations of api sequences for malware detection. In: Proc. 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis 2023, pp. 261-273.   CrossRef
  8. T. Fawcett: An introduction to roc analysis. Pattern Recogn. Lett. 27 (2006), 8, 861-874.   DOI:10.1016/j.patrec.2005.10.010
  9. V. Havlíček, A. D. Córcoles, K. Temme, A. W. Harrow, A. Kandala, J. M. Chow and J. M. Gambetta: Supervised learning with quantum-enhanced feature spaces. Nature 567 (2019), 7747, 209-212.   DOI:10.1038/s41586-019-0980-2
  10. I. Kwon and E. G. Im: Extracting the representative api call patterns of malware families using recurrent neural network. In: Proc. International Conference on Research in Adaptive and Convergent Systems 2017, pp. 202-207.   CrossRef
  11. E. Mariconti, L. Onwuzurike, P. Andriotis, E.De Cristofaro, G. Ross and G. Stringhini: MAMADroid: Detecting android malware by building markov chains of behavioral models. In: Proc. Network and Distributed System Security Symposium (NDSS). Internet Society 2017.   CrossRef
  12. J. R. McClean, S. Boixo, V. N. Smelyanskiy, R. Babbush and H. Neven: Barren plateaus in quantum neural network training landscapes. Nature Commun. 9 (2018), 4812.   CrossRef
  13. T. Mikolov, K. Chen, G. Corrado and J. Dean: Efficient estimation of word representations in vector space. arXiv preprint, arXiv:1301.3781, 2013.   CrossRef
  14. O. Or-Meir, N. Nissim, Y. Elovici and L. Rokach: Dynamic malware analysis in the modern era--a state of the art survey. ACM Comput. Surveys, 2019.   CrossRef
  15. M. Emre Sahin, Edoardo Altamura, Oscar Wallis, Stephen P. Wood, Anton Dekusar, Declan A. Millar, Takashi Imamichi, Atsushi Matsuo and Stefano Mensa: Qiskit machine learning: an open-source library for quantum machine learning tasks at scale on quantum hardware and classical simulators. arXiv preprint, arXiv:2505.17756, 2025.   CrossRef
  16. T. Saito and M. Rehmsmeier: The precision-recall plot is more informative than the roc plot when evaluating binary classifiers on imbalanced datasets. PLOS ONE 10 (2015), 3, e0118432.   DOI:10.1371/journal.pone.0118432
  17. M. Schuld: Supervised quantum machine learning models are kernel methods. arXiv preprint, arXiv:2101.11020, 2021.   CrossRef
  18. S. Thanasilp, S. Wang, M. Cerezo and Z. Holmes: Exponential concentration in quantum kernel methods. Nature Commun. 15 (2024), 1.   CrossRef
  19. I. Zelinka and E. Amer: An ensemble-based malware detection model using minimum feature set. In: Mendel 25 (2019), pp. 1-10.   DOI:10.13164/mendel.2019.2.001
  20. F. O. Catak, A. F. Yazi, O. Elezaj and J. Ahmed: Deep learning based sequential model for malware analysis using windows exe api calls. PeerJ Computer Sci. (2020), 6, e285.   DOI:10.7717/peerj-cs.285