IntelliOps: A Generic Multi-Source Monitoring Framework with Predictive Analytics for Enterprise Infrastructure

Main Article Content

Alper Ozpinar
Muhammet Mustafa Alarçin
Hakkı Kıvanç Yeker

Abstract

This paper presents IntelliOps, a novel monitoring framework that integrates multi-source system monitoring with predictive analytics capabilities for financial technology infrastructure. The proposed framework aggregates performance metrics from multiple monitoring platforms and consolidates them through a unified API, providing comprehensive visibility into both hardware and software performance metrics. IntelliOps introduces an innovative approach by synthesizing traditional monitoring methodologies with advanced machine learning techniques, incorporating time series predictive models (LSTM, GRU, RNN) and contemporary forecasting libraries for anomaly detection and predictive maintenance.


The framework's architecture consists of three primary components: (1) a centralized data collection system that integrates heterogeneous monitoring sources, (2) an analytical engine that processes infrastructure and application-level metrics, and (3) a machine learning pipeline that performs predictive analysis on the aggregated data. Our implementation analyzes a longitudinal dataset spanning over one year from a large-scale fintech platform, encompassing metrics such as multi-layer response times (caching, message queuing, runtime environment, databases), request volumes, error rates, and deployment events.


Experimental results demonstrate the framework's efficacy in anomaly detection and predictive maintenance, achieving high accuracy across diverse datasets. The evaluation reveals that our hybrid methodology, incorporating both supervised and unsupervised learning techniques, yields superior performance in risk segmentation and anomaly detection compared to conventional threshold-based monitoring systems. Additionally, the integration of modern time series analysis techniques with classical statistical models enables robust detection of seasonal patterns and trends, facilitating proactive infrastructure management.


This research advances the field of systems monitoring by providing a structured methodology for implementing deep learning models in targeted monitoring scenarios, thereby enhancing system performance and mitigating potential disruptions across diverse operational environments. The framework's adaptability and scalability make it particularly suitable for complex financial technology infrastructures where system reliability and performance are paramount.

Downloads

Download data is not yet available.

Article Details

How to Cite
Ozpinar, A., Alarçin, M. M. ., Halim, V. ., & Yeker, H. K. (2024). IntelliOps: A Generic Multi-Source Monitoring Framework with Predictive Analytics for Enterprise Infrastructure. The European Journal of Research and Development, 4(4), 378–393. https://doi.org/10.56038/ejrnd.v4i4.588
Section
Articles

References

Awotunde, J. B., Adeniyi, E. A., Ogundokun, R. O., & Ayo, F. E. (2021). Application of big data with fintech in financial services. In Fintech with artificial intelligence, big data, and blockchain (pp. 107–132). Springer. DOI: https://doi.org/10.1007/978-981-33-6137-9_3

Bajao, N. A., & Sarucam, J. (2023). Threats Detection in the Internet of Things Using Convolutional neural networks, long short-term memory, and gated recurrent units. Mesopotamian Journal of Cybersecurity, 2023, 22–29. DOI: https://doi.org/10.58496/MJCS/2023/005

Baresi, L., Garriga, M., & De Renzis, A. (2017). Microservices identification through interface analysis. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 10465 LNCS, 19–33. https://doi.org/10.1007/978-3-319-67262-5_2 DOI: https://doi.org/10.1007/978-3-319-67262-5_2

Bin, L., Chuang, L., Jian, Q., Jianping, H., & Ungsunan, P. (2008). A NetFlow based flow analysis and monitoring system in enterprise networks. Computer Networks, 52(5), 1074–1092. https://doi.org/10.1016/J.COMNET.2007.12.004 DOI: https://doi.org/10.1016/j.comnet.2007.12.004

Campos, J. (2009). Development in the application of ICT in condition monitoring and maintenance. Computers in Industry, 60(1), 1–20. DOI: https://doi.org/10.1016/j.compind.2008.09.007

Cassar, I., Francalanza, A., Aceto, L., & Ingólfsdóttir, A. (2017). A survey of runtime monitoring instrumentation techniques. Electronic Proceedings in Theoretical Computer Science, EPTCS, 254, 15–28. https://doi.org/10.4204/EPTCS.254.2 DOI: https://doi.org/10.4204/EPTCS.254.2

Cerny, T., Donahoo, M. J., & Trnka, M. (2018). Contextual understanding of microservice architecture. ACM SIGAPP Applied Computing Review, 17(4), 29–45. https://doi.org/10.1145/3183628.3183631 DOI: https://doi.org/10.1145/3183628.3183631

Ciuffoletti, A. (2015). Automated Deployment of a Microservice-based Monitoring Infrastructure. Procedia Computer Science, 68, 163–172. https://doi.org/10.1016/j.procs.2015.09.232 DOI: https://doi.org/10.1016/j.procs.2015.09.232

Daoud, M., El Mezouari, A., Faci, N., Benslimane, D., Maamar, Z., & El Fazziki, A. (2021). A multi-model based microservices identification approach. Journal of Systems Architecture, 118. https://doi.org/10.1016/j.sysarc.2021.102200 DOI: https://doi.org/10.1016/j.sysarc.2021.102200

de Toledo, S. S., Martini, A., & Sjøberg, D. I. K. (2021). Identifying architectural technical debt, principal, and interest in microservices: A multiple-case study. Journal of Systems and Software, 177. https://doi.org/10.1016/j.jss.2021.110968 DOI: https://doi.org/10.1016/j.jss.2021.110968

Dias-Neto, A. C., Matalonga, S., Solari, M., Robiolo, G., & Travassos, G. H. (2017). Toward the characterization of software testing practices in South America: looking at Brazil and Uruguay. Software Quality Journal, 25(4), 1145–1183. https://doi.org/10.1007/S11219-016-9329-3 DOI: https://doi.org/10.1007/s11219-016-9329-3

Ekundayo, F., Atoyebi, I., Soyele, A., & Ogunwobi, E. (2024). Predictive Analytics for Cyber Threat Intelligence in Fintech Using Big Data and Machine Learning. Int J Res Publ Rev, 5(11), 1–15. DOI: https://doi.org/10.55248/gengpi.5.1124.3352

Hannousse, A., & Yahiouche, S. (2021). Securing microservices and microservice architectures: A systematic mapping study. Computer Science Review, 41. https://doi.org/10.1016/j.cosrev.2021.100415 DOI: https://doi.org/10.1016/j.cosrev.2021.100415

Jansen, B. J. (2006). Search log analysis: What it is, what’s been done, how to do it. Library & Information Science Research, 28(3), 407–432. DOI: https://doi.org/10.1016/j.lisr.2006.06.005

Kosinska, J., Balis, B., Konieczny, M., Malawski, M., & Zielinski, S. (2023). Toward the Observability of Cloud-Native Applications: The Overview of the State-of-the-Art. IEEE Access, 11, 73036–73052. https://doi.org/10.1109/ACCESS.2023.3281860 DOI: https://doi.org/10.1109/ACCESS.2023.3281860

Li, B., Springer, J., Bebis, G., & Hadi Gunes, M. (2013). A survey of network flow applications. Journal of Network and Computer Applications, 36(2), 567–581. https://doi.org/10.1016/j.jnca.2012.12.020 DOI: https://doi.org/10.1016/j.jnca.2012.12.020

Lin, T.-T., & Siewiorek, D. P. (1990). Error log analysis: statistical modeling and heuristic trend analysis. IEEE Transactions on Reliability, 39(4), 419–432. DOI: https://doi.org/10.1109/24.58720

Lin, Y. H., Shih, W. C., & Chang, Y. K. (2022). Efficient hierarchical hash tree for OpenFlow packet classification with fast updates on GPUs. Journal of Parallel and Distributed Computing, 167, 136–147. https://doi.org/10.1016/j.jpdc.2022.04.018 DOI: https://doi.org/10.1016/j.jpdc.2022.04.018

Meng, L., Ji, F., Sun, Y., & Wang, T. (2021). Detecting anomalies in microservices with execution trace comparison. Future Generation Computer Systems, 116, 291–301. https://doi.org/10.1016/j.future.2020.10.040 DOI: https://doi.org/10.1016/j.future.2020.10.040

Naiman, D. Q. (2004). Statistical anomaly detection via httpd data analysis. Computational Statistics and Data Analysis, 45(1), 51–67. https://doi.org/10.1016/S0167-9473(03)00115-4 DOI: https://doi.org/10.1016/S0167-9473(03)00115-4

Ponce, F., Soldani, J., Astudillo, H., & Brogi, A. (2022). Smells and refactorings for microservices security: A multivocal literature review. Journal of Systems and Software, 192. https://doi.org/10.1016/j.jss.2022.111393 DOI: https://doi.org/10.1016/j.jss.2022.111393

Qassim, Q. S., Zin, A. M., & Ab Aziz, M. J. (2017). Anomaly-based network IDS false alarm filter using cluster-based alarm classification approach. International Journal of Security and Networks, 12(1), 13–26. https://doi.org/10.1504/IJSN.2017.081056 DOI: https://doi.org/10.1504/IJSN.2017.081056

Rezaei Nasab, A., Shahin, M., Liang, P., Basiri, M. E., Hoseyni Raviz, S. A., Khalajzadeh, H., Waseem, M., & Naseri, A. (2021). Automated identification of security discussions in microservices systems: Industrial surveys and experiments. Journal of Systems and Software, 181. https://doi.org/10.1016/j.jss.2021.111046 DOI: https://doi.org/10.1016/j.jss.2021.111046

Shinozawa, Y., & Vivian, A. (2015). Determinants of money flows into investment trusts in Japan. Journal of International Financial Markets, Institutions and Money, 37, 138–161. https://doi.org/10.1016/j.intfin.2015.02.005 DOI: https://doi.org/10.1016/j.intfin.2015.02.005

Shumway, R. H., Stoffer, D. S., & Stoffer, D. S. (2000). Time series analysis and its applications (Vol. 3). Springer. DOI: https://doi.org/10.1007/978-1-4757-3261-0

Vale, G., Correia, F. F., Guerra, E. M., De Oliveira Rosa, T., Fritzsch, J., & Bogner, J. (2022). Designing Microservice Systems Using Patterns: An Empirical Study on Quality Trade-Offs. Proceedings - IEEE 19th International Conference on Software Architecture, ICSA 2022, 69–79. https://doi.org/10.1109/ICSA53651.2022.00015 DOI: https://doi.org/10.1109/ICSA53651.2022.00015

Waseem, M., Liang, P., Shahin, M., Di Salle, A., & Márquez, G. (2021). Design, monitoring, and testing of microservices systems: The practitioners’ perspective. Journal of Systems and Software, 182, 111061. https://doi.org/10.1016/J.JSS.2021.111061 DOI: https://doi.org/10.1016/j.jss.2021.111061

Xia, H., Fang, B., Roughan, M., Cho, K., & Tune, P. (2018). A BasisEvolution framework for network traffic anomaly detection. Computer Networks, 135, 15–31. https://doi.org/10.1016/j.comnet.2018.01.025 DOI: https://doi.org/10.1016/j.comnet.2018.01.025

Xin, R., Chen, P., & Zhao, Z. (2023). CausalRCA: Causal inference based precise fine-grained root cause localization for microservice applications. Journal of Systems and Software, 203. https://doi.org/10.1016/j.jss.2023.111724 DOI: https://doi.org/10.1016/j.jss.2023.111724

Zhang, M., Arcuri, A., Li, Y., Liu, Y., & Xue, K. (2023). White-Box Fuzzing RPC-Based APIs with EvoMaster: An Industrial Case Study. ACM Transactions on Software Engineering and Methodology, 32(5). https://doi.org/10.1145/3585009 DOI: https://doi.org/10.1145/3585009

Zhao, R., Wang, D., Yan, R., Mao, K., Shen, F., & Wang, J. (2017). Machine health monitoring using local feature-based gated recurrent unit networks. IEEE Transactions on Industrial Electronics, 65(2), 1539–1548. DOI: https://doi.org/10.1109/TIE.2017.2733438