Layered Intelligence Medallion-Driven MLOps on Azure Databricks
Keywords:
Medallion Architecture, Databricks, Azure Cloud, MLOps, Data Intelligence, Data Engineering, Data Pipelines, Data Governance, Data Architecture, Machine Learning, Model Deployment, CI CD, Model Lifecycle, Data Processing, Scalable Systems, Analytics Systems, Cloud Platforms, Data Integration, Automation, Enterprise Data.Abstract
This Medallion Architecture design and implementation leverages Databricks and Azure technology while aligning with MLOps principles to engender a comprehensive, scalable solution for enterprise data intelligence, a topic of increasing relevance across commercial sectors. As organizations expand their bases of data assets, the need for an integrated framework that supports both analytics and machine learning with a clear vision for data intelligence emerges as a priority.
Today, machine learning is applied across many industries and is advancing rapidly. However, the majority of ML models remain on pilot status or just a few have been productized. Furthermore, their use, maintenance, and governance sprinkle a wide array of manual and poorly controlled actions. The Achilles heel of these initiatives that are such a big promise of added value is the implementation of a continuous integration/continuous deployment (CI/CD) flow. ML-Ops is the answer to these impediments and allows exploring ways to build ML products properly.
References
1. Amershi, S., Weld, D., Vorvoreanu, M., Fourney, A., Nushi, B., Collisson, P., Suh, J., Iqbal, S., Bennett, P. N., Inkpen, K., Teevan, J., Kikin-Gil, R., & Horvitz, E. (2022). Guidelines for human-AI interaction. Communications of the ACM, 65(1), 82–90.
2. Arnott, D., Lizama, F., & Song, Y. (2022). Machine learning operations: Foundations and future directions. Information Systems Frontiers, 24(6), 1887–1905.
3. Ashmore, R., Calinescu, R., & Paterson, C. (2022). Assuring the machine learning lifecycle. Computer, 55(5), 20–30.
4. Awan, A. A., Yaseen, M., & Ali, M. (2022). Data governance challenges in cloud-based machine learning systems. Future Internet, 14(9), 265.
5. Baier, L., Kühl, N., & Satzger, G. (2022). How to cope with change? Prescriptive maintenance of machine learning pipelines. Business & Information Systems Engineering, 64(4), 425–441.
6. Bencomo, N., Cheng, B. H. C., Belaggoun, A., & Issarny, V. (2022). Self-adaptation for machine learning systems. ACM Computing Surveys, 55(8), 1–36.
7. Bosch, J., Olsson, H. H., Crnkovic, I., & Svensson, J. (2022). Engineering AI systems: A research agenda. AI Magazine, 43(2), 145–156.
8. Breck, E., Polyzotis, N., Roy, S., Whang, S. E., & Zinkevich, M. (2022). Data validation for machine learning. Proceedings of the VLDB Endowment, 15(12), 3470–3483.
9. Brundage, M., Avin, S., Clark, J., Toner, H., Eckersley, P., Garfinkel, B., Dafoe, A., Scharre, P., Zeitzoff, T., Filar, B., Anderson, H., Roff, H., Allen, G., Steinhardt, J., Flynn, C., hÉigeartaigh, S., Beard, S., Belfield, H., Farquhar, S., & Amodei, D. (2022). Toward trustworthy AI development. AI and Ethics, 2(3), 421–437.
10. Budler, M., & Komac, B. (2022). Explainable artificial intelligence in enterprise systems. Expert Systems with Applications, 197, 116673.
11. Casado, M., Chou, P., & Ferris, C. (2022). Cloud-native machine learning operations and orchestration. IEEE Cloud Computing, 9(4), 14–24.
12. Chaki, N., Thillaiarasu, N., & Cortesi, A. (2022). Machine learning engineering for cloud platforms. Journal of Cloud Computing, 11(1), 85.
13. Chen, C., Lin, Y., Chen, Y., & Chou, W. (2022). Data quality assessment for large-scale AI applications. Information Sciences, 607, 1180–1195.
14. D’Amour, A., Heller, K., Moldovan, D., Adlam, B., Alipanahi, B., Beutel, A., Chen, P. C., Deaton, J., Eisenstein, J., Hoffman, M., Horvitz, E., Hussain, Z., Liao, Q. V., Obermeyer, Z., Park, F., Platt, J., Simon, I., Tsay, J., Vogeli, C., ... Amershi, S. (2022). Underspecification presents challenges for credibility in modern machine learning. Journal of Machine Learning Research, 23(1), 1–61.
15. Dehghani, Z. (2022). Data mesh principles and logical architecture. IEEE Software, 39(4), 88–91.
16. Ebert, C., Gallardo, G., Hernantes, J., & Serrano, N. (2022). DevOps for AI. IEEE Software, 39(1), 13–17.
17. Emani, C. K., Cullot, N., & Nicolle, C. (2022). Understandable machine learning for cloud-based environments. Knowledge-Based Systems, 244, 108516.
18. Farahani, A., Voghoei, S., Rasheed, K., & Arabnia, H. (2022). A brief review of domain adaptation. Advances in Data Science and Adaptive Analysis, 14(1), 2250002.
19. Foidl, H., Felderer, M., & Ramler, R. (2022). Monitoring machine learning systems in production. Software Quality Journal, 30(4), 1153–1180.
20. Garg, A., Kaur, K., & Kumar, R. (2022). Cloud-based AI governance frameworks. Computers & Security, 119, 102755.
21. Ge, M., Bangui, H., & Buhnova, B. (2022). Big data for artificial intelligence systems. Future Generation Computer Systems, 127, 379–390.
22. Ghofrani, J., Hossain, M. S., & Muhammad, G. (2022). Edge-cloud collaborative AI systems. IEEE Network, 36(6), 254–261.
23. Giray, G. (2023). A software engineering perspective on MLOps. Software Quality Journal, 31(4), 1235–1260.
24. Grafberger, A., Charrada, E., Jansen, S., & Roth, A. (2023). MLOps challenges and opportunities. Journal of Systems and Software, 199, 111615.
25. Hummer, W., Muthusamy, V., Rausch, T., Dube, P., El Maghraoui, K., Murthi, A., Oum, P., & Shahir, M. (2023). ModelOps: Cloud-native machine learning lifecycle management. IEEE Internet Computing, 27(1), 28–37.
26. Ishwarappa, & Anuradha, J. (2023). Enterprise AI governance and compliance. International Journal of Information Management Data Insights, 3(1), 100177.
27. Jarrahi, M. H., Memariani, A., & Guha, A. (2023). Human-AI collaboration in organizations. Business Horizons, 66(2), 229–241.
28. John, M., Olsson, H. H., Bosch, J., & Crnkovic, I. (2023). Towards industrialized machine learning systems. Empirical Software Engineering, 28(2), 42.
29. Kaul, V., Enslin, S., & Gross, S. A. (2023). History of artificial intelligence in medicine. Gastrointestinal Endoscopy, 97(4), 807–812.
30. Kim, M., Zimmermann, T., DeLine, R., & Begel, A. (2023). Data scientists in software teams. IEEE Software, 40(2), 68–76.
31. Kreuzberger, D., Kühl, N., & Hirschl, S. (2023). Machine learning operations (MLOps): Overview, definition, and architecture. IEEE Access, 11, 31866–31879.
32. Kumar, A., Patel, A., & Sharma, S. (2023). Responsible AI and governance frameworks for enterprises. AI and Ethics, 3(4), 1011–1024.
33. Lwakatare, L. E., Raj, A., Bosch, J., Olsson, H. H., Crnkovic, I., & Civili, C. (2023). Large-scale machine learning systems engineering. Information and Software Technology, 156, 107133.
34. Mahdavinejad, M. S., Rezvan, M., Barekatain, M., Adibi, P., Barnaghi, P., & Sheth, A. (2023). Machine learning for Internet of Things data analysis. Digital Communications and Networks, 9(2), 213–228.
35. Makinen, S., Skogstrom, H., Laaksonen, J., & Mikkonen, T. (2023). Who needs MLOps? Survey of MLOps benefits and challenges. IEEE Access, 11, 87641–87656.
36. Mariani, G., Boracchi, G., Matteucci, M., & Roveri, M. (2023). Machine learning model monitoring. ACM Computing Surveys, 56(2), 1–37.
37. Martinez-Fernandez, S., Bogner, J., Franch, X., Oriol, M., Siebert, J., Trendowicz, A., Vollmer, A., & Wagner, S. (2023). Software engineering for AI-based systems. ACM Transactions on Software Engineering and Methodology, 32(3), 1–44.
38. Mikalef, P., Gupta, M., Pappas, I. O., & Krogstie, J. (2023). Big data analytics and AI capability. Information & Management, 60(1), 103650.
39. Mishra, A., & Otaiwi, Z. (2023). DevOps and MLOps integration. Journal of King Saud University–Computer and Information Sciences, 35(6), 101597.
40. Mohamed, N., Al-Jaroodi, J., & Lazarova-Molnar, S. (2023). AI lifecycle management in cloud environments. Future Generation Computer Systems, 140, 284–297.
41. Nahar, N., Ara, F., & Hossain, M. S. (2023). Cloud-native AI systems and orchestration. IEEE Access, 11, 75452–75470.
42. Nguyen, D. C., Ding, M., Pathirana, P. N., Seneviratne, A., Li, J., Niyato, D., & Hwang, W. J. (2023). Federated learning for industrial AI. IEEE Communications Surveys & Tutorials, 25(1), 258–293.
43. Ozkaya, I. (2023). Systematic approaches to AI engineering. IEEE Software, 40(3), 18–23.
44. Paleyes, A., Urma, R. G., & Lawrence, N. D. (2023). Challenges in deploying machine learning. Communications of the ACM, 66(1), 74–84.
45. Ponce, E., Aquino, A., & Tovar, E. (2023). AI observability and model governance. Electronics, 12(18), 3879.
46. Rahman, M. M., Hossain, M. S., Alrajeh, N. A., & Muhammad, G. (2023). AI-enabled cloud computing systems. IEEE Network, 37(1), 34–41.
47. Raji, I. D., Smart, A., White, R. N., Mitchell, M., Gebru, T., Hutchinson, B., Smith-Loud, J., Theron, D., & Barnes, P. (2023). Closing the AI accountability gap. Proceedings of the ACM Conference on Fairness, Accountability, and Transparency, 33–44.
48. Sculley, D., Holt, G., Golovin, D., Davydov, E., Phillips, T., Ebner, D., Chaudhary, V., Young, M., Crespo, J. F., & Dennison, D. (2023). Hidden technical debt in machine learning systems revisited. Communications of the ACM, 66(8), 88–96.
49. Serrano, N., Gallardo, G., & Hernantes, J. (2023). Infrastructure automation for AI systems. IEEE Software, 40(2), 77–84.
50. Shahin, M., Babar, M. A., Zhu, L., & Huchard, M. (2023). Machine learning software engineering practices. Information and Software Technology, 158, 107205.
51. Sharma, V., Gupta, S., & Mishra, R. (2023). Enterprise AI architecture and governance. Journal of Enterprise Information Management, 36(5), 1411–1430.
52. Siebert, J., Joeckel, M., Heidrich, J., Trendowicz, A., Nakamichi, K., Ohashi, K., Namba, H., & Yoshida, N. (2023). Machine learning testing: Survey and taxonomy. Software Quality Journal, 31(2), 463–512.
53. Tamburri, D. A. (2023). Sustainable MLOps architectures. IEEE Software, 40(5), 76–83.
54. Trienes, J., Demir, B., Pfahl, D., & Schneider, K. (2023). Quality assurance in AI engineering. Empirical Software Engineering, 28(5), 108.
55. van der Aalst, W. M. P. (2023). Process mining and AI operations. Business & Information Systems Engineering, 65(2), 171–176.
56. Villamizar, M., Kalinowski, M., Garcia, A., Nunes, R., & da Silva, L. F. (2024). Continuous delivery for machine learning systems. Journal of Systems and Software, 208, 111882.
57. Wang, X., Zhao, Y., Li, J., & Zhang, H. (2024). Cloud-scale machine learning governance frameworks. Future Generation Computer Systems, 151, 475–489.
58. Weber, B., Fischer, M., & Matthes, F. (2024). Enterprise MLOps maturity models. Information Systems Frontiers, 26(1), 147–165.
59. Wirth, R., Kuhl, N., & Satzger, G. (2024). Operationalizing trustworthy AI in organizations. Business & Information Systems Engineering, 66(1), 1–15.
60. Xu, L. D., Lu, Y., & Li, L. (2024). Artificial intelligence for smart enterprises. IEEE Transactions on Industrial Informatics, 20(2), 1456–1465.
61. Yang, J., Chen, X., & Wang, H. (2024). AI observability and model drift detection. Expert Systems with Applications, 237, 121532.
62. Zhang, Y., Zhou, M., & Liu, Q. (2024). Lakehouse architecture for enterprise AI platforms. IEEE Access, 12, 45782–45798.
63. Zhou, Z. H., Chen, X., & Li, B. (2024). Data-centric AI and machine learning operations. Artificial Intelligence Review, 57(4), 92.
Additional Files
Published
Data Availability Statement
None
