Architecting a Successful Modern Data Analytics Platform in the Cloud

After we discussed the concepts for Building a Successful Modern Data Analytics Platform in the Cloud, it is time to architect it. This post will review the reference architectures for our scalable, flexible and robust design in both Amazon Web Services (AWS) and Microsoft Azure.

Introduction

I worked in the last 20 years with hundreds of companies to harness the ever-evolving technology tools to bring to life business ideas. Some companies were startups born to the cloud age (Waze, Viber, and JFrog, for example). Some were tech giants (Amazon and Intuit, For instance) with many technical people and software developers. Yet, some are large enterprises that are trying to reinvent themselves to remain competitive for the future. This post is focused on the latter type of companies, which are the backbone of our economy today (Capital One, AmEx, Liberty Mutual, Orbia, Grupo Salinas, and PepsiCo, for example).

The way to stay competitive and to adopt modern technologies, such as Cloud (faster time-to-value) and Artificial Intelligence (smarter usage of data, advanced analytics, and machine learning), is by setting up an environment for the “Flywheel of AI.” I explained the details of the flywheel in a previous post (The Flywheel of Machine Learning Systems), and in short, it is to start rotating the flywheel with:

• Feeding of more and more business use cases that can benefit from data and machine learning models,