Cutting through the confusion: data warehouse vs. data lake vs. data lakehouse

Data warehouses are designed to store structured, curated data, organizing datasets in tables and columns. This data is easily available to users for traditional business intelligence, dashboards, and reporting.

A three-tier architecture is the most commonly used approach to designing data warehouses. It comprises:

Data warehouses have been around for a few decades. Traditionally, data warehouses were hosted on premises, meaning companies had to purchase all hardware and deploy software locally, either paid or open-source systems. They also needed a whole IT team to maintain the data warehouse. On the bright side, traditional data warehouses were bringing in (and still do so today) a fast time-to-insight with no latency issues, total control of data together with one hundred percent privacy, and minimized security risk. With cloud ubiquity and technology advances, many organizations are looking to modern data architecture and migrating their data warehousing solutions to the cloud where their data is both stored and analyzed using some type of an integrated query engine. There are a variety of established cloud data warehouse solutions in the market. Each provider offers its unique set of warehouse capabilities and different pricing models. For example, Amazon Redshift is organized as a traditional data warehouse. Snowflake is similarly. Microsoft Azure is an SQL data warehouse, while Google BigQuery is based on a serverless architecture offering in essence software-as-a-service (SaaS), rather than infrastructure or platform-as-a-service like, for instance, Amazon Redshift. Among well-known on-premises data warehouse solutions are IBM Db2, Oracle Autonomous Database, IBM Netezza, Teradata Vantage, SAP HANA, and Exasol. They are also available on the cloud. Cloud-based data warehouses are obviously cheaper because there is no need to buy or roll out physical servers. Users pay only for the storage space and computing power as needed. Cloud solutions are also much easier to scale or integrate with other services. Serving highly specific business needs with top data quality and fast insights, data warehouses are here to stay for long.

Data warehouses deliver high-speed and high-performance analytics on petabytes and petabytes of historical data. They are fundamentally designed for BI-type queries. A data warehouse might give an answer about, for instance, sales in a particular time period, grouped by region or division, and year-on-year movements in sales. Key use cases for data warehouses are:

Data lakes mostly collect unrefined raw data in its original form. Another key difference between the data lake and the data warehouse is that data lakes store this data without arranging it into any logical relationships that are called schemas. However, this is how they enable more sophisticated analytics. Data lakes pull in (i) transactional data from business applications such as ERP, CRM, or SCM, (ii) documents in .csv and .txt formats, (iii) semi-structured data such as XML, JSON, and AVRO formats, (iv) device logs and IoT sensors, and (v) images, audio, binary, PDF files.

Data lakes use a flat architecture for data storage. Its key components are:

The rise of big data in the early 2000s has brought both grand opportunities and grand challenges for organizations. Business needed new technology to analyze these massive, messy, and ridiculously fast-growing datasets to capture a business impact from the big data. In 2008, Apache Hadoop came up with an innovative open-source technology for collecting and processing unstructured data on a massive scale, paving the way for big data analytics and data lakes. Shortly after, Apache Spark emerged. It was easier to use. In addition, it provided capabilities for building and training ML models, querying structured data using SQL, and processing real-time data. Today data lakes are predominately cloud-hosted repositories. All top cloud providers such as AWS, Azure, and Google offer cloud-based data lakes with cost-effective object storage services. Their platforms come with various data management services to automate deployment. In one scenario, for instance, a data lake might consist of a data storage system like the Hadoop Distributed File System (HDFS) or Amazon S3 integrated with a cloud data warehouse solution like Amazon Redshift. These components would be decoupled from services in the ecosystem which might include Amazon EMR for data processing, Amazon Glue that provides the data catalog and transformation functionality, the Amazon Athena query service, or Amazon Elasticsearch Service that is used to build a metadata repository and index data. Local data lakes are still common because of usual cloud concerns like security, privacy, or latency. There are also on-premise storage vendors that offer some products for data lakes, but their data lake offerings, however, are not well-defined. Unlike data warehouses, data lakes don’t have many years of real-world deployments behind them. There is still much criticism describing the data lake concept as blurry and ill-defined. Critics also argue that few people in any organization have the skills (or enthusiasm for that matter) to run exploratory workloads against raw data. The idea that data lakes should be used as a central repository for all enterprises’ data needs to be approached with caution, they say. There has also been a provocative talk that data lake days are numbered. The following reasons are cited:

The main idea about data lakes is to give business access to all available data from all sources as quickly as possible. Data lakes do not just give a picture of what happened yesterday. Storing massive amounts of data, data lakes are designed to enable organizations to learn more about both the present (using streaming analytics) and the future (using big data solutions, including predictive analytics and machine learning). Key use cases for data lakes are:

Marketing aside, the key idea about a data lakehouse is to bring computing power to a data lake. Architecturally, the data lakehouse usually consists of:

Whether you want to build a data storage solution from scratch or modernize your legacy system to support ML or improve performance, the right answer won't be easy. There’s still a lot of mess about key differences, benefits, and costs, with offerings and pricing models from vendors rapidly evolving. Besides, it’s always a difficult project even if you have stakeholders’ buy-in. However, there are some key considerations when choosing the data warehouse vs. data lake vs. data lakehouse. The primary question you should answer is: WHY. A good point here to remember is that key differences between data warehouse, lakes, and lakehouses do not lie in technology. They are about serving different business needs. So why do you need a data storage solution in the first place? Is it for regular reporting, business intelligence, real-time analytics, data science, or other sophisticated analysis? Is data consistency or timeliness more important for your business needs? Spend some time developing use cases. Your analytics needs should be well defined. You should deeply understand your users and skillsets too. A few rules of thumbs are: