Welcome to Data Lakes Overview. After watching this video, you will be able to: Define what a data lake is. List benefits of using a data lake. And, compare data lakes to data warehouses. A data lake is a storage repository that can store large amounts of structured, semi-structured, and unstructured data in their native format, classified and tagged with metadata. While a data warehouse stores data processed for a specific need, a data lake is a pool of raw data where each data element is given a unique identifier and is tagged with metatags for further use. You would opt for a data lake if you generate, or have access to, large amounts of data on an ongoing basis but don’t want to be restricted to specific or pre-defined use cases. Data lakes are sometimes also used as a staging area for transforming data prior to loading into a data warehouse or a data mart. Let’s break this information down a bit. A data lake is a data repository that can store a large amount of structured, semi-structured and unstructured data in its native format. You do not need to define the structure and schema of data before loading the data into the data lake. You do not even need to know all the use cases for which you will be analyzing the data. A data lake exists as a repository of raw data, straight from the source to be transformed based on the use case for which it needs to be analyzed, which does not mean that a data lake is a place where data can be dumped without governance. A data lake is also a reference architecture that is independent of technology. Data lakes can be deployed using cloud object storage, such as Amazon S3. Or, large-scale distributed systems such as Apache Hadoop, used for processing Big Data. You can deploy data lakes on relational database management systems, as well as NoSQL data repositories that can store very large amounts of data. Data lakes offer a number of benefits. Data lakes can store all types of data including: unstructured data, such as documents and emails semi-structured data, such as JSON and XML files, and structured data from relational databases. Scalability is another data lake benefit. Data lakes can make use of scalable storage capacity—from terabytes to petabytes of data. By retaining data in its original format, data lakes save organizations time that would have been used to define structures, create schemas, and transform the data. The ability to access data in its original format enables fast, flexible reuse of the data for a wide range of current and future use cases. Some of the vendors that provide technologies, platforms, and reference architectures for data lakes include: Amazon Cloudera Google IBM Informatica Microsoft Oracle SAS Snowflake Teradata And, Zaloni. All in all, data lakes were designed in response to the limitations of data warehouses. Depending on the requirements, a typical organization will require both a data warehouse and a data lake as they serve different needs. Let’s compare data lakes with data warehouses: When it comes to data, in a data lake, data is integrated in its raw and unstructured form. A data warehouse is different. Here all data has already been processed and conformed to standards prior to loading to the warehouse. Talking about schema, when using data lakes, you do not need to define the structure and schema of the data before loading into the data lake. A data warehouse on the other hand requires strict conformance to schema and therefore a schema needs to be designed and implemented prior to loading the data. How does data quality differ when looking at data lakes and data warehouses? In data lakes the data might or might not be curated, for example raw data. And data is agile and does not necessarily comply with governance guidelines. In comparison, the data in data warehouses is curated and adheres to data governance. Here, we are looking at users of data lakes and data warehouses. Data scientists, data developers, and machine learning engineers are the typical users of data lakes. Data warehouses, on the other hand, are mainly used by business analysts, and data analysts. In this video, you learned that: A data lake is a storage repository that can store large amounts of structured, semi-structured, and unstructured data in their raw or native format, classified and tagged with metadata. You do not need to define the structure and schema of data before loading into the data lake. Data lakes offer several benefits, such as storage for all types of data, scalable storage capacity, time savings, and flexible data reuse. Finally, you learned that data lakes can be used as a kind of self-serve staging area for a variety of use cases, including machine learning development and advanced analytics.
