Hello, and welcome, time to recap on what you've learned in this course. The world of data is evolving with larger amounts of data and differing data types. You've considered how newer processing techniques are challenging traditional technology roles and changing the day-to-day work of data professionals. You briefly reviewed the two broad types of data, structured and unstructured. In relational database systems like Microsoft SQL server, Azure SQL database and Azure SQL data warehouse, the data structure is defined at design time in the form of tables. Unstructured data is stored in non-relational systems, commonly called unstructured or no SQL systems. And the data structure isn't defined at design time. You might remember that a data engineer works mainly with unstructured data and a wide variety of new data types such as streaming data. As a data engineer, you'll extract raw data from those structured or unstructured data pools. Because the data source might have a different structure than the destination, you'll transform the data from the source schema to the destination schema. The transformation processes you looked at where ETL, and the alternative approach of ELT. A disadvantage of the ETL approach is that the transformation stage can take a long time. This stage can potentially tie up source system resources. An alternative approach is to extract, load, and transform, ELT. When businesses are considering migrations from on-premises to cloud based solutions, there are several factors to take into consideration. You saw that Microsoft Azure reduces the complexity of building and deploying servers, allowing you to use a web user interface for simple deployments and to use powerful scripts for more complex deployments. You explored how Microsoft Azure can work for a range of industries, including the web, healthcare, and Internet of things, IoT. And you saw how Azure makes a real difference, accelerating big data analytics and AI solutions. In learning your new skills, you saw how the data engineer, data scientist, and artificial intelligence engineer roles are developing. And you followed a high level architecture example of how to holistically design a project by following the five phases of data, sourcing, ingest, preparation, analysis, and consumption. You saw that Azure storage accounts are the base storage type within Azure, and that Azure storage offers four configuration options Azure blob, Azure files, Azure queue, and Azure table. You were introduced to the tools that can ingest data into Azure storage. You explored the key factors to consider in deciding on the optimal storage. And you saw that the key features of Azure storage accounts are that they are scalable, secure, durable, and highly available, with hardware maintenance updates and critical issues handled by Azure. For security, you saw how Azure storage encrypts data, giving you control over who has access. Then you saw how data storage in Azure Data Lake Storage works with its Hadoop compatible data repository and compute aspect that includes platforms like HD insight, Hadoop, and Azure Databricks. You explored the key features of Data Lake Storage. You considered its data ingest using Azure data factory, Apache Sqoop, Azure Storage Explorer, the AzCopy tool, Power Shell, or Visual Studio. And you saw that you can query data by using the u SQL language while learning that with Azure Active Directory ACLs, security administrators can control data access. You found out that Azure Cosmos DB is a globally distributed multi-model database with key strengths in the areas of uptime, replication, and consistency. And you saw an example of where it can help resolve the business problems of an e-commerce retailer. You now know that it can be deployed using several API models, accepts ingest data from Azure data factory and other sources such as JSON, and can handle queries as stored procedures, triggers, and user defined functions. Finally, you saw how Azure Cosmos DB supports data encryption, IP firewall configurations and access from virtual networks. You reviewed the Platform as a Service version of Azure SQL Database, a managed relational database service supporting structures such as relational data and unstructured formats such as spatial and XML data. You saw how you might utilize its comprehensive security and availability to scale up and scale down OLTP systems on demand. You saw the key features of Azure SQL Database. Considered that it can ingest data through application integration from a wide range of developer SDKs, and looked at using T sequel to query the data. Finally, you saw how it security features help your application meet security and compliance requirements.
