Securing your data lake running on cloud storage is of paramount importance. We'll discuss the key security features that you need to know as a data engineer to control access to your objects. Cloud storage implements two completely separate but overlapping methods of controlling access to your objects. Cloud IAM, the policy and the access control lists or ACLs. Cloud IAM is standard across the Google Cloud platform. It's set at the bucket level and uniformly applies access rules to all objects within that bucket. Access control lists can be applied at the bucket level or to individual objects. So it provides more fine-grained access control. The Cloud IAM controls are as you would expect. Cloud IAM provides project roles and bucket roles. Which include bucket reader, bucket writer, and bucket owner. The ability to create or change access control lists is an IAM bucket role. In the ability to create and delete buckets and to set IAM policy, is a project level role. Custom roles are also available. Project level viewer, editor, and owner roles make user members of special internal groups that give them access by being members of those bucket roles. You can check out the online documentation for more details. When you create a bucket, you're offered the option of disabling access lists and only using Cloud IAM. Access lists are currently enabled by default. This choice used to be immutable. But now you can disable access lists, even if they were in force previously. As an example, you might have some name at email.com, Bob at example.com giving a reader access to a bucket through IAM. And you've also given them write access to a specific file in that bucket through an access control list. You can give such permissions to service accounts as well as an individual application. All data in Google Cloud is encrypted at rest and in transit and there is no way to turn off the encryption. The encryption is done by Google using encryption keys that we manage, Google managed encryption keys or GMEK. We use two levels of encryption. First, the data is encrypted using a data encryption key, and then the data encryption key itself is then encrypted using a key encryption key or a KEK. These KEKs are automatically rotated on a schedule that use the current KEK stored in Cloud KMS, or the Key Management Service. You don't have to do anything. This automatically happens. If you want to manage the KEK yourself, you can. Instead of Google managing the encryption key, you can control the creation and the existence of the KEK that is used. This is what we call customer managed encryption keys or CMEK. You can avoid Cloud KMS completely and supply your own encryption and rotation mechanism. This is called CSEK. Which data encryption option you use generally depend on your business, legal and regulatory requirements. So be sure to talk to your company's legal counsel. The fourth encryption option is client-side encryption. Client-side encryption simply means that you've encrypted the data before it's uploaded and then you have to decrypt the data yourself before it's used. Google Cloud Storage still performs GMEK, CMEK, or CSEK encryption on the object. It has no knowledge of the extra layer of encryption that you might have added. Cloud storage supports logging of data access and those logs are immutable. In addition to cloud audit logs in Google Cloud Storage access logs. There are various holds and locks that you can place in the data itself. For auditing purposes, you can place a hold on an object and all the operations that could change or delete the objects are suspended until that hold is released. You can also lock a bucket and no changes or deletions can occur until the lock is released. Finally, there is a locked retention policy as we previously discussed. And it continues to remain in effect and prevent deletion whether a bucket lock or an object hold or enforce or not. Data locking is different from encryption. Where encryption prevents somebody from understanding the data, locking prevents them from modifying the data. There are a whole host of special use cases supported by cloud storage. For example, decompressive coding. By default, the data you upload is the same data that you get back from cloud storage. This includes gzip archives, which are usually returned as gzip archives. However, if you tag an object properly in the metadata, you can cause cloud storage to decompress the file as it's being served. Benefits of the smaller compressed file are faster upload and lower storage cost compared with the uncompressed files. You can set up a bucket to be requester pays. Normally, if data is accessed from a different region, you have to pay network egress charges, that's traffic going out of the network. You can make the requester pay, so that you pay only for data storage. You can create a sign URL to anonymously share an object in cloud storage. And even have the URL expire after a period of time. It's possible to upload a single object in pieces and create a composite object without having to concatenate the pieces after upload. Now, there's so many more useful features in Google Cloud Storage, but we have to move on.
