In this first lesson, we'll discuss the streaming of data into BigQuery and using Google Data Studio to visualize results. Streaming data is not added to BigQuery via a load job, there is a separate BigQuery method called streaming inserts. Streaming inserts allows you to insert one item at a time into a table. New tables can be created from a temporary table that identifies the schema to be copied. Usually the data is available within seconds. The data enters a streaming buffer where it is held briefly until it can be inserted into the table. Data availability and consistency are considerations. Candidates for streaming our analysis are applications that are tolerant of late or missing data or data arriving out of order or data that is duplicated. The stream can pass through other services, introducing additional latency and the possibility of errors. Since streaming data is unbounded, you need to consider the streaming quotas. There is both a daily limit and a concurrent rate limit. You can find more information about these in the online documentation. You can disable best effort deduplication by not populating the insert ID field for each row inserted. When you do not populate insert ID, you get much higher streaming ingest quotas for the US region, one million per second versus 500,000 inserts per second. This leads to a pertinent question, when should you ingest a stream of data rather than use a batch approach to load data? The answer is when the immediate availability of the data is a solution requirement. The reason, well, in most cases, loading batch data is not charged, loading streaming data is a charged activity. So use batch loading or repeated batch loading rather than streaming unless that real time is a requirement of the application. Here's an example of the code used to insert streaming data into a BigQuery table. In this example, the message body has already been decoded. In a full example, a step will be required to extract the appropriate message elements to be inserted. After streaming into a BigQuery table has been initiated, you can review the data in BigQuery by querying the table receiving the streaming data. When working with data in BigQuery, including streaming data, you can use Data Studio to explore the data further. After you execute a query, you can choose Data Studio from the explore data options to immediately start creating visualizations as part of a dashboard. This is the Data Studio homepage. There are two ways to create a new report from scratch. Select Blank Report in the templates panel in the middle of the screen. Or click the "Create" button in the navigation pane on the left of the screen. Note that you can have any or all of these data sources in a single Data Studio report. In addition to the Google connectors, there is an increasing list of partner connectors to choose from as well. Since Data Studio reports can be shared, you should be aware of the ramifications of adding a data source. When you add a data source to a report, other people who can view the report can potentially see all the data in that data source. Anyone who can edit the report can use all the fields from any added data sources to create new charts with them. Click "Add To Report." Having selected a dataset, you can specify what elements of the dataset you wish to visualize. This includes selecting the dimensions and metrics that you want to use from the available fields of your dataset. The Edit Data Source picker in front of the data source name can be selected to edit the dataset fields. Easily change your data table view to a chart by clicking "Chart" in the Properties panel and selecting a chart type from the options provided. You can edit the style of your chart or even change your chart type selection later. You can also revert back to a data table view. You can also add separate charts by selecting "Add a chart" from the toolbar, resize the components on the canvas to arrange data tables in different chart types as required. In the same way that you define dimensions and metrics earlier, do the same for your chart by adding selections from the available fields list. Tip, the sequence of the fields under metric will determine the order in which the data is displayed in the chart. Use the drag feature to easily change the sequence of the fields. Give your report a name. Since Data Studio is based on Google Drive, note that you can have duplicate filenames. Click the "View" toggle button to view the end-user version of the report. Here's your report. Notice it looks very similar to when you're editing it, but as a viewer, you can't modify the report. When a viewer mice is over the chart, they are able to view live data. In this example, the viewer is able to see that in the year 2000, there were 31 natural disasters attributable to extreme temperature. Note that users cannot edit your reports unless you give them permission. One of the Google Cloud products that helps manage the performance of dashboards is BigQuery BI Engine. BI Engine is a fast in-memory analysis service that is built directly into BigQuery and is available to speed up your business intelligence applications. Historically, BI teams would have to build, manage, and optimize their own BI service and OLAP cubes to support reporting applications. Now with BI Engine, you can get sub-second query response time on your BigQuery datasets without having to create your own cubes. BI Engine is built on top of the same BigQuery storage and compute architecture and service as a fast in-memory intelligent caching service that maintains state.
