When we're working on communication systems, one of the things that we often run into is the need to buffer communications at some point, where either the amount of information that's showing up is more than we can handle at once, or we may just want to maintain that data until someone can actually come process it. Message queuing is a concern and we're going to look at some different tools that you'll run into as you're pulling together your IoT systems, typically, there's some message patterns that you'll see with these types of systems, there's also some popular tools and platforms you can use, we're going to look in particular at ZeroMQ, RabbitMQ, and a queuing system on AWS called SQS. The tricky bit about sending messages back and forth between devices and gateways and elements of IoT systems, for the most part, it's related to scaling and interoperability. You want to make sure that the messaging is standardized between those devices so that the messaging actually works, and then you have to consider what's going to happen as these systems start to get bigger and as your networks start to transfer more and more data. Again another place where prioritization and queuing becomes an issue. Related to this of course when you're moving messages around, you have to think about how you're going to deploy these systems, or how you're going to monitor them, how you'll do over their upgrades, and then also just the security of that information. If you're sending anything like health care or personal information, it's going to have to be encrypted and you're going to have to make sure that that messaging is safe. Some typical patterns that you see in messaging, you're probably aware of some of these, publish and subscribe is probably the most common one, the nice part about publishing and subscribing, is it's independent for the publisher and the subscriber, the publisher can send any events or information that it wants to when it's ready, then the subscribers can pick up that information when they're ready to sign up for it, so it's a very generic approach and it's very good for networks of devices that might be dropping off, joining the network or intermittently accessing a network. Request response is what we've seen in a web page, and this really requires both the services on each side to be set up and ready to talk, in things like CoAP and HTTP, you're sending the setup information with every set of messages, with a WebSocket approach, you can set up once at the beginning, and then stay connected for messages, but still there is a back and forth that's occurring in those protocols. Point to point messages or peer to peer messages, again another approach where you're sending messages directly from one element to another, and again, all these things are related to the different protocols that support them, some of which we've already talked about in detail. There are other ways to do this survey type systems, Databuses, and then as we're looking at the messaging, what we'll find is some tools are going to use an intermediate broker. Something in between the, potentially lets you use multiple protocols, or at the very least queue up messages in a third-party elements so that subscribers can come pick them up when they're ready. There's a good website called Enterprise Integration Patterns, which is based on a book that's also very good that focuses on messaging patterns in trying to integrate pieces of networks. It goes all the way from endpoint to endpoint with the message types, the channels, router routes, way to monitor the traffic and wait it to translate messages, in a generic set of patterns and again, the reason to reference patterns is to pick up the experience we use, that you'll get from understanding what people have done in the past with these type of systems. I highly recommend both the book and the website although the website is so complete, you may not really need the book. A lot of messaging tools out there. Again, we're going to talk about a Brokerbase tool, RabbitMQ, there are others. We're going to talk about specific messaging framework built on sockets called ZeroMQ and we're going to talk about the Amazons Simple Queue Service, SQS, which is another good approach to queuing data for other people to use. Again, lots of other messaging approaches out there. Messaging has been a part of software development for a long time, so things like MSMQ for Microsoft, the EAI, and the SQA architectures have been around for a while, Enterprise Service Bus. Java has its own message service built-in. These are all potential approaches to messaging, very much dependent on what you're trying to do with your system, but I think it's good to focus on some of the more generic message tools like Rabbit, like Zero that can be used in just about any application with just about any language. ZeroMQ is one of the simplest approaches you can try to put it in place. It is based essentially on IP sockets and it provides a nice API for sending atomic messages across various types of transport. It allows for pub-sub, fan-out, request-response messaging patterns. It's available for almost every programming language, operating system and hardware combination you can think of, and again, a very simple and easy to learn API to work with. The nice part about Zero is there's no brokers. There's nothing that you have to start and manage. It's just simply applications talking to each other in the sending messages. Again, if you're looking for a generic messaging platform, I think Zero is one to dig into. There's a "Hello World" version of this in Python, you can take a look at it. The library for ZeroMQ in Python is called zmq. Again, to put together a request reply pattern in Python is a relatively straightforward thing for you to look at. RabbitMQ claims to be the most widely deployed open source message broker. Again, Rabbit has this intermediate element that provides the ability to service multiple types protocols in queue data in various ways. It provides for the standard patterns, pub-sub request reply, etc. It does support TLS and LDAP for authentication and authorization, really a nice tool set to use. Again, at the basic level, learning Rabbit is pretty straightforward. Another example for you of a send/receive using the Pika library. Again, very easy to set up this communication in Python between someone producing events or messages and someone receiving them. Do take a look at Rabbit and Pika. The tool that's probably most important when you're working with AWS is SQS. SQS provides a fully managed message queuing service. It has two different kinds of queues: a Standard queue and a First-in, First-out queue. The reason to use the FIFO queues is to guarantee that messages are processed once in the order that they're set. For most things, the Standard queue probably will work for you, especially if you handle timestamping in your own message monitoring. It's a very easy tool to use, and most importantly, when you're working with the IoT tools and Lambda in AWS, having access to SQS is a great place to use for debugging because you can drop your incoming messages, you can make a copy of them and put them in an SQS queue, and you can go to the SQS console to go look at those messages and make sure that things are coming in to your system the way you want. Again, besides using it in your applications, SQS is a really great debugging tool to confirm that messages are moving through AWS the way that you would expect.
