Low Power Wireless: CoAP

In my previous two articles (see the November and December 2016 issues of LJ), I described setting up a 6LoWPAN network and then integrating that into the internet so that the low power sensor/actuator nodes can talk to internet hosts and vice versa. This is one of the major mechanisms currently proposed for bringing the Internet of Things (IoT) to life.

Once you have established a communications pathway, however, you need to look at how you are going to use that pathway to exchange information—specifically, the protocols and the data types. The currently favored protocols are MQTT (MQ Telemetry Transport) and CoAP (Constrained Application Protocol), and they each fill different roles. MQTT is a messaging system using publish/subscribe, which has been adapted for low power devices. CoAP is similar to, and based on, HTTP but is heavily optimized for low power devices. This article focuses on CoAP.

The World Wide Web is built on the HTTP protocol. This is a traditional client/server model, where clients connect to a server over TCP and make requests of the server, which in turn prepares replies and delivers them to the client. The outstanding success of the Web has led to this being used as the model for CoAP, with the following appropriate changes:

REST (REpresentational State Transfer) is the philosophy behind HTTP, described by Roy Fielding (the principal HTTP 1.1 architect) in his PhD thesis. In a horribly emasculated form, he says that 1) resources are identified by URIs, such as Web URLs, and 2) resources are accessed using only four verbs: GET, PUT, POST and DELETE, with defined meanings (although when to use PUT and when to use POST is still debated). These definitions are:

CoAP will most likely be run as a server on sensors and actuators. These won't be highly endowed with RAM, and they are actually unlikely to be able to run Python, which takes megabytes of RAM. Even micro-Python takes about 180kB of RAM. Most likely, they will run compiled code, using a library such as the libcoap C library (https://libcoap.net).

I'm running these examples on Raspberry Pis, so I'm using Python for simplicity. There are many Python packages for CoAP and many implementations of CoAP for other languages (see coap.technology/impls.html). The Ubuntu x86 repositories have the aiocoap Python package, so you can install that on your desktop with:

sudo apt-get install python3-aiocoap

The RPi repositories currently have no CoAP packages, so you will have to install something (again!). You need to get the CoAP package on the sensor RPi. Download it from aiocoap, the Python CoAP library (https://github.com/chrysn/aiocoap). It contains the Python libraries in the aiocoap directory as Python code. You can move that directory to, say, /usr/lib/python3.4, so that it can be found from any Python 3 program:

git clone --depth=1 https://github.com/chrysn/aiocoap.git
cd aiocoap/
sudo mv aiocoap /usr/lib/python3.4

The package also contains clientGET.py, clientPUT.py and server.py. These not only demonstrate the CoAP package, but they also test some features. (I'll adapt these to our purpose here.)

I'm going to use the CPU temperature example from my previous two articles, as it is about as simple as one can get.

The sensor has to be exposed as a resource—that is, have a URI (here a URL). This will use the scheme coap:// or coaps://, its IPv6 address and its URI path, such as temperature. Note that the sensor will be running as a server—the client will be making queries to the server.

To the client, the URL will look like this:

coap://[fd28::2]/temperature

using the global IPv6 address you set on the “sensor” RPi in the previous article. The IPv6 address needs to be in square brackets ([...]) to avoid the colons (:) being confused with a URL Port option. The default UDP port is 5683.

The aiocoap package uses the recently added yield from Python generator system. I won't go into that here (it is non-trivial). The major parts to note are what you configure in the client and server.

The client needs to set the method as GET to fetch the CPU temperature of the server, using the server's URI. Then it reads a response and does something to it. Here you just print the response. The client is:

#!/usr/bin/env python3

import asyncio

from aiocoap import *

@asyncio.coroutine
def main():
    protocol = yield from Context.create_client_context()

    request = Message(code=GET)
    request.set_request_uri('coap://[fd28::2]/temperature')

    try:
        response = yield from protocol.request(request).response
    except Exception as e:
        print('Failed to fetch resource:')
        print(e)
    else:
        print('Result: %s\n%r'%(response.code,
                                response.payload.decode('utf-8')))

if __name__ == "__main__":
    asyncio.get_event_loop().run_until_complete(main())

The server uses the asynchronous I/O package asyncio. Again, you can ignore the details of this. The important thing is to add resources that can be accessed by CoAP user agents (clients). You add a new resource with:

root.add_resource(('temperature',), TemperatureResource())

which sets the URI-Path (/temperature) of the resource on this host and a class (TemperatureResource) to be invoked when the resource is requested. Any number of resources can be added, such as pressure, motion and so on, each with their own class handler.

The handling class is the most complex, and there are many possibilities. The simplest will subclass from aiocoap.resource.Resource and will have a method render_get, which is called when a GET for a representation of the resource is needed. For the example sensor, this gets the CPU temperature as before and then wraps it into an aiocoap.Message. Here's the server code:

#!/usr/bin/env python3

import asyncio
import aiocoap.resource as resource
import aiocoap

from subprocess import PIPE, Popen

class TemperatureResource(resource.Resource):
    def __init__(self):
        super(TemperatureResource, self).__init__()

    @asyncio.coroutine
    def render_get(self, request):
        process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE)
        output, _error = process.communicate()
        return aiocoap.Message(code=aiocoap.CONTENT, payload=output)

def main():
    # Resource tree creation
    root = resource.Site()
    
    root.add_resource(('temperature',), TemperatureResource())
 
    asyncio.async(aiocoap.Context.create_server_context(root))

    asyncio.get_event_loop().run_forever()

if __name__ == "__main__":
    main()

The output from running the client against this server is similar to this:

Result: 2.05 Content
"temp=36.9'C\n"

as in previous examples.

What I've basically done at this point is hack up an example to show how CoAP works, but the IoT isn't going to succeed if programmers act like that. My sensor will need to work in your environment, talking to other people's systems. The IoT isn't going to be a simple monolithic environment. It's going to be a mess of multiple systems trying to talk to each other.

Standards and conventions will need to be agreed upon, and not just between people, but in ways that can be read and confirmed by machines. I've used CoAP over 6LoWPAN, and that is just one battle that is raging. The next one is over data formats and device descriptions—both using them and discovering them.

HTTP has mechanisms to query and to specify data formats. For HTTP, this is managed by Content Negotiation, and this idea is carried across into CoAP: a client can request particular data formats, while the server may have preferred and default formats.

XML is generally regarded as too heavyweight. JSON is better, but as a text format, it still carries baggage. CBOR (Concise Binary Object Representation) is an IETF RFC for a binary encoding of JSON and is becoming popular. It has an advantage of being self-contained, unlike other recent binary formats, such as Google's Protocol Buffers, which require an external specification of the data.

A JSON format of the sensor data could look like this:

{
  "temperature" : 37.9,
  "units" : 'C'
}

CBOR translates this into a binary format, which may be more concise.

To use CBOR, first you need to install it. Python packages normally are installed using pip, and the RPi does not come with this installed. So install both it and the cbor module (note that you want the Python 3 versions):

sudo apt-get install python3-pip
sudo pip3 install cbor
sudo pip3 install LinkHeader

Then, a JSON equivalent data type can be encoded using cbor.dumps, which creates a byte array and is decoded by cbor.loads, which turns it back into a Python type. A Python dictionary is equivalent to the JSON of a JavaScript class object given above.

The server is modified by code to create a Python dictionary and then turn it into CBOR. The client is likewise modified to decode the CBOR data into a Python dictionary. You also will do some elementary content specification, using IANA-registered numbers. The application/cbor number is 60, from the IETF RFC 7049.

The relevant part of the server is this:

CONTENT_FORMAT_CBOR = 60

class TemperatureResource(resource.Resource):
    def __init__(self):
        super(TemperatureResource, self).__init__()

    @asyncio.coroutine
    def render_get(self, request):
        process = Popen(['vcgencmd', 'measure_temp'], stdout=PIPE)
        output, _error = process.communicate()
        list = re.split("[='\n]", output.decode('utf-8'))
        dict = {'temperature' : float(list[1]), 'unit' : list[2]}
        mesg = aiocoap.Message(code=aiocoap.CONTENT,
                               payload=cbor.dumps(dict))
        mesg.opt.content_format = CONTENT_FORMAT_CBOR
        return mesg

And, here's the relevant part of the client:

request = Message(code=GET)
request.set_request_uri('coap://[fd28::2]/temperature')

try:
    response = yield from protocol.request(request).response
except Exception as e:
    print('Failed to fetch resource:')
    print(e)
else:
    if response.opt.content_format == CONTENT_FORMAT_CBOR:
        print('Result: %s\n%r'%(response.code,
                                cbor.loads(response.payload)))
    else:
        print('Unknown format')

This prints something like this:

Result: 2.05 Content
{'temperature': 37.4, 'unit': 'C'}

The code above is fine for interacting with a temperature sensor—once you know what that is! You may have hundreds of sensors of different types, and all you may know is their IPv6 address. To complete this, you need to know the following:

At the moment, there are no industry-agreed-upon answers to those questions. One could say that (unfortunately) this is another of the differentiators in the IoT world. The IETF in RFC 7252 and RFC 6690 has made some progress, but there are still open issues, and they are not uniformly adopted.

From RFC 6690, each device should have a URI-path of /.well-known/core, which can be accessed by an HTTP GET coap://<IPv6-addr>/.well-known/core request. RFC 6690 specifies that the representation must be in CoRE Link Format, which I will describe soon.

Two new link attributes are added to the standard Web link headers of RFC 5988, such as title. The new attributes are the following:

The values of these attributes can be strings, URLs or anything—this isn't specified. The resource type is expected to be some “well known” value that identifies the type of device, such as jan.newmarch:temperature-sensor. Yes, I just made that up—there are several proposals but no standards yet.

The value of if is supposed to be some specification of the REST interface for the device—that is, how to call it using GET, PUT and so on, and what is returned from those calls. How an interface is described isn't specified by RFC 6690. Although possibly using WADL (Web Application Description Language) is suggested, the Open Connectivity Foundation uses RAML (RESTful API Modeling Language), and the Wikipedia page on RESTful APIs lists a dozen more, probably used by some group or other.

Investigating REST API languages is beyond the scope of this article, so let's just assume the well known core resource has a value like this:

</temperature>;rt="jan.newmarch:temperature-sensor";
               if="https://jan.newmarch.name/temperature-sensor"

Here /temperature is the relative URL of the resource, the value of rt is the “well known” device type, and the value of if is the description of the device. Assume that https://jan.newmarch.name/temperature-sensor contains WADL or RAML or some other description that allows you to deduce that requesting the resource /temperature using GET will return a CBOR object with fields temperature and unit, with float and string values, respectively.

The format of the well-known resource is defined to be in application/link-format, which according to the IANA CoAP Content-Formats site (https://www.iana.org/assignments/core-parameters/core-parameters.xhtml#content-formats) has CoAP code 40. The format is actually just UTF-8.

The server is modified by adding another resource:

root.add_resource(('.well-known', 'core'), WKCResource(root))

where WKCResource is a class in the aiocoap module, which keeps a list of all the resources supplied by this device.

When the client GETs the resource /.well-known/core, it will get a comma-separated list like this:

</.well-known/core>; ct=40,
</temperature>;
    if="https://jan.newmarch/temperature-sensor";
    rt="jan.newmarch.name:temperature-sensor"

For each resource, a client should extract the rt value. If it recognizes it as a temperature device, then it should carry on. If it doesn't, it should look up the if URL and extract what the GET method can do, and then carry on. That code is not covered here. The CoRE Link Format string can be parsed using the Python LinkHeader package.

This series has addressed the issues of setting up a 6LoWPAN low power wireless network, using the OpenLabs radios on Raspberry Pis, followed by bringing these devices into internet visibility. This concluding article looks at data formats and protocols for the IoT.

Many topics have been omitted. The major one is that of security, as the system I have described here is wide open to snooping and hacking. The security mechanisms are all there, but they are a full topic in their own right.

I also have ignored the issue of how external clients find the IP addresses of the clients. This is answered by internet draft “CoRE Resource Directory draft-ietf-core-resource-directory” (https://tools.ietf.org/html/draft-ietf-core-resource-directory-09).

I haven't addressed networking within a 6LoWPAN network. There are a variety of models, such as mesh networking, and they build on the IEEE802.15.4 networking model.

Finally, I haven't mentioned other pieces of hardware dealing with IEEE802.15.4 and 6LoWPAN. These include modules from Texas Instruments, Firefly and Libelium, with many others coming along.