The Mesh Potato

The Mesh Potato is an 802.11bg mesh router with a single FXS port (Figure 1). Adjacent Mesh Potatoes automatically form a peer-to-peer network, relaying telephone calls without landlines or cell-phone towers. The Mesh Potato hardware and software is open. The power, Ethernet and FXS ports are robust to developing-world conditions like static, lightning, bad power and accidental abuse. The Mesh Potato comes in a weatherproof box for outdoor mounting and costs about the same as any other Wi-Fi router (less than $100).

An analog phone connects to the Mesh Potato via the FXS port. FXS (Foreign eXchange Station) is a telephone interface that supplies power, dialtone and generates ringing voltage. When you make a phone call, your Mesh Potato talks to the potato down the street, which talks to the next potato, and eventually to the destination. The mesh network can be augmented via backbone links and connected to the rest of the world using VoIP trunks.

Figure 1. The Mesh Potato

This article describes the history of the Mesh Potato Project, including how it was conceived and its development so far. I also discuss the Mesh Potato's design and the technical challenges we have faced.

In June 2008, I attended the Village Telco workshop in Cape Town, South Africa. The Village Telco (and I quote) is an easy-to-use, scalable, standards-based, wireless, local, do-it-yourself telephone company toolkit. Put simply, the idea is that “some guy in a village” can build a local telephone network and make a sustainable business by charging a nominal fee for calls to the PSTN (Public Switched Telephone Network) via VoIP trunks. We were in Cape Town to work out how to build the Village Telco software and hardware.

Steve Song of the Shuttleworth Foundation pulled together a fascinating team of people from the development, VoIP, mesh networking and business communities. The team was small (about ten people) and very hands-on in its outlook and skill sets (Figure 2). The breakfast and dinner conversations were fascinating—funny stories about broken-down hotels in some developing countries and sad stories about the poverty of others.

Figure 2. Village Telco Workshop 2008. Top from left to right: Jason Hudson, Jeff Fletcher, Johann Hugo, Alberto Escudero-Pascual, Steve Song, Jeff Wishnie and Alan Levin. Bottom: David Rowe, Elektra, Rael Lissoos.

One of the outcomes was the decision to build a little box called the Mesh Potato. We started out thinking we would use off-the-shelf hardware, like wireless routers and ATAs. Suddenly, it dawned on us that we didn't have to accept non-optimal, off-the-shelf hardware. We had the skills to design and build exactly the hardware we needed for the project. We also chose to make the hardware design open, just like the software.

Since then, we have come a long way. Through a series of development projects funded by the Shuttleworth Foundation, we have designed, debugged and built about 20 Alpha Mesh Potatoes (Figure 3). The first phone calls over Mesh Wi-Fi were made in June 2009, almost exactly one year after the project kicked off. We currently are preparing for a Beta run of Mesh Potatoes, with full production scheduled for early 2010.

Figure 3. Prototype Mesh Potato (on the Right)

We keep hearing how popular mobile (cell) phones are in the developing world. I have seen how well a humble cell-phone works, penetrating to the corners of some really remote areas of the world. So why do we need a Wi-Fi-based system like the Village Telco?

The answer is simple. The call costs for mobile phones are very expensive for many people in the world. In many cases, it's roughly the same cost as a mobile call in a developed country. If you are earning $1/day, a 50-cent mobile call is very expensive (Figure 4).

Figure 4. If cell phones could talk!

Although mobile phones have delivered remarkable benefits to developing countries, the mobile oligopolies that have emerged in the process have kept call charges artificially high. Worse, mobile operators tend to function as “walled gardens” in order to entrench their market share. Just compare the price of an e-mail message on the Internet (zero) and via a cell phone (20 cents for a text message), and you get some idea of the problem.

Communities in the developing world need an alternative. Hence the need for the Village Telco—a system that uses commodity Wi-Fi technology and unlicensed spectrum to provide low-cost phone calls.

The Mesh Potato runs B.A.T.M.A.N. (see Resources) mesh routing software, Asterisk, the Speex voice codec and Oslec echo cancellation. No cell-phone towers, no landlines, no big Telcos are required. Local entrepreneurs can roll out their own Village Telco system using a modest server and a bunch of Mesh Potatoes—community-owned telephony.

The mesh network is self-organising and self-healing. If a node goes down, B.A.T.M.A.N. automatically re-routes the calls. We are building custom hardware specifically for developing communities using open hardware and software principles. I am intrigued by the idea of developing custom open hardware devices—no need to accept whatever is available off the shelf. Most of the value in any router-type product is delivered by the software, which these days is usually Linux. The idea of relying on closed, proprietary, not-quite-right hardware is obsolete.

The Mesh Potato is as open as we can make it. We have minimised binary blobs and deliberately chosen open over proprietary software. The Mesh Potato is Atheros-based, as this allowed the use of the MadWifi open-source WLAN driver. We use the Speex and GSM codecs instead of g729 and Oslec instead of a proprietary echo canceler. The hardware schematics are available on-line.

The Mesh Potato will be mass-produced in large numbers. Open projects like this will start to exert influence over future telephony systems. For example, if 1,000 Village Telco operators are trunking calls encoded in Speex, VoIP trunk operators will need to support Speex. This represents an important paradigm shift. The Open community now has a chance to set standards, rather than have to play along with “standards” based on closed hardware and software.

I have developed open hardware telephony products in the past, including the IP04, which is manufactured by Atcom (see Resources). So it was natural that we team with Atcom for the board-level PCB layout and volume manufacture of the Mesh Potato. Atcom is a VoIP hardware company from Shenzhen, China, that understands and embraces open hardware and open software. Atcom is handling the board-level PCB layout and volume manufacture of the Mesh Potato.

Figure 5 is a mud map of the Mesh Potato hardware. The Mesh Potato uses an Atheros AR2317 System-on-a-Chip (SoC), which is a very low-cost router chip that combines an MIPS processor running at about 200MHz with 802.11bg Wi-Fi. It has built-in interfaces for LEDs, SDRAM and serial Flash. Best of all, it is well supported by OpenWRT and MadWifi. The FXS hardware, drivers and other firmware we have developed are generic. It is possible to port them to other router architectures. In very high volumes, it would make sense to integrate the FXS chipset functionality onto the SoC.

Figure 5. Mesh Potato Hardware Architecture

Development of the Mesh Potato kicked off in September 2008. Along the way, we had a few design issues and many challenging bugs to fix. As part of the open design philosophy, we have documented the design and even some of the “bug hunts” on the Village Telco blog (see Resources).

A key question was CPU load. Could a humble router CPU support Asterisk, a speech codec, an echo canceller and route several other phone calls over the mesh at the same time? To answer this question, we designed a test with all of these software modules running at the same time. As this was in the early days, and we didn't have any FXS hardware, we simulated the speech samples coming from the FXS port.

To model the maximum load of the system, we thought about a worst-case scenario of one mesh node routing 15 phone calls for its peers. This means the node would have to receive, then re-transmit, voice packets for 15 simultaneous phone calls. At the same time, the node had a phone call of its own, which meant the speech codec, echo canceller and Asterisk were all running. To test this scenario, we set up some Asterisk boxes to generate calls and used commodity Atheros Wi-Fi hardware to run the prototype Mesh Potato firmware.

The test passed. Call quality was maintained, provided we used 80ms voice packets to reduce the overhead of many small VoIP packets.

The MadWifi driver had a nasty “stuck beacon” problem that was specific to ad hoc mode, which is required for mesh networking. Nodes attempt to adjust their internal clocks based on reception of beacons from other nodes. Under certain situations, this caused a race condition, which locked up the driver's transmit queue. This means the driver would stop working for about 30 seconds.

Elektra worked hard with the MadWifi developers to establish and test a workaround. The driver is started in access-point (rather than ad hoc) mode, and then we create a virtual ad hoc access point that does not transmit beacons:

$ wlanconfig ath0 create wlandev wifi0 wlanmode adhoc nosbeacon

Beacons are unnecessary for our mesh network, and B.A.T.M.A.N. broadcasts its own packets at regular intervals. In access-point mode, there is no attempt to adjust the MAC clock, so the race condition is avoided.

Low-cost Wi-Fi SoC devices are highly integrated. They have minimum hardware interfaces—just enough for their core purpose.

To support an FXS interface, we normally use a CPU that has some sort of Time Division Multiplex (TDM) bus to support one or more 64kb/s bit streams of speech samples. The TDM bus hardware then handles DMA of the speech samples, presenting them as buffers to the device driver.

Every chip I have worked on in 20 years of telephony hardware design had a TDM bus. The AR2317 has many other features we need (low cost, Wi-Fi, OpenWRT support), however, there's no TDM bus!

So, I worked out a scheme to send and receive speech samples via the RS-232 console port, using some external glue logic and a small microcontroller to buffer the speech samples. It's not elegant, but it's reasonably low cost and it works, allowing us to use a low-cost AR2317 SoC for a very different application (VoIP) than it was designed for.

As the speech samples arrive in the RS-232 port, they are buffered by the SoC serial FIFO. When the FIFO fills, it interrupts the CPU every 1ms. Now, 1ms is a very high interrupt rate. What this means is every 1ms, the CPU must stop what it's doing and run the Interrupt Service Routine (ISR). This can cause a big performance hit, as the instruction cache contents must be flushed and replaced every 1ms.

To minimise this performance hit, I rewrote the Linux drivers/serial/8250.c interrupt handler to be as small as possible (Listing 1). This is small enough to consume only a tiny amount of valuable instruction cache and is likely to stay resident in the cache between interrupts. The functions it “calls” are either macros or very short inline functions.

Being a small team on a modest budget, we have experienced difficulty obtaining all the data we require from the SoC vendor, Atheros. This is a problem with some chip vendors. Unless you can pay big up-front fees or have a one-million chip order pending, it is hard to get support or even basic data. This is a pity, as I feel many chip vendors rely increasingly on the Open Source community for their firmware and build tools and hence their chip sales. Also, we are trying to design a mass-market product that will sell more of their chips.

This is not true of all chip vendors. Analog Devices has been very helpful with the IP04-based open hardware embedded IP-PBX (see Resources) that uses the Analog Devices Blackfin CPU. The company provided comprehensive data, time from many support engineers and even funded some of the IP04's development.

One particular problem area with the AR2317 has been RF calibration. Each AR2317 chip is slightly different and must be calibrated on the production line using automated test equipment. The software to perform this calibration and even the calibration registers' documentation is very closed. This has made it hard to calibrate our prototypes to obtain optimum RF performance.

However, where there is a will, there is a way. With Atcom's help, we have teamed with other Wi-Fi router vendors who have the required calibration equipment on their production lines.

In early 2009, we started design of the actual custom Mesh Potato hardware. By the middle of 2009, we had built approximately 20 prototype Mesh Potatoes, and in July, we held the second Village Telco Workshop to test the alphas and plan the next phase of the project.

One nice demo was a simple Asterisk dialplan, placed on each Mesh Potato:

exten => _XX,1,Dial(SIP/4000@10.130.1.${EXTEN})

This allowed us to construct a serverless, peer-to-peer voice network. This dialplan takes a two-digit number XX and lets you dial another Mesh Potato with the IP of 10.130.1.XX. It's an instant local phone network on just a few watts/node, no server required (let alone a cell-phone tower or central office exchange). Simply switch on your potato, and you can make calls. Imagine the applications for Katrina-style disaster situations where you need instant communications.

We spent some time setting up mesh networks and testing the limits of the system by listening to voice quality. Using the B.A.T.M.A.N. debug modes, we could see the mesh hops go around corners and through windows to relay calls from one node to another.

We still have a lot to learn about everything that affects call quality. There are many factors, such as Wi-Fi propagation, antennas, speech coding, jitter buffers, interference and system load. We are planning a small R&D project to study and optimise call quality in marginal conditions.

We need effective ways to instruct people on how to set up a reliable mesh network (like a picture book or videos or real-time metrics of quality such as a GUI or dialtone).

Wandering around in the South African winter sunshine with a Mesh Potato and a battery, I had an “ah-ha” moment that frankly sent shivers down my spine. This thing really works! You sometimes lose track of the big picture when you are engineering all the details.

Our big goal now is to simplify the installation and configuration as much as possible. At the workshop, we spent some time trying to get a Mesh Potato connected to an Asterisk server, and it was the usual time-consuming Asterisk conf file and command-line frustration. It's hard the first time, but gets easier as you gain experience. However, we want to make Village Telco setup easy for thousands of first-time users. This experience drove the point home: we need to make configuration as straightforward as possible.

It has been a pleasure to work with the Shuttleworth Foundation, Elektra and Atcom on this project. We also have had amazing input from the participants in the two Village Telco Workshops and members of the Village Telco Google Group. And, we still have a lot to do. By early 2010, we plan to resolve the remaining calibration issues, perform Beta trials and obtain type approval for the Mesh Potato. At the higher levels of the Village Telco Project, we need to integrate a billing system and the Afrimesh GUI, and integrate into a simple one-click installation.

I am confident we will achieve this and more. We have shown that a small, talented team can develop custom Wi-Fi hardware specifically for their needs. Community-based product development for community-based telephony—how cool is that!