An Interview with Inder Singh

I first met Dr. Singh in New York City, while attending LinuxWorld. He was quite cordial, soft-spoken and informed. His passion for his work was apparent. We talked, over breakfast, about the future of embedded Linux, the release of BlueCat Linux 1.0 and Linux in general. Dr. Singh is the CEO and Chairman of LynuxWorks, a company focused on providing embedded applications to OEMs and beyond. I spoke with him in early July about the current and future state of the embedded Linux world.

LJ: What made you choose Linux? What tools/developments would like to see added to Linux?

Inder: At LynuxWorks (formerly Lynx Real-Time Systems) we have always been a strong proponent of open standards in the field of real-time and embedded systems. We started out with implementing a real-time operating system (RTOS) from the ground up, LynxOS, which was highly compatible with UNIX (just as Linux is today), and we have played a pioneering role in the POSIX standardization effort. We were the first to implement the POSIX real-time extensions and POSIX threads, which are widely implemented in most UNIX systems today, and LynxOS was the first non-UNIX system to obtain POSIX certification.

Now, Linux has emerged as what we all hoped UNIX would be, a POSIX-compliant open system, which is available from a large number of suppliers on a wide array of hardware platforms and rapidly gaining wide software support.

Thus, Linux represents a great opportunity for us and a natural evolution of our open systems strategy. LynxOS is already a POSIX compliant, hard real-time operating system that is scalable down to very small footprints (minimum code size 32 Kbytes). And LynxOS is in many ways closer to Linux than any other RTOS. As a POSIX certified RTOS, it is highly compatible with Linux at the source level.

Additionally, LynxOS uses the same development environment as Linux, since both are based on the GNU tool chain. And LynuxWorks customers have frequently used large amounts of application code from the Linux environment (as well as from other UNIX systems such as Solaris) and moved them to LynxOS. This generally involved little porting work beyond recompiling and building. Similarly, Linux device drivers are easily ported to run on LynxOS.

The LynxOS kernel provides a highly scalable, fully preemptible hard real-time kernel specifically designed for real-time embedded systems. However, much of the non-kernel part of the LynxOS distribution is based on open-source technology and is already very close to Linux. This includes the GNU C/C++ compiler tool chain, X-windows, TCP/IP networking software, a large number of BSDI utilities, Samba and Apache. LynuxWorks is already heavily involved in and very familiar with the open-source model.

So it was a natural step for us to be the first established RTOS vendor to embrace Linux. As a part of the Lynx Linux Initiative (L2I), which we announced last November, we introduced BlueCat, a version of Linux specifically tailored for embedded applications that builds on our experience and infrastructure developed over the last decade serving our embedded customers. At the same time, we decided to evolve our LynxOS technology to bring the manifold benefits of Linux to LynxOS users. Future releases of LynxOS will feature full binary compatibility with Linux. The objective of that move is to insure that a binary executable file that runs on Linux can be loaded and executed directly on a LynxOS system (on the same CPU architecture, of course) without requiring recompilation or porting.

LJ: Can you briefly talk about pros and cons of using Linux?

Inder: For the embedded arena, which is very fragmented and populated with proprietary OS solutions, Linux provides for the first time the potential of an open, multi-vendor platform with an exploding base of software and hardware support.

The Open Source community development model, with the Web playing a crucial role, has been key to both the quality and robustness of Linux, as well as its popularity, and this will continue to be a major strength. In the embedded market, the royalty free nature makes it very attractive for high volume, cost-sensitive applications such as many of the emerging “post PC” appliance type of products. The wide availability of source and a huge and growing web-based developer community facilitates quick availability of support for new semiconductor devices, including ports to new CPU architectures. This makes Linux an important vehicle for tracking the technology curve.

The growing population of Linux-literate programmers, consultants and service organizations is a big plus.

The Linux kernel itself was developed for general purpose use, and its ongoing development is likely to be driven by the needs of the server area, particularly upwards scalability in competition with Solaris and NT. Thus it will fall upon the embedded Linux vendors like LynuxWorks and others to provide distributions tailored for embedded use. These vendors are addressing areas like cross development environments, tools for embedded systems and support for a variety of hardware configurations specific to the embedded world. Such embedded versions of Linux will serve the needs of a significant subset of embedded applications. However, many embedded applications require hard real-time performance, or smaller footprints that have been the province of proprietary RTOSes. There is only so much that you can do with an implementation that was designed for a different purpose. LynuxWorks is extending the benefits of Linux to such applications by its unique approach of supporting the Linux APIs and ABIs with LynxOS, which has been developed from the ground up for such applications. One of the nicest things about open standards is that you can have multiple implementations from different vendors that meet the needs of different constituencies.

LJ: What tools/developments would like to see added to Linux?

Inder: A well defined hardware abstraction layer (HAL) would make Linux a lot easier to port to a variety of different CPU architectures and hardware platforms. This is particularly important for embedded systems, but it would be very helpful for the server area as well, as the world moves beyond the Wintel architecture. In LynxOS, we use what we call a CSP/BSP (chip support package/board support package) architecture which separates out the hardware dependent code related to the CPU, and the supporting peripherals at the board level, from the hardware independent code in the rest of the OS.

LJ: You recently changed your company name from Lynx to LynuxWorks. Why, and how do you explain the unique spelling?

Inder: The name change makes a clear statement about our commitment to Linux as our corporate strategic direction. It is kind of interesting that our previous name Lynx, which predates Linux, becomes Lynux with the addition of a “u”. The unique spelling with the “y” emphasizes our continuing commitment to our current LynxOS technology, which is an integral and important part of our strategy for bringing the benefits of Linux to the embedded world.

We have a taken a unique approach to embedded Linux by offering our customers a choice of two kernels: BlueCat, which is our open-source Linux distribution tailored for embedded applications; and LynxOS, our Linux-compatible, POSIX compliant world-class, real-time operating system (RTOS) technology with a track record of a decade in demanding, mission-critical applications.

The dual OS approach offers choice and flexibility to our customers and enables us to bring most of the benefits of Linux to a wide range of applications, including those traditionally served only by embedded RTOSs.

From the embedded perspective, you can look upon Linux in two ways:

The former has a lot to do with the where Linux is today, but it is the latter aspect of Linux as a platform for a growing base of open-source as well as commercial applications, in particular, that will make Linux have its greatest impact on the computer industry, and especially so in the embedded industry.

The key to Linux maintaining its momentum, and having the huge impact on the computer industry that many proponents are hoping for, lies in its achieving wide acceptance as an open standard platform for applications.

The fact that Linus Torvalds chose to implement the UNIX interfaces rather than simply create a new operating system, and that Linux substantially conforms to POSIX, has been largely responsible for its popularity, and it further strengthens its position as an open standard.

In the embedded world, this aspect of Linux is particularly significant, because this is such a fragmented industry without any kind of a standard or leading platform. Around half of all embedded designs still use in-house operating systems, and the leading commercial RTOS product accounts for only a fraction of new embedded designs. As a result, the embedded world is missing a robust software industry such as the software industry that developed around DOS and Windows for the desktop. Embedded developers do a lot of reinventing the wheel and developing much more software from scratch, instead of building on top of existing tools and application software like their mainstream counterparts.

Linux as an open standard has the potential to serve as the missing platform that can foster an embedded software industry. As embedded applications are becoming more complex, the boundary between embedded and mainstream applications is blurring. So more and more of the mainstream software becoming available on Linux is already useful for embedded developers.

LJ: Congratulations on being named to the directorial board of the recently formed Embedded Linux Consortium. What role do you see yourself playing within the Consortium? Will it be greater on the technical side, with regard to developing standards, etc., or on the marketing/promotional side, with regard to getting OEMs to consider embedded Linux as platform?

Inder: Thank you.

I strongly believe that Linux has an enormous and beneficial role to play in the embedded area. In fact, Linux will eventually have an even bigger play in embedded systems than in servers, and I believe that the Embedded Linux Consortium can play a very influential role.

The initial focus of the ELC is largely in the promotional, marketing and communications area, to create greater awareness of Linux's benefits and create market momentum, but it will undoubtedly evolve depending in large part on the desire of the membership.

It is very important for the embedded Linux community to avoid a rerun of the “UNIX Wars”. The ELC can play a key role in getting the embedded Linux community to rally behind standards activity such as the Linux Standards Base and POSIX, although the ELC is not planning to directly engage in the development of standards at this time.

LJ: What are the issues that embedded Linux systems providers need to agree on, given the fact that they are competing with each other also?

Inder: Avoiding fragmentation even as we compete with each other is really the most important issue, which I have covered in answer to your previous question on the subject. The key is to establish Linux as an open standard platform for applications to foster the growth of an embedded software industry around Linux. A broad agreement to build in the standardization work of the POSIX and LSB groups instead of creating new APIs or standards that favor a single vendor, and a recognition that it is in our mutual interest to support Linux as an open standard in preference to proprietary single-vendor solutions, even as we compete and add our own kinds of unique value, is something we should all work towards to avoid a replay of what happened to UNIX over the last decade.

It is also important for embedded Linux providers to be good members of the Open Source community, make meaningful contributions to the open code base and adhere to not only the letter but also the spirit of the open-source rules. I believe in this strongly, even though at LynuxWorks we provide both open-source and licensed products.

LJ: In regards to embedded Linux deployment: How promising is the handheld arena for embedded Linux—given the current preference for OSes like Windows CE/Windows Pocket PC? What are some of the other major areas for embedded Linux that LynuxWorks' has taken or plans to take advantage of?

Inder:In spite of the current dominant position of PalmOS, and the strong Microsoft push with Windows CE and PocketPC, I believe Linux can still establish a major position in the handheld arena. This whole field of “post PC appliances” is still in a state of flux, with a plethora of new Internet connected devices of many kinds about to hit the market over the next year or two. As an open system supported by many vendors, Linux is a very attractive candidate, and many large vendors that we are working with are in fact designing many next generation products around Linux. The low royalty costs (can't get much cheaper than zero!), freedom from dominance by a single vendor and the huge community of potential developers of add-on software makes Linux a very attractive choice.

In addition to the emerging applications in this post PC appliance arena, we are also addressing the automobile, point of sale terminals and wireless markets, as well as the broad Internet and communications infrastructure area, the aerospace and military industry and instrumentation with our Linux based thrust. In fact, I have been pleasantly surprised at the broad interest in Linux across all the markets that we have been addressing.

LJ: The standard C library for Linux is the GNU C library, or glibc, which is rather large. (libc-2.1.3.so on my box is almost 900K, and a statically linked “hello world” is more than 200K.) How can embedded Linux vendors keep embedded code small while at the same time maintain the advantages of a large pool of contributors and testers?

Inder:This is one of the areas in which embedded Linux vendors can add special value by providing facilities that offer more configurability and greater control over the “footprint”. As your question makes clear, this applies not just to the kernel, but compilers, libraries and other parts of the environment. In the case of libraries, the objective would be to include only the minimum amount of code required for a given system, and cutting down on loading any unneeded code.

LJ: Everyone's talking about preventing fragmentation, to the point where fragmentation-prevention strategies seem to be fragmenting. What's the One True Way to prevent fragmentation?

Inder: The success of an operating system is dependent on the applications support that it generates more than any other single factor. So I would say that the One True Way to prevent fragmentation is a strong, industry-wide commitment to ensure that there is a clear definition of what constitutes “a Linux application”, and that an application developer has the assurance that a single version of his or her application will run on any Linux system.

The Linux Standard Base (LSB) project, whose specific goal is “to develop and promote a set of standards that will increase compatibility among distributions and enable software applications to run on any compliant Linux system”, is intended to prevent this very problem. It deserves strong support from application developers and users.

LJ: There's a plethora of cheap boards, free developer tools and Internet discussions about embedded Linux. It's a virtual toy store. If I'm a Linux hobbyist looking to build a cool toy, like an embedded Internet stereo for my car or an autopilot for my model airplane, where do I start?

Inder: First I would look at what are the requirements of the entire system that I am trying to build. Then, I would select a basic hardware platform and a version of embedded Linux.

For a hobbyist application, I would recommend starting with an inexpensive Pentium PC motherboard, perhaps one of the many small variants with smaller form factors. For a commercial application, one would also explore other architectures, including system-on-chip (SOC) solutions which provide a variety of different adjunct processing functionalities (i.e., communications controllers, display controllers, etc.) that are being added onto the core chip and satisfy a variety of embedded system needs.

Next, continuing to focus on a commercial rather than a hobby application, I would find a Linux supplier that fully tests this processor technology with their embedded Linux Product. The available testing on the standard PC is very well supported but on other architectures is more “spotty.” I would also look at the long term stability of the vendor and if they are ISO 9001 quality process certified. Additionally, I would look at what the chances are that the vendor will be around for the next ten years, for support and additional upgrades to the Linux product. Most embedded systems have a much longer lifecycle than the lifecycle of a standard PC. Finding the right software vendor for a long term solution can often be most important to the success of the project. So, in some cases, choosing the right processor might come from the list of the supported platforms from your preferred software provider.

I would also use the Web as a tool to look for other pieces required for the solution, such as digital tuners and Linux drivers to support the required hardware components. This is one of the great benefits of using Linux—the Web based community and all of the useful hardware and software solutions, as well as advice, that are available.

LJ: TiVo printed a notice in their product manual stating that any customer who wanted a copy of their modified Linux kernel could get one by writing to a certain address. This is a great way for a company to both fulfill its obligations under the GPL and build a mailing list of hackers. Is this the plan other vendors of devices containing Linux will adopt?

Inder: This is a reasonable solution to complying with the GPL requirements for device manufacturers who use Linux as the OS for their embedded software. They only need to do this if they make modifications to the Linux kernel itself. If they simply use an embedded Linux product such as BlueCat Linux from a vendor like LynuxWorks and build their embedded application on it without making any changes to the kernel, which is the norm, then this is not an issue for them. It is the responsibility of the embedded Linux OS vendor to provide the sources to all the changes made to tailor Linux for embedded applications. We do this by including the sources in our distribution, and we plan to make the sources available on our web site as well.

LJ: The Linux kernel developers are struggling with the issue of devices that can be added and removed on the fly, like PCMCIA cards and USB devices. Do you think that the need to support these devices will make the kernel too complicated for embedded systems? And how responsive do you think the “mainstream kernel” is to the needs of embedded Linux developers?

Inder: You are going to see increased use of both USB and PCMCIA interfaces and devices in embedded systems. Supporting dynamic plugging and unplugging of such devices is essential, especially for embedded products where you should never have to reset or reboot a system. The important thing for embedded systems is to make sure that, as for many other facilities, this is configurable and you do not carry the code whether your need the facility or not.

The issue is broader than USB and PCMCIA. Dynamic reconfiguration of any embedded system is desirable whether it is for detecting devices on system start or being able to add or remove devices on the fly. This often allows system developers to have one core kernel (for system support and configuration control considerations), especially in embedded devices where the customers might need a minimal memory footprint. In this case, only the needed kernel drivers and applications are added to the system upon detection. In some cases, embedded developers are looking at having the required drivers on the piece of hardware that is inserted and, at device installation, the driver is extracted from the hardware and installed into the system. This level of dynamic reconfiguration increases the usefulness of the system as a whole.

LJ: Any plans to fund more open-source software that can be available to embedded Linux developers? Tools like media compression and decompression or voice and handwriting recognition would come in handy, for example.

Inder: We recently announced the contribution of our LynuxWorks Messenger technology for BlueCat Linux to the Open Source community to establish a new standard for advanced CompactPCI inter-board messaging in distributed and high availability (HA) systems. Implemented as a kernel extension and associated ApplicationProgramming Interface (API) for messaging, LynuxWorks. Messenger is a technology that enables CPU boards (system and non-system) to exchange information on a peer-to-peer basis across the CompactPCI backplane.

We have also contributed multi-threading extensions to the GDB debugger to the Open Source community.

We fully intend to continue to support and make significant contributions to the Open Software community.

LJ: Thank you very much for your time.