UpFront

diff -u: What's New in Kernel Development

Zack Brown

Issue #171, July 2008

Linus Torvalds has called on all Cogito users to switch over to using git natively. When Linus first created git, he envisioned a tool that would provide a clean set of very low-level data-tracking actions. There was no point, he felt, in writing a bunch of complex, high-level revision control features, especially considering his existing penchant for system internals. The git tool was to be the “system call” layer, on top of which people could script their own favorite feature set. Cogito was the first attempt at such a wrapper, and for a while, it was the only way for people to use git without having to have a deep understanding of git's data-tracking concepts. But nowadays, git has added its own set of porcelain commands on top of the deeper plumbing. People still can use the lower-level commands to script their favorite front-end interface, but git's native high-level interface is likely to stay more up to date than any of those scripts, because it is so integrated into the original tool. Cogito itself, as Linus points out, has not been maintained for more than a year, and he says it's time for everyone just to switch over. This also will, he points out, have the added benefit that any problems uncovered by users will be debugged and fixed more easily, as the developers won't have to worry about which tool might have the bug.

Kernel development is always much messier than application development. When an application crashes, you can start to debug the core file or restart the application immediately for another test. When the kernel crashes, you typically have to reboot, which can be time-consuming. Kernel developers always are looking for ways to restart a test quickly or to identify potential bugs without actually having to experience the bad results of running into them. One technique that's been available for a while is to run the kernel as a user application itself on an already-running system, so that if the userland instance crashes, the rest of the system remains intact. That is so cool! Another attempt has recently come from Thomas Gleixner. His concept is to insert a whole new layer of debugging code in the kernel (it would be disabled for folks who just want to use their system) to track what happens to RAM after it has been allocated for clusters of variables. If the data suddenly changes when it wasn't supposed to, Thomas' code sees this as a big red flag and logs it. So, instead of discovering a bug because the entire system locks up, developers now, in many cases, can discover bugs before any symptoms affect the user experience. It's possible that Thomas will merge his code with something Chris Mason did a while ago. In Chris' project, a background thread would allocate memory, mark it with data, and then check periodically to see whether it had been corrupted. Thomas' and Chris' approaches are different and complementary in a couple different ways. For one thing, Thomas' code interacts with actual kernel objects that are currently in use, while Chris' checks an unrelated block of memory. Also, Thomas' code performs its checks after specific kernel operations, while Chris' does its checks after planned intervals of time. Both are good techniques, and it's likely that Thomas' code will tend to gather additional features over time.

Some kernel folks are considering alternatives to the MAINTAINERS file. Krzysztof Halasa has suggested the file is no longer necessary and should be replaced by formatted comments in the code itself. This way, people interested in finding out whom to talk to about a particular piece of the kernel would be able to find what they needed right in the part of the kernel in which they were interested. Jan Engelhardt is in favor of this, but added the suggestion that instead of comments, maintainer information should be embedded in the source code itself. That way, compiled into the kernel binary, it could be read via a simple command by anyone who was interested. But, Krzysztof points out that the only people who are going to be interested in this data are people hacking on the kernel, so there's no need to bloat the kernel binary with the information. Regardless, it does seem as though there is some support for reconsidering how maintainership information is handled. But, it may be that one of the MAINTAINERS file's greatest values is how visible it is. It's just fun to poke around in it! So undoubtedly, a wide range of issues will be hashed and rehashed by everyone before a decision is made.

It's probably safe to say that the 2.2 kernel is in the utter deep dark icy freeze of death. The latest release candidate for 2.2.27 is from January 2005, and efforts to continue to patch it have been met with resistance. Willy Tarreau points out that any release, even a release candidate, might be interpreted by users as an indication that the 2.2 kernel was being maintained actively—an impression he did not feel corresponded to the truth. For one thing, he says that there are known security fixes that also have not been included in any of the patches. He feels the 2.2 kernel is just too out of date to bring back. But Xose Vazquez Perez feels that if the 2.2 kernel is going to be that deeply frozen, it shouldn't be listed on the front page of kernel.org as if it were a living kernel. If it's really dead, let it die, Xose argues. And, if it's not really dead, let it be updated. But, it may be that the 2.2 kernel still should be recognized just for historical value, even if it won't be developed anymore.

Apparently, 802.11 is complex—like, more than usual for the kernel. It turns out that wireless devices are regulated differently everywhere, so hardware manufacturers have started producing just a generic device, relying on the software to implement the regulations—ouch. Luis R. Rodriguez recently suggested constructing a massive database to keep track of the large and growing variety of legal restrictions the kernel will have to take account of in order to implement 802.11 properly. And, where would all this data be stored? In the kernel, naturally! Luis at first suggested an external Web site, something interactive that the whole community could participate in maintaining. But, as Bruno Randolf said, this would make the kernel sources, as far as 802.11 was concerned, dependent on this external source, while the kernel itself should be the true source, he said. So, the kernel probably will grow some very complex 802.11 legal information in the near future.

Man vs. Myth: Greg K-H and the Kernel Driver Project

Doc Searls

Issue #171, July 2008

Don't tell Greg Kroah-Hartman that Linux hurts for device drivers. He's heard too much of that rap, and he's already done plenty to stop it. We should thank him and help pick up the ball. I'm doing both here.

The beginning of the end of the Missing Drivers Myth came at the 2006 Ottowa Linux Symposium, where Greg said, “Linux supports more different types of devices than any other operating system ever has in the history of computing.”

Still, the OSDL (later the Linux Foundation) board—composed mostly of large vendors—listed device drivers as the #2 “most pressing issue”. So the Linux Driver Project (LDP) was created. Alas, Greg reports on his blog, “No vendors showed up.” But after he announced, “Tell me all of the hardware that you know of that is not supported by Linux!”, he writes, “the response from users was overwhelming”. Thus, a canonical wiki list was created at the LDP.

After this, Greg went to each vendor personally, and the conversation almost always went like this:

GREG: “What hardware do you ship that is not currently supported by Linux?”

VENDOR: “It all is.”

GREG: “But wait, why are you claiming that 'Linux drivers' is your second most pressing issue today with Linux?”

VENDOR: “I don't know.”

Thanks to those clues, missing drivers is out of the board members' top ten pressing issues.

But, there always is work to be done. As Greg puts it, that work falls into four categories of user complaints. Here they are, with excerpts of Greg's responses to each:

  1. Printer and scanner support: “...already being handled very well by the Linux Printing Project and the SANE Project. Printer and scanner drivers in Linux are user-space programs and libraries and have nothing to do with the kernel at all. If you have any issues with these types of devices, please go ask the developers of those projects about them.”

  2. Older devices no longer manufactured that people really want to see working on their Linux machines someday: “...is hard. It would be great for Linux to support all of these older devices, but without the specs for the device, or in many cases, a company that is still in business, Linux support is going to be very difficult to achieve....Luckily, for almost all modern hardware devices, it is not necessary.”

  3. Wireless device support: “the Linux-Wireless group of developers has done an amazing amount of work in the past year, adding a whole new wireless protocol stack to the Linux kernel, as well as numerous different hardware drivers, some initially created by vendors and others created by reverse-engineering the hardware with no vendor help or approval. The latest kernel.org releases contain a raft of new hardware support for wireless drivers, and the number of active drivers in the queue to be added in the near future is quite large. Alas, there are still some wireless vendors that do not provide Linux support directly. Two of these, Atheros and Broadcom, have drivers created by the community through reverse-engineering efforts....Hopefully, this will change in the future.”

  4. Video input device support: “...there is an active Linux developer community in this area, but it seems to be hampered by a different development model...and a lack of full-time developers, not to mention a high degree of interpersonal conflicts that seem quite strange to outsiders. Support for a large majority of these devices is slowly trickling into the main kernel tree—the most important being the USB video class driver, which will support almost all new USB video devices in the future, thereby removing the major problem most users will face when purchasing a new video device.”

In addition to further education, Greg has opened development by keeping all code related to the LDP in a quilt patch series that is automatically included in the linux-next-daily kernel releases, which are then contained in a git tree that “provides a place where developers can provide changes, updates and see where they can help out if they wish to do so in a much easier manner. It also provides a way for companies participating to observe the status of their code in a much more open manner.”

It would be nice if Atheros and Broadcom were among those companies.

For more, visit linuxdriverproject.org.

Greg's blog post is at www.kroah.com/log/linux/linux_driver_project_status-2008-04.html.

LJ Index, July 2008

1. Towns in Vermont that voted unanimously for Internet access: 10

2. Lowest percentage voting to approve Internet access: 80

3. Fiscal 2007 McKesson revenues in billions of dollars: 93

4. Fiscal 2007 McKesson software sales in billions of dollars: 1.9

5. Latest annual growth in McKesson software sales, in millions of dollars: 300

6. Number of health-care services applications offered by McKesson: 70

7. Number of McKesson applications already running on Linux: 50

8. Percentage of new McKesson applications planned to run on Linux: 100

9. Number of remaining applications McKesson plans to have running on Linux: 20

10. Top percentage of expected of Linux-based capital expense budget savings for health-care providers: 70

11. Percentage decrease in downtime for Novell SUSE Linux between Yankee Group's 2007 and 2007–2008 Global Server Operating Reliability Survey: 73

12. Percentage decrease in downtime for Red Hat Enterprise Linux between Yankee Group's 2007 and 2007–2008 Global Server Operating Reliability Survey: 73

13. Percentage decrease in downtime for Debian Linux between Yankee Group's 2007 and 2007–2008 Global Server Operating Reliability Survey: 25

14. Percentage increase in downtime for Windows Server 2003 between Yankee Group's 2007 and 2007–2008 Global Server Operating Reliability Survey: 25

15. Percentage of Yankee Group 2007–2008 survey respondents running at least one RHEL server: 26

16. Percentage of Yankee Group 2007–2008 survey respondents running at least one Novell SUSE server: 17

17. Percentage of Yankee Group 2007–2008 survey respondents running at least one Debian server: 24

18. Percentage of Yankee Group 2007–2008 survey respondents running at least one Ubuntu server: 22

19. Size in billions of dollars of the logistics and manufacturing applications market on Linux, by 2011: 1.2

20. Size in billions of dollars of the human capital management market on Linux, by 2011: 2

1, 2: Tim Nulty

3–10: InformationWeek

11–18: Yankee Group and the Institute for Advanced Professional Studies (IAPS)

19, 20: IDC analyst Al Gillen, in InformationWeek

Kernel Candy

Doc Searls

Issue #171, July 2008

In April 2008 (at the time of this writing), the Linux Foundation published a progress report with the plain-wrapper title “Linux Kernel Development”, authored by Greg Kroah-Hartman, Jonathan Corbet and Amanda McPherson. Relying heavily on Jonathan's gitdb and other tools, they probed the kernel.org Web site and the git kernel depository and organized the results into a neatly arranged assortment of tasty nuggets. Here are a few of them:

  • For each day during the past 2.5 years, 3,621 lines were added, 1,550 lines removed and 1,425 lines changed. “That rate of change is larger than any other public software project of any size.”

  • Fifteen percent of kernel code contributions during the past three years have come from the top ten individual developers. Thirty percent has come from the top 30 developers.

  • The top five individual developers, in order, were Al Viro, David S. Miller, Adrian Bunk, Raif Baechle and Andrew Morton. Each contributed more than a thousand changes.

  • More than 70% of kernel development is being done by contributors being paid for their work.

  • The size of the individual development community has doubled in the last three years.

  • Of 31 listed corporate sources of kernel code changes (that is, companies employing individual developers contributing changes), the top two, with 13.9% and 12.9%, respectively, were None and Unknown. These were followed in order by Red Hat (11.2%), Novell (8.9%), IBM (8.3%) and Intel (4.1%). “The numbers presented are necessarily approximate”, the report says.

  • Companies outside the IT business contribute too. For example, “the 2.6.25 kernel will include an implementation of the PF_CAN network protocol, which was contributed by Volkswagen. PF_CAN allows for reliable communications between components in an interference-prone environment—such as that found in an automobile.”

The concluding line is a model of hedged understatement: “With the current expansion of Linux in the server, desktop and embedded markets, it's reasonable to expect this number of contributing companies—and individual developers—will continue to increase.”

Source: https://www.linux-foundation.org/publications/linuxkerneldevelopment.php.

What Are They Using? Nicco Mele: a Man and His Cave

Doc Searls

Issue #171, July 2008

Nicco Mele was born in West Africa, has lived all over the world, speaks many languages, and produces, among many other things, the Junglecast podcasts.

He also is probably the only human—and certainly the only geek—who has worked for a full house of presidential candidates: Howard Dean, Barack Obama, Hillary Clinton and John McCain. I first met Nicco in the server room of the Dean campaign in Burlington, Vermont, in the winter of 2004. The servers were running Linux. Geeks with laptops were flopped all over the place, hacking in Drupal and otherwise pioneering the political hackery now being leveraged in countless campaigns and political Web sites and services. Nicco ran that show.

Now he runs EchoDitto, an Internet strategy and consulting company with offices in New York, Washington, DC and Cambridge, Massachusetts, where we are neighbors and get to hang out. “At work, we rely heavily on customized Drupal and WordPress setups”, he says. But what he'd rather talk about is his home tech life:

I'm a believer in open source, well beyond just technology. For example, I strive for all of our home electronics to be open source wherever possible.

This begins with the Man Cave. When my wife and I, as newlyweds, bought our first home about 9 months ago, my wife—terrified of the proliferation of cables around our old apartment—offered me the basement for all of my technological needs. And lo! The Man Cave was born. With a 110" screen, a hi-def projector and surround sound—and complete darkness in the bright noon of day, the Man Cave is ideal for consuming Battlestar Galactica. Managing all this with open source? Enter my MythTV box, recently upgraded thanks to the wonders of—Woot!—Core 2 Quad Q6600 processor with four cores on one chip. Two 320GB 7200 RPM SATA drives for 640 gigs. Add some free software, and I've got a DVR and media server that is so crazy exciting, it's not funny. Now, if only I could get a voice-activated remote: “Computer, increase volume!”

For more, start with nicco.org, and follow links from there. There are many rich veins to mine.

Olympic MIDs

Doc Searls

Issue #171, July 2008

From a Linux perspective, the most remarkable location at January's CES (Consumer Electronics Show) was Intel's Mobility booth. It was so crowded, I could barely elbow my way to the display cases. When I asked to see gear that ran Linux, I was told Linux was running on most of the gear they were showing there.

And sure enough, it was. I got to see and handle MIDs, or Mobile Internet Devices, from Lenovo, Clarion, Aigo, Samsung and Digifriends. Most appeared to be running China's Red Flag Linux and the CoolFox browser—obviously, a Mozilla derivative. The UIs were modeled on the iPhone, with square application icons and a “coverflow”-like file browser that let you fan horizontally or vertically through choices.

Since then, it has become clear that Intel and its OEMs are pushing to have some of these products ready for the Summer Olympics in Beijing. It should be interesting to see not only how much reporting in the wild is made possible by these new Linux handhelds, but also how many cool new hacks on them will be encouraged by the setting.

Of particular interest to me is the Lenovo Ideapad U8 Mobile Internet Device, for two reasons.

First, Lenovo is already the first major PC hardware OEM to market Linux aggressively. Its US index page (lenovo.com/us/en) at the time of this writing (in April 2008) says, “ThinkPad notebooks with Linux” right up front, rather than buried somewhere down the directory tree.

Second, the Ideapad U8 is an interesting device. It lacks the slide-out keyboard of the Nokia N810 (the Linux pioneer in the category), but it has a virtual QWERTY keyboard that can pop up along the bottom of the screen—plus a numeric hardware keypad on the right of the screen for T9 predictive text entry. In addition to the touchscreen (for finger or stylus), it has a hardware pointer reminiscent of the familiar red ThinkPad nub, but with no moving parts. And it has a bunch of other features you'd expect in a Linux handheld PC.

The Ideapad U8 was vetted at the Spring 2008 Intel Developer Forum in Shanghai, where Intel also shared details of the Menlow platform on which the device (along with many others) is based. Menlow puts the new Silverthorne CPU on the single-chip Poulsbo chipset. Significantly, at least one of the units being shown came with a colorful back badged with Olympics imagery.

An open question at this point is hackability. Nokia has openly courted and supported individual developers through the Maemo development platform. Nokia's purchase of Trolltech earlier this year also says it's serious about Linux development on mobile devices. Will Menlow-based MIDs veer in the “applianced” direction (like the iPhone) or the “generative” direction (like the Nokia MIDs)? For more about the difference, read “A Tale of Two Futures” in this month's EOF on page 96.

From what I gather by talking to Intel folks, these new MIDs are more likely to veer generative, on the model of the ASUS Eee PC, which ships with Xandros jiggered for “consumer” use, but which also remains completely hackable. We look forward to reports from Linux Journal readers as these new MIDs flow into the market starting this summer. May the best ones win.

They Said It

Doc Searls

Issue #171, July 2008

Where your talents and the needs of the world cross, there lies your vocation.

All paid jobs absorb and degrade the mind.

Thinking of and delivering IT as a service allows IT to become part of the business, and not merely the dumb bits behind it. Open source and SaaS make it all happen. Savvy IT shops will invest in both.

Windows Is Collapsing: How What Comes Next Will Improve.

—Title of a Gartner presentation by Michael Silver and Neil MacDonald, www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9076698

If you know Linux, you're going to know we sell Dell products with Linux on them.

Our hospitals aren't ready yet for Linux on the desktop, but it's coming....If you look at the total costs of hospitals and the pressure on hospitals to continue to lower their costs, it's coming.

—Michael Simpson of McKesson Provider Technologies, www.computerworld.com/action/article.do?command=viewArticleBasic&articleId=9052142

Come to LinuxJournal.com and Stay a While

Katherine Druckman

Issue #171, July 2008

LinuxJournal.com is a pretty great on-line community. We now have a dedicated irc channel at #linuxjournal on irc.freenode.com (or at www.linuxjournal.com/irc) where the Linux Journal staff hangs out with readers and good conversations ensue. You might find me in there, so come say hi. You also are likely to find many of our regular authors and other staff members, so if you ever want to chat with us, we welcome you to drop in.

We also encourage you to drop by LinuxJournal.com and stay a while. You are very likely to find multiple articles that interest you. Be sure to check out the related links under each article to find other stories of interest. Search the site, or browse by topic, and you'll very likely find what you are looking for. With tens of thousands of articles at LinuxJournal.com, if it's about Linux and open source, we probably have it covered.