UpFront

diff -u: What's New in Kernel Development

Zack Brown

Issue #180, April 2009

The kernel boot process is one of the strangest parts of the whole operating system. One aspect of it is initramfs, where the kernel must first boot up in a RAM-based root filesystem, before it can mount the disk-based filesystems and transition the kernel to consider those the root filesystem. initramfs traditionally has not been part of the official kernel, so each Linux distribution has come to maintain its own unique implementation. In general, this means the sequence of events in the boot process of each distribution is different. This can affect the way hardware is initialized and the state it's in when the user finally sees a login prompt.

Jeremy Katz recently announced Dracut, a new distribution-independent implementation of initramfs, inspired by discussions at the most recent Kernel Summit. The project's goal is to simplify and standardize the boot process and move the initramfs implementation into the official kernel tree. Each Linux distribution then would boot up identically, but would be able to layer any additional functionality on top of the main initramfs implementation.

Jeremy's work is based on the Red Hat initramfs. Hannes Reinecke is working on his own implementation, based on OpenSUSE. It's likely the two projects will merge at some point. The project will need to bear several issues in mind. As Neil Horman explained, some systems elect to stay in the RAM-based filesystem for stability reasons and never transition to a disk-based root filesystem. Any successful initramfs would have to support that feature, in order to work with various embedded system projects.

Keeping track of what goes into the kernel is always pretty tough. Each new version could contain thousands of patches. There are various ways to track them all, including using git or one of the git repository browsers. John Hawley recently set up a new method at patchwork.kernel.org. This page keeps track of all patches sent to a variety of kernel-related mailing lists, including the main one, linux-kernel. It presents the patches in color and is nicely formatted, and it lets folks change the status of each patch, so it's clear which have been accepted.

Although a variety of git documentation exists, some of it is fairly involved. For folks who'd rather just learn what they need to know, David Howells has written “The Git Hater's Guide To The Galaxy” and submitted it for inclusion in the kernel source tree. It's not the most in-depth git doc out there, but it covers a lot of the stuff David himself uses it for on a daily basis, and it may be more accessible than the more detailed docs out there.

One problem with running dæmons is that the init process loses track of the relationship between the dæmon process and the parent processes that spawned it. As Scott James Remnant has discovered, there doesn't seem to be any straightforward way in Linux for init to know by which process a given dæmon was spawned. That information is currently lost by the spawning procedure.

Scott has coded up support for some new signals to carry this information into the kernel. If accepted into the main tree, this code will allow system logs to track effectively the activities of the application that spawns any dæmons, along with the dæmons it spawns. Among other things, this will improve debugging efforts for any applications that spawn dæmons.

Most modern CPUs have special registers for tracking performance. Things like perfmon use these, but recently Thomas Gleixner and Ingo Molnar took a shot at designing something better. Their initial effort actually has met with fierce resistance from Paul Mackerras and David S. Miller, who felt that the approach Thomas and Ingo had taken was not likely to meet the needs of real systems. The debate does not seem anywhere near over, and certainly Thomas and Ingo will produce some code to prove their points. Probably something excellent will come out of all this, but currently, the two sides have not had much of a meeting of the minds. Paul has said there's no way to track a process effectively without stopping it, and Thomas and Ingo say, yes, it can be done. Time will tell.

EUCALYPTUS: a Tree Growing in the Cloud

Doc Searls

Issue #180, April 2009

From the Linux (and Linux Journal) perspective, there's an issue with clouds—those back-end Web services that compose Utility Computing. They're proprietary. Amazon owns AWS (Amazon Web Services: S3, EC2 and a growing number of others). Google, Microsoft and other companies own theirs as well.

These are open in the sense that most services are substitutable, which makes them utilities. If you don't like storing your data at Amazon's S3, you can park it elsewhere. The documents you create and keep at Google Docs can be put elsewhere as well. Elastic computing is more tricky, but it should be substitutable as well.

In a way, all these data centers are available to the public as a utility service. They are an important trend in computing, comprising the growth of a vast open SLA (service-level agreement) infrastructure for computing resources spread across the Net. (Nicholas G. Carr wrote about all this in his book The Big Switch—Rewiring the World, from Edison to Google. I visited the topic in “The Bigger Switch”, my EOF column in the May 2008 issue of Linux Journal.)

But, why should The Cloud consist only of proprietary back ends? How about building our own cloud services, our own way?

I was pleased to find an answer from colleagues at UCSB (where I'm a fellow at the Center for Information Technology & Society), namely Rich Wolski and his merry band of computer scientists in the Computer Science Department. There they have created EUCALYPTUS (Elastic Utility Computing Architecture for Linking Your Programs To Useful Systems). It's what they call “an open-source software infrastructure for implementing Elastic/Utility/Cloud computing using computing clusters and/or workstation farms”. It is interface-compatible with Amazon's EC2 (the de facto standard for elastic cloud computing) and designed for modification and extension as needed for you-name-it client-side interfaces. Most important for Linux Journal readers, “EUCALYPTUS is implemented using commonly available Linux tools and basic Web service technology making it easy to install and maintain”, they say.

The issue for IT folks is a range of choices that should include DIY as well as commercial back ends. Rich Wolski puts it this way:

Cloud computing permits “self-service” IT independently of whether the provider is an external company or an internal IT organization. Part of what makes IT onerous these days is that the infrastructure organization also must be a customer service organization. Cloud computing really streamlines customer service for IT in the same way that Web-based e-commerce streamlined customer service for retail. It doesn't eliminate the need entirely (you still can get someone on the phone even when a company has an e-commerce site), but it does improve the business efficiency considerably.

As I write this, version 1.5 is coming along. (Version 1.0 was released in June 2008.) And, there are ambitious plans. For more information, check out eucalyptus.cs.ucsb.edu.

Freescale

Doc Searls

Issue #180, April 2009

Freescale may be the first semiconductor company to associate itself aggressively with portable Linux devices. The former Motorola semiconductor division is sharply targeting the low-priced Linux-based Netbook market, which is hot in the world market and just starting to get warm in the US.

Early this year, Freescale launched a new processor called the i.MX515 Cortex A8 Netbook Processor. Company literature positions the part for “low-power, gigahertz performance Netbooks at sub-$200 price points”. According to CNET, Glen Burchers, director of global marketing for Freescale's consumer products group, says “Because the primary function (of a Netbook) is accessing the Internet, Linux and Firefox are a good operating system and application for that purpose.”

Freescale is developing a reference design with ASUS-subsidiary Pegatron. It will feature the 1GHz i.MX51 processor, Ubuntu Linux, Adobe's Flash Player, a new power management chip and the SGTL5000 ultra-low-power audio codec.

Some data:

  • The i.MX515 is a 65nm CPU based on an ARM11 Cortex-A8 blueprint.

  • Clocks from 600MHz to 1GHz.

  • Includes the SGTL5000 ultra-low-power audio codec.

  • OpenVG and OpenGL graphics cores are available.

  • Up to eight hours battery life with displays up to 8.9".

  • Memory interface supports both DDR2 and mobile DDR1.

  • MC13982 power-management integrated circuit, for reduced size and weight and more battery life.

The company is working first with Tier 1 OEMs (so watch for ASUS units) and expects products to start hitting the market in Q2 of this year—in other words, about now.

Resources:

Freescale Semiconductor: www.freescale.com/webapp/search/Serp.jsp

“New Freescale Processors Target Linux Netbooks”: arstechnica.com/journals/linux.ars/2009/01/07/new-freescale-processors-target-linux-netbooks

“Freescale Chip Aims at 1GHz, $199 Netbook”: news.cnet.com/8301-1001_3-10130690-92.html

“Freescale, Nvidia Chase Netbook Sockets”: www.eetimes.com/news/latest/showArticle.jhtml?articleID=212700361

LJ Index, April 2009

1. Lowest legal price in dollars of the Ghosts I–IV album by Nine Inch Nails: 0

2. Price in dollars charged for Ghosts I–IV at Amazon.com: 5

3. Position of Ghosts I–IV among Amazon's top-selling albums for 2008: 1

4. Millions of female drivers in the UK who are “putting themselves and other drivers at risk by wearing the wrong footwear when behind the wheel”: 11.5

5. Preference percentage for a “higher resolution” photo that isn't, in a choice between two actually identical pictures: 75

6. Billions of dollars spent annually on Internet advertising by 2011: 106

7. Percentage of US Internet audience that viewed on-line video in November 2008: 77

8. Billions of on-line videos viewed by US Internet users: 12.7

9. Percentage increase of above over the same period in 2007: 34

10. Google's (Linux-served) YouTube rank as a US on-line video property: 1

11. Billions of YouTube videos viewed in November 2008: 5.1

12. Google Sites again ranked the top US video property, with nearly 5.1 billion videos.

13. YouTube percentage of all videos viewed in November 2008: 40

14. YouTube share of Google video properties: 98

15. Millions of US Internet viewers in November 2008: 147

16. Average number of videos users watched by US viewers in November 2008: 87

17. Duration in minutes of the average on-line video: 3.1

18. Duration in minutes of the average on-line video viewed at (Linux-served, via Akamai) Hulu.com: 11.9

19. Number of results in a search for “Linux” at AMD.com (US): 1,281

20. Number of results in a search for “Linux” at Freescale.com: 1,445

1: Nine Inch Nails

2, 3: Amazon.com

4: Sheila's Wheels (a UK insurance company), citing a 2007 YouGov survey of 754 female drivers

5: ArsTechnica, citing Journal of Consumer Research

6: CRM Today

7–9 and 11–17: MarketingVOX, sourcing comScore's Video Metrix, Marketing Charts

10, 18: SearchDNS.Netcraft.com

19: AMD

20: Freescale

Linus' Blog

Doc Searls

Issue #180, April 2009

That's what it's called. I found out about it when a pal sent a link under the subject line “Hell hath frozen over.” Look up the above on Google and you'll find it. Blogspot, Netcraft tells us, is in the Google, Inc. netblock, and running on Linux.

Shopping on Penguins

Doc Searls

Issue #180, April 2009

I was pointed recently to Zappos (www.zappos.com) as a near-perfect example of a company that brings the principles of open source to business. Its site is inventive and fancy (as you'd expect a clothing retailer to be), but not a triumph of design over utility. What's more, it's fast. I can check Web site responsiveness with some confidence, because our little apartment near Boston has 20Mb/s symmetrical service from Verizon FiOS (that's fiber, and pretty cheap, considering), and the speeds I get at the office I share at Harvard are more than twice that.

So I decided to see what they were running that site on, by checking with Netcraft.com's “What's that site running?” service. The answer was Linux. Zappos itself is in an Akamai Netblock, and of the 13 other results (all foo.zappos.com), the results for OS were Linux or “unknown”.

Then I decided to look beyond Zappos to other on-line retailers. Here are the results for the top ten, as listed by InternetRetailer.com, with sales volume numbers in parentheses. Results for companyname.com are first, and other results for each company are summarized in text (Netcraft gives results in lowercase):

  1. Amazon.com Inc. ($14.8 billion): linux.

  2. Staples Inc. ($5.6 billion): linux, through akamai.

  3. Office Depot Inc. ($4.9 billion): the top result is linux for www.officedepot.com, but officedepot.com (without the dubs) is windows server 2003. Other foo.officedepot.com sites are a mix of the two and “unknown”.

  4. Dell Inc. ($4.2 billion): f5-big-ip, though l.dell.com (Dell laptops) is linux and a couple of foo.dell.com sites are “unknown”.

  5. HP Home & Home Office Store ($3.4 billion): hp-ux.

  6. OfficeMax Inc. ($3.2 billion): f5-big-ip, with a mix of solaris, windows server 2003, linux and “unknown” among other officemax sites.

  7. Apple Inc. ($2.7 billion): “unknown”, but the company uses linux through akamai for a number of foo.apple.com sites.

  8. Sears Holding Corp. ($2.6 billion, includes Sears.com and Kmart.com): for sears.com, linux, plus some “unknown”, solaris, f5-big-ip and windows server 2003 for sears.com.mx. For kmart.com, linux (through akamai), plus linux (with and without akamai) for various foo.sears.com sites. Kmart.com.au is windows 2000. Most of the rest are linux, plus a couple “unknown” and one f5-big-ip.

  9. CDW Corp. ($2.4 billion): the top result is linux for www.cdw.com (through akamai). cdw.com (sans dubs) is f5-big-ip. The rest are a mix of linux and f5-big-ip, with one windows server 2003.

  10. Newegg: windows server 2003, though linux shows up in five out of the nine foo.newegg.com results.

Although that summation is far from a complete picture, or even a completely accurate one for this Top Ten, it's clear that the picture can't be painted without linux as the most primary color.

Resources:

Internet Retailer Top 500 Retail Web Sites: www.internetretailer.com/top500/list.asp

Netcraft: netcraft.com

They Said It

If computers of the kind I have advocated become the computers of the future, then computation may someday be organized as a public utility. We can envisage computing service companies whose subscribers are connected to them by telephone lines....

The system could develop commercially in fairly interesting ways. Certain subscribers might offer services to other subscribers. One example is weather prediction. A weather-predicting company that is a subscriber to the central computer predicts the weather but keeps the predictions in its private files. If you subscribe to its service, your programs can gain access to those files. You may even have weather-predicting programs run for your benefit to answer your own particular questions....Other subscribers might furnish economic predictions. The computing utility could become the basis for a new and important industry.

—John McCarthy, in 1961

At present, a newspaper, magazine or book is a package produced by a large organization. With the information utility, the physical production and distribution disappears, allowing a much smaller organization to put out the same packages of text and pictures. Moreover, the user does not face a one-shot decision to buy Time or Newsweek. He will be able to read the “cover” or table of contents of each, read such items as strike his fancy, and the system will bill him for what he reads from each source. In fact, since the cost of keeping a file of information in the computer and making it publicly available will be small, even a high school student could compete with the New Yorker if he could write well enough and if word of mouth and mention by reviewers brought him to public attention.

Does anyone really think that a company-owned platform is going to win here, that it won't be swamped by an open federated system of servers that peer, like e-mail? If so, I'd like to hear why. We went through this exercise repeatedly in the tech industry; the lesson of history is clear—closed systems have their place and time, at the beginning of a new layer, when users need simplicity over everything else, they serve as training wheels when everyone is a newbie. Eventually we grow out of the need to have our hands held and the freedom of open systems becomes attractive, and we jump.

Nobody will doubt the value of openness for the mobile industry anymore.

as a thank you to our fans for your continued support, we are giving away the new nine inch nails album one hundred percent free, exclusively via nin.com.

the music is available in a variety of formats including high-quality MP3, FLAC or M4A lossless at CD quality and even higher-than-CD quality 24/96 WAVE. your link will include all options—all free. all downloads include a PDF with artwork and credits.

for those of you interested in physical products, fear not. we plan to make a version of this release available on CD and vinyl in july. details coming soon.

—Nine Inch Nails, dl.nin.com/theslip

While Linux-based netbooks have not gained much consumer acceptance in the US, their success varies greatly by geographical market. In developing countries, Linux-based Eee PCs have fared better....Even if a majority of netbooks run Windows, the minority that run Linux are the most successful non-Windows, non-Macintosh consumer PCs in the industry in terms of penetration.

Even More Videos on LinuxJournal.com

Katherine Druckman

Issue #180, April 2009

We love Web videos. We especially love videos that teach us new quick-and-easy tricks, so we've been cooking up some more videos over in the LinuxJournal.com kitchen. We'll have several short video tips per week in the Tech Tip of the Day section. Visit us often to see some fun tricks to make your life just a little better. Here's one of my favorites: www.linuxjournal.com/video/transfer-your-terminal-screen.

What They're Using: Philip Jacob

Doc Searls

Issue #180, April 2009

I first met Phil Jacob on a consulting gig (yes, I got paid, but that was a long time ago), and quickly discovered that he was one of our most careful readers. Subsequently, he also became one of our most constructive critics—in the best sense of the word.

Phil is a born connoisseur. He applies that nature as founder and CTO of StyleFeeder (www.stylefeeder.com), a startup with a “personal shopping engine” that combines machine learning techniques and social data to provide users with a richer shopping experience. Among the features under StyleFeeder's hood is one that personalizes search results from a dataset of more than ten million products using a custom-built recommendation engine. It also has a mature data tier shared across 100 MySQL databases in order to handle its scalability requirements. Given that StyleFeeder is only six people, they've done a lot with a little during the past two years. Phil also is a technology advisor to PRX, a nonprofit licensing and distribution platform for public radio content. He's also married to a well-known food blogger, Beatrice Peltre, who runs La Tartine Gourmande (www.latartinegourmande.com), so at least he eats well.

When I asked him to share some of what he's using, he concentrated his report on a single application—music:

Without a doubt, the device that attracts the most attention from guests visiting my house is the Slim Devices Squeezebox sitting in the living room. They usually don't notice it at first, but there's always that “double-take” moment when I change the music using my laptop or my iPod Touch.

I store my music in the basement on a Dell machine running Ubuntu with a bunch of FireWire disks attached for media files. This server runs Slim Devices' GPL-licensed SqueezeCenter software, which is a dæmon that is mostly written in Perl, surprisingly enough. I happen to know Perl quite well as I programmed with it on a full-time basis for many years, so I was actually quite reluctant to buy this device when I first saw it. What I desperately wanted to avoid was a situation that required work just to get my music playing. And imagine a dinner party going silent because of a segfault in your own code!

I did, however, end up making a few changes to the software, which was only possible due to the open nature of it (at one point, I had reconfigured it to use MPlayer internally rather than LAME because MPlayer wasn't as abusive to the CPU). One thing I really like about it is that I have a plugin installed that keeps a log of everything that gets played, which means that I could actually tell you what I was playing at 8:14pm last Thursday. One of my forthcoming weekend projects is to syndicate this onto my personal Web site, probably with a programming language that I'm looking to learn (OCaml is near the top of my list these days).

I access the music on my basement server when I'm at work by port-forwarding HTTP connections on my router to either Apache or to the SqueezeCenter's built-in streaming capabilities, depending on what I want to do. My router runs dd-wrt, a Linux-based firmware that runs on various types of home routers. It's very stable and provides a bunch of cool features that will appeal to power users.

Asked for last words on the subject, Phil adds, “We run RHEL and CentOS at StyleFeeder, some on EC2 and some at Contegix in St. Louis. We run behind Apache and nginx on the Web tier. We own zero servers and run dd-wrt on our office LAN as well....Typing one-handed with a baby in the other....” Babies are a new hack for Phil.