Interview with Dr Stephen Wolfram

There are myriad ways to describe the prolific Dr Stephen Wolfram—inventor, author, groundbreaking scientist, IT CEO and entrepreneur and MacArthur Genius Grant recipient (also toss in Renaissance man to emphasize the diversity of his achievements). Despite Wolfram's many laurels, there's a fair chance you never have heard of him. This accomplished Briton, who has lived most of his adult life here on American soil, developed Mathematica, the world-standard technical computing package, and is refining the ambitious Wolfram|Alpha computational knowledge engine, a long-term project to make the world's knowledge computable and accessible to everyone.

Dr Stephen Wolfram

Before Mathematica and Wolfram|Alpha, Wolfram published his first scientific paper at age 15 and knocked out his PhD in theoretical physics from Caltech by age 20. He's a man who seems to be living well ahead of his age as well as his time.

Recently, Linux Journal caught up with Dr Stephen Wolfram to learn more about this fascinating man and the important, cutting-edge work he continues to do.

JG: Your “computational knowledge engine” Wolfram|Alpha has been running for more than two years now. How has it been doing out in “the real world”? And, are there any surprises regarding who is using it and how?

SW: It was a difficult decision when to first release Wolfram|Alpha. Eventually, we just couldn't learn much more without seeing actual users using it. Watching that has been fascinating, and it has let us steadily make Wolfram|Alpha better. Our goal is to understand queries in whatever form a user chooses to give them—with no “manual”. We have a whole calculus of linguistic variation, but it's amazing all the strange forms people actually use. These days though, we can understand more than 95% of queries the first time, which I think is pretty good.

As far as users and uses are concerned, I'm continually amazed at the diversity of people I run into who turn out to be avid Wolfram|Alpha users. We intended Wolfram|Alpha to become a very widespread tool, and that's steadily happening.

JG: A fellow thinker, Rudy Rucker, called Wolfram|Alpha a “platonic search engine”—one that unearths eternal truths that may never have been written down before. Is that an accurate observation?

SW: It's a nice description. At a practical level, I can say that these days the majority of queries people enter in Wolfram|Alpha get zero hits in a search engine. But we can handle them because we're computing answers afresh, not trying to find something someone happened to write down before.

Sometimes the truths we compute are “eternal”, like the solution of a differential equation or the property of a polyhedron. But a lot of the time, they're dynamic. They are based on data that comes from the outside world, like the weather somewhere, or the price of a stock or the position of a spacecraft. And at some level, what starts as a philosophical issue about eternal truths ends up as a very practical issue for the Wolfram|Alpha caching system.

JG: What is your long-term vision for Wolfram|Alpha?

SW: I'd like to see as much of the world's knowledge as possible be computable, so that it can be used to answer whatever specific query one might have. With the progress of technology, we've been able to automate so many things. But a lot of the achievements of our civilization are still encapsulated only in the knowledge of human experts. I'd like to automate using that knowledge and get to the point that any question that could be answered by a human expert based on accumulated knowledge and data can be answered automatically by Wolfram|Alpha. I think that kind of automation is going to let us humans get a lot further on many fronts.

At an engineering level, Wolfram|Alpha gives us a whole new approach that I call “knowledge-based computing”. We're used to the idea that software has to be built from raw computational primitives. But Wolfram|Alpha gives us a platform where we can in a sense start from the knowledge of the world, and then build from that. The result is a major change in the economics of all sorts of software development, as well as making lots of new and sometimes unexpected things possible. We're seeing some of these things as Wolfram|Alpha extends from just being a Web site to supporting a whole ecosystem.

As of right now, Wolfram|Alpha takes short textual queries and computes results from them. We're about to release a version that can take all sorts of other input—whether it's images or data files or programs. And we're going to see it not just compute results, but also cause things to happen. For example, last year we released a new version of Mathematica that uses Wolfram|Alpha to synthesize runnable programs from free-form linguistic input.

JG: What have been your technical challenges regarding Wolfram|Alpha?

SW: I've done some pretty complex projects in my life, but I have to say that Wolfram|Alpha is outrageously more complex than any of the others. It's got an incredible number of different kinds of moving parts.

We've got to get computable data about every kind of thing. We've built a pipeline for curating data that's a mixture of automation and expert human input. And even though we've handled thousands of domains now, it's amazing that almost every new one seems to have some new twist. And, to get it all right often requires talking to the world expert in that domain.

After the data, we've got to actually implement all the methods and models and algorithms that are needed to compute useful things from that data. It's all done in Mathematica (which is what makes it possible), but by now Wolfram|Alpha is about 15 million lines of Mathematica code.

Then there's the linguistic understanding, which at first I thought—based on the history of natural language processing (NLP)—might just be plain impossible. But the problem we're solving is sort of the inverse of the usual NLP problem, and particularly by using insights from an area of basic science that I've developed, we've been able to make incredible progress.

There are issues with generating output too—automating aesthetic judgments and the architecture of information presentation. And then, of course, there's lots of supercomputer-style infrastructure that's needed to actually run the computations for Wolfram|Alpha.

From a software-engineering point of view, we've made use of all the tools that we've developed during the past 25 years for Mathematica. These days, the tools are almost all written in the Mathematica language, and they do a really good job of automating building, testing and deployment. Wolfram|Alpha is an incredibly nontrivial system to monitor and test, but so far we've been pushing out new versions successfully every week ever since it launched.

JG: I imagine your development team for Wolfram|Alpha must be quite eclectic. Can you tell us a little bit about that team?

SW: Yes, it's a very eclectic team. On the data and algorithms side, we have experts in lots and lots of different fields. For the overall frameworks, I think the highest concentrations of backgrounds are physics and computer science. There's also a huge concentration—particularly in the linguistic algorithms area—of people who've worked on science based on my book A New Kind of Science.

Wolfram|Alpha almost by definition requires incredible diversity of skills and knowledge. We've ended up having to keep a WhoKnowsWhat database at our company so we can quickly find who might happen to know about forestry, or Vietnamese character sets, or machine tools or whatever.

It's also interesting that Wolfram|Alpha needs people with skills that I doubt have ever been defined before—for example, linguistic curation.

JG: As you look back on 25 years and eight versions of your original and flagship application, Mathematica, how would you assess its impact and importance in the world of technical computing?

SW: Before Mathematica, it was really rare for technical people to “do their own computing”. They usually had to have special “programmers” who did it. Mathematica has made that unnecessary, and the result is that an awful lot of advances have been made. It's pretty satisfying to see how much has been discovered and invented through the years with Mathematica. Fields like experimental mathematics have pretty much been made possible by it.

I'm not quite sure, but I think I probably coined the term “technical computing”—not that I like it very much. I have to say that I think we pretty much defined what “technical computing” should be. Through the years, we've redefined it a few times, and I think that particularly as we bring Wolfram|Alpha and Mathematica together, we're going to see the most dramatic changes that have happened since we first released Mathematica.

Even after 25 years though, the story of Mathematica is only just beginning. Through the years, I've slowly understood more and more that's possible on the basis of the fundamental ideas in Mathematica, like symbolic programming. We're now pretty much finished with the to-do lists that I had for Mathematica in the late 1980s, but now I realize there's so much more that's possible.

Through the years, Mathematica has emerged as an incredibly strong platform for algorithm-rich software development. We certainly use it very successfully for pretty much everything at our company, and of course, it's the basis for Wolfram|Alpha. It's used in plenty of other organizations too, but we're building some things now that I suspect will make it a lot more widespread in the future.

JG: Can you share some interesting insights about Mathematica that would interest our technical readers?

SW: At a technical level, the core of Mathematica is its symbolic programming language. And, that language is based on one big idea: that everything—whether it's data, documents, programs, interfaces, whatever—can be represented in a very uniform way, as a symbolic expression. And because of that unification, it's possible to have a fairly small number of very powerful primitives that give a vast amount of functionality.

In terms of scope, we've had a pretty simple principle: just implement every definite algorithm we can. At the beginning, we emphasized things that would often be classified as “math”, but we long ago expanded far, far, beyond that. Whether it's image processing, or control theory, or network analysis or text processing, if there are useful algorithms, they're probably in Mathematica.

A strong principle in developing Mathematica has been to automate as much as possible. The humans say what they want to achieve; it's up to Mathematica to figure out which algorithm to use, or whatever, to achieve that. We've also worked very hard to keep the design of Mathematica as coherent and consistent as possible. And, the result of all this is that in a sense we get to build Mathematica with bigger and bigger bricks. Each new piece of functionality finds it easy to use larger and larger chunks of existing functionality.

We've done a lot of work on both system design and software engineering during the years, and it's increasingly been paying off. So if you look, for example, at a plot of the number of functions supported by Mathematica versus time, it has a pretty impressive, perhaps exponentially looking form in recent years.

JG: To what extent did Mathematica inspire and inform Wolfram|Alpha?

SW: Well, of course, at a practical level, Wolfram|Alpha is implemented in the Mathematica language. It's all Mathematica. And if we hadn't had Mathematica, Wolfram|Alpha never would have been possible.

We use Mathematica for all its built-in algorithms, but more important, we use it as an incredibly powerful language for representing knowledge and defining new algorithms. We also use it as a large-scale deployment environment—actually delivering the computations and results from Wolfram|Alpha on the Web and elsewhere. And, needless to say, all our software engineering and testing and monitoring systems are written in Mathematica too.

In a sense, Mathematica and Wolfram|Alpha are conceptual opposites. Mathematica is based on a very clean and precise language that can be used to build very large structures. Wolfram|Alpha has to handle all the messiness of the world, and of human language, and initially is used mainly for one-shot queries.

For me, it was very interesting to think about Wolfram|Alpha after so many years of working on Mathematica. I could, in a sense, do everything exactly the opposite way. But, now it's really exciting to see how these different approaches can be brought together, so in the latest version of Mathematica, we use Wolfram|Alpha technology to let people enter free-form natural language inputs and get precise Mathematica code out.

It's pretty interesting. It's a new way to do programming—just by using natural language. And, I am frankly quite surprised at how well it already works, and how useful even I, as an expert Mathematica programmer, find it.

JG: Unlike many software developers, you've had your application Mathematica available on the Linux platform for years. When did the first Linux version arrive, and what motivated Wolfram Research to be such an early Linux supporter?

SW: I just looked at my e-mail archive, and I see that the first internal discussion about Linux at our company happened in 1993, and we started the actual port of Mathematica to Linux in late 1994. I'd have to read through lots of messages in my e-mail archive to remember exactly how the discussion about Linux went. But in general, it's been our principle ever since the beginning that we want Mathematica to be available on as many platforms as possible.

It's good we have a well-developed Linux version of Mathematica, because nowadays all our Wolfram|Alpha servers are running Linux!

JG: Could Wolfram|Alpha or Mathematica benefit from a model similar to open-source software—that is, with a distributed group of contributors to complement your core development teams?

SW: In Mathematica, we use lots and lots of open-source software these days. In fact, all the acknowledgements for it are getting to be the length of a small book. It's wonderful when we can use a strong piece of open-source software for an algorithm or a component. It lets us put more effort into the unique parts of the Mathematica system, like algorithm automation and its overall design.

In Wolfram|Alpha, we have an increasingly large volunteer program, with people around the world helping us with particular data curation tasks that they're interested in. We're also thinking about how to develop open projects and collaborations to work on very large-scale data projects that we think would be good for the world, but which we can't afford to do ourselves.

JG: You and [the late] Steve Jobs went way back. How did that relationship come about?

SW: Steve Jobs was working in stealth mode on the NeXT computer. It must have been 1987. We met through a chain of acquaintances and ended up making a deal to have Mathematica bundled on every NeXT machine sold. It was actually the first major deal we made for Mathematica. And Steve Jobs was also the person who pushed me to use the name “Mathematica” instead of various alternatives.

JG: You were the youngest person ever to win the MacArthur “Genius” Grant/Award. What did you win it for?

SW: I got that award in 1981. When Rod MacArthur (son of the late John MacArthur) called me to tell me about it, I actually asked what it was for, and he said they didn't ever specifically say that. So I guess I don't specifically know.

I can say that in 1981, I was 21 years old, and I had mainly worked on two things. First, starting when I was 14, I had published all sorts of papers in theoretical physics—mainly about particle physics and about cosmology. I'd also become very involved in using computers to automate the calculations I needed to do. And starting in 1979, I'd been developing a symbolic manipulation software system that was in some ways a forerunner of Mathematica.

JG: From the outside, it appears that you are a person who is living the way he wishes without much compromise. Is that observation accurate?

SW: It's certainly what I aspire to, and I've spent a lot of effort doing my best to get myself into that position.

Of course, I've made plenty of choices. For example, I could work on things that are somehow maximally intellectual, but don't have practical significance and don't make money. Or I could try to make the maximum amount of money, but work on things that are intellectually rather dull. I've chosen a middle path, which I'm pretty happy with. I get to do lots of very fascinating things, but they're useful enough that they've let me make a very decent amount of money, which gives me the freedom to go on doing interesting things.

I've spent years building up Wolfram Research and collecting lots of very brilliant people there who I enjoy working with. I've never taken the company public or had investors. So I've been able to concentrate on long-term things that I think are really interesting (like Mathematica and Wolfram|Alpha). And I also don't get fired as CEO when I spend years working on basic science. (Although actually that basic science has now turned out to be crucial to our products too, but that's a different story.) My theory is that having Wolfram Research is the best way I can turn ideas that I care about into real things in the world.

I've been a remote CEO for more than 20 years now, working at home most of the time. I've ended up using technology to make my work as efficient as possible. I figure that whatever I can automate and make routine, I don't have to think about. So I can spend my thinking effort on the stuff that's really worth thinking about. And that I really enjoy.

JG: From a young age, you've been ahead of your time, regularly producing innovations in mathematics, science and computing that typically come from someone ten or 20 years older, if ever. What can you tell us about what is going on internally—your thoughts, impulses, philosophy and so on—that would help us understand how you've managed to be so ahead of your time all of your life?

SW: Thanks for the kind words! I guess I like to understand things at the most fundamental level possible, and when I succeed in doing that, it often seems fairly obvious to me what direction the things should go in. It's been very common in my life that I think about doing something, and then a decade or two later it becomes very popular.

I think I achieve the most by doing large long-term projects (although I find it a lot of fun to do the occasional short project), but one doesn't get to do that many large projects in a lifetime. One has to pick and choose. I like to pick ones that I think won't get done as part of the general development of things—ones where I can feel I'm making a unique contribution by doing them.

JG: Readers of Linux Journal tend to be “outside the box” kind of people. (We think you'd fit right in!) Do you have any advice for any budding Stephen Wolframs out there—people who have great ideas and want to make important contributions to their field, but aren't sure how to proceed?

SW: I think the first thing is to have confidence. Figure things out as best you can for yourself—perhaps asking lots of people for advice too—then stick to what you've concluded. At any given time, there's always a conventional wisdom about how things should be done or what's possible. If you believe something different, go with what you believe. You always can define success to be whatever it is that you achieve.

I happen to find people really interesting (a great asset if you're trying to build a company like ours!), and I've worked with lots and lots of people who've done great things. My main conclusion is that there are an awful lot of different ways to achieve things. For any given person, it's important to understand yourself as well as you can. Even after all these years, I gradually understand myself better and better, and every increment in understanding seems to help me a lot in managing to do the right things.

It's a fascinating problem matching people (including oneself) with the best possible directions and niches in the world. It's amazing how often people get stuck in the wrong niches—sometimes because they just didn't know or understand other niches that would be much better for them, and sometimes because they think they have too much investment in one niche and can't see how to bridge to another. My observation (for myself and others) is that with sufficient creativity, one always can come up with a strategy to get from here to there, and the result is much more satisfaction, fun and productivity.

JG: Do you have more big initiatives up your sleeve, or are you content right now to focus on Mathematica and Wolfram|Alpha?

SW: I've done three large projects so far in my life: Mathematica, A New Kind of Science (NKS) and Wolfram|Alpha. They're all never-ending projects, and I hope to go on pursuing all of them forever, but I also hope to do some more big projects.

I have quite a long list of projects I'd like to do, mostly very large and ambitious. The big challenge is to pick the right time to do them. I'd been thinking about the general idea of Wolfram|Alpha for several decades, but always had decided it was too early to try actually doing it. One has to wait until the ambient technology is to the right point. Some of the projects I'm considering may be close; some may be 50 years away.

One project I'm committed to doing is to use the work I've done in A New Kind of Science to try to find a truly fundamental theory of physics. I'm ready to dive into that project, but right now, the amazing collection of opportunities around Wolfram|Alpha are keeping me far too busy.

There are also some very, very interesting technology projects that are made possible by NKS; I just have to figure out when is the right decade to start them. I often think about projects for decades before I actually dive into doing them. But it's always exciting when a new project gets going, and one starts seeing how something can be created from nothing. It's always a great experience, both for me and for the people I'm working with.

JG: Thank you for your insights, and good luck in your fascinating and important work!

SW: Thanks for some very interesting questions!