Open-Source Prototyping Delivers Tools for Real Programmers

Defying an industry synonymous with innovation, software programming tools have resisted change for the last 20 years. Michael Van De Vanter, together with Java[tm] inventor James Gosling and colleagues at Sun Microsystems Laboratories, are out to change that. They are experimenting with a programmer-friendly architecture and plug-compatible tools on the NetBeans[tm] platform. By taking their prototypes out of the lab and into the open-source community for real-world feedback, they are positioning their project for success.

November 2002 Michael Van De Vanter is a little surprised when he speaks about the state of the art of software development. In a business renowned for innovation, the innovators' tools - the editing, compiling, and debugging software used to write source code and known as integrated development environments (IDEs) - have bucked the trend. "IDEs... have a lot of features," says Michael, a Sun Microsystems Laboratories Principal Investigator, "but the fundamental technologies - the compiler, the editor - really represent technology that was mature 20 years ago."

This oddity represented an opportunity for Michael and Java[tm] inventor James Gosling, and for Senior Staff Engineers Tom Ball and Tim Prinzing who have researched and used developer tools for years. "There have been a lot of good ideas and interesting approaches that weren't being reflected in the commercially available tools, particularly the IDE interface," says Michael. In July 2000, Michael and James decided to combine forces on a new project. "[T]he opportunity we saw was to to take a fresh look at the world of IDEs. And bring some of these ideas, some new, some old, about how to raise the bar, to make things better than just an accumulation of features."

Their project is called Jackpot. Jackpot is all about making programming tools more productive by making them easier to use and more effective at reducing code complexity. To this undertaking Michael and James brought an intimate appreciation of how programmers actually work along with new architectural approaches.

They also brought a canny research methodology that is already paying off: a commitment to prototype Jackpot using the open-source NetBeans[tm] platform. By taking their prototypes out of the lab and into the open-source developer community for real world feedback, they have positioned their project for success. "We think our approach will be validated -- and completed -- by the developer community," says Michael.

We think our approach will be validated -- and completed -- by the developer community.
--Michael Van De Vanter

To date, Jackpot has yielded a metrics module and a program editor architecture that incorporates pluggable language-analysis components.

Faith, a "compelling intuition," and a bit of fate played roles in the choice of NetBeans as the test and integration platform for Michael Van De Vanter and James Gosling's Jackpot project. NetBeans, the modular, standards-based integrated development environment (IDE) was the product of a small but widely known company in Prague, the Czech Republic. About the time that Michael and James were launching Jackpot, Sun Microsystems acquired NetBeans. In June of 2000, NetBeans was open sourced. Sun[tm] ONE Studio (formerly Forte[tm] for Java[tm]) product line is based on NetBeans.) It was good timing for the Jackpot project, which came together the following month. more»

Integrated Feedback

A version of the Jackpot metrics module prototype is available on the NetBeans platform. Created by Tom Ball, the metrics module analyzes source code using a set of configurable diagnostics. It then helps the programmer "drill down" to potential troublespots. The metrics "help produce better code by pointing at high maintenance areas that need programmer attention in order to produce code that is easier to maintain," says Michael.

"The advantage is that you find problems sooner and that you ship code cleaner, so it will be cheaper to maintain."

Jackpot's metrics are calibrated to track code complexity. "There are known correlations between [code] complexity and the lifetime cost of code," says Michael.

Metrics utilities, as Michael notes, have been around for years. "Source code metrics were traditionally the province of management and auditing tools," says Michael. "As a result, they tended to come somewhat later in the development cycle and less impact back on the actual design of the code." This is a costly flaw.

"Experience [shows] that the earlier you get feedback, the better," says Michael. "It's widely understood in the software engineering community that cost of changing something in a program increases exponentially during its lifetime. So when you are still still developing, it takes a second. By the time you get to integration, it takes longer. By the time you ...ship the product, and it's actually in deployment, the cost of fixing something goes up enormously."

Tom sought to incorporate metrics into the developer's environment in ways that make it inviting to use. The key is integrating the metrics usefully into the process of writing source code. "The first thing you need is integration into the environment. So that it's just right there, with all of your other work. So that within the flow of the programmer, the metrics is just a natural part of it."

When it comes to metrics analysis, the team acknowledges a caveat: that metrics are more like guidelines than hard science. In a world where "software systems have become larger, more complex, more concurrent, more distributed and more dependent on other people and software," the team wrote in a white paper, it's not always possible to write simple code. Accordingly, the metrics module allows programmers to adjust thresholds, set audit levels, and even to turn off diagnostics when needed. These overrides "cut down on the bother factor," says Michael. And that makes programmers more likely to actually use it.

Improving "Visual Bandwidth"

The Jackpot project has also produced an experimental code processor with an architecture that allows programmers to plug in specialized editors. The specialized editors are for the variety of languages found in most source code: the machine language of the code itself, the human language of comments, the typographical lexicon of white space and the highly-specific syntaxes of data such as strings or embedded HTML.

"The approach we're taking is an architectural one," says Michael. Tim Prinzing, author of the Swing Text Framework, "is creating a code editor architecture that supports dynamic creation of specialized editors for pieces of the code, on demand, as you're editing." explains Michael.

The specialized editors use language-analysis to interactively turn themselves on and off. For example, when the programmer's cursor moves over a comment in the code, a human language comment editor is enabled. As programmers move about the program, other editors are activated for data, strings, or chunks of embedded HTML. This approach may permit James to implement a personal quest: a specialized editor for mathematical code. The idea is to use language analysis and typography to present the text in visually appropriate ways - ways that make it more readable and easier to understand.

"Reading is still the dominant task for programmers, even while writing code," says Michael. "We have rich technologies for [making] readable documents, but little is brought to bear on source code."

Michael explains: "Twenty years ago people were using plain text editors such as vi and EMACS for both code and natural language documents. Now almost no one uses vi or EMACs for natural language documents. They use Word, or FrameMaker, or Star Office[tm], because it's been specialized for natural language processing and uses the power of visual feedback in ways that are specialized for natural language documents. It's the WYSIWYG approach pioneered at Xerox Park. That process of specialization has not taken place in the area of code editing... the fundamental editing part of it, what you type and what it shows you, hasn't really changed in over 20 years.

Good programmers know where the semicolons go. What they need is more visual bandwidth to show useful information about the code.
--Michael Van De Vanter

Jackpot's specialized code editors would not only bring a WYSIWYG approach to viewing source code, but also an extensible set of utilities that would aid in a variety of programmer tasks. Among them, "refactoring" - the updating, adapting, and porting that is the fate of most source code over its useful lifetime.

Also at the prototype stage is what Michael calls a general purpose analysis engine driven by more powerful language-oriented technologies. This module would allow sophisticated analysis. The goal is to help a developer construct an alternate view of a program based on a different set of rules; renaming variables, classes, and methods; and conducting queries to show where and how data are used.

The underlying goal of Jackpot is to more usefully balance the conflicting demands that every software program makes on the programmer: that code be intelligible to machines as well as to the humans who develop and maintain it. "Good programmers know where the semicolons go," says Michael. "What they need is more visual bandwidth to show useful information about the code."