Technology that helps us find new music we like? A search engine that reinvents the music experience? You heard it here first.

September 26, 2006 - It has never been easy to find new music that you like. No one has time to listen to everything; recommendations aren’t always reliable; what’s popular isn’t necessarily what’s good; and the sheer size of music collections keeps mushrooming.

What’s more, finding new music you like is only getting harder, despite the advent of online recommendation services and search tools that let you sort through enormous collections in seconds. As music collections have continued to grow, music search tools simply have not evolved. They are still blunt instruments that categorize music according to factors that are outside the music—artist, album, song, and genre—as opposed to within the content of the music itself.

A research project at Sun Labs is exploring new methods of searching music by its acoustic content and context. This project is aimed at helping people find and organize their music based on the properties of the music itself: lyrics, musical theme, melody, tempo, rhythm, and instrumentation.

2D Visualization of the Music Space

Led by researcher Paul Lamere, the Search Inside the Music project uses acoustic similarity to help people find music that “sounds similar” to music that they already like. The system can also combine acoustic similarity with social data such as the listening habits of people with similar musical tastes and use that as a basis for recommendations.

For the moment, let’s table the philosophical questions about whether technology should help us make music choices, and simply consider the question of whether it can. The fact is, the music search technology at Sun Labs is surprisingly sophisticated at handling the staggering complexity of identifying acoustic similarities.

Analyzing Audio Content: No Cheap Trick

Which song, performed by a band other than the Beatles, sounds the most “Beatle-y?” A simple question, but how would you even begin trying to answer it? Sun Labs’ Paul Lamere began where any self-respecting researcher would start: with hard data.

“Every song is really a series of acoustic features and characteristics that can be measured, analyzed, tracked, and compared,” he said. “So the first thing we do is generate metadata directly from the audio content.” A few of the features that can be extracted and analyzed include pitch, harmony, key, timbre, instrumentation, tempo, rhythm patterns, and intensity or energy level.

Play List Based on Music Similarity

Like some other music recommendation systems, the Sun Labs system is able to do “collaborative filtering”, that is, using algorithms to examine the behaviors of millions of music listeners and make recommendations based upon patterns mined from this large dataset. “Collaborative filtering can be a good way to recommend music,” said Mr. Lamere, “but it has limitations—for example ‘popularity bias’ causes hit songs and artists to be recommended over less-known songs, so it’s difficult for new bands to break out; and because the sample is large there is high inertia—the systems are slow to recognize shifts in taste or new trends.”

By contrast, the Sun Labs system is blind to the popularity of a given song, so a new or emerging artist is on equal footing with an established band such as U2. Content-based recommenders such as the Sun Labs system can help people find new music before it is popular music, and it can help people find old music that is still good music.

3D Visualization with Song Playlist

“Ultimately the best system for recommending music will be a system that can combine the best attributes of content-based and collaborative filtering systems,” said Mr. Lamere.

The Musical Journey

One of the most impressive capabilities of the Search Inside the Music system is its ability to generate a visualization of the acoustic distance among songs of different genres. Sun Labs researchers have analyzed feature data from a collection of about 5000 songs, categorized the songs according to the genre specified by the artist, and plotted them on a three-dimensional grid according to their scored acoustic similarity.

The result is a striking visualization that not only shows which specific songs are similar to other songs of the same genre, but that also illustrates the degree of acoustic similarity between songs of different genres.

What’s the practical value of this type of visualization? One example: Sun Labs has devised a way to quickly create customized playlists based on acoustic similarity. For example, let’s say you’ve had a rough day at work; you’re leaving the office and heading into heavy rush-hour traffic, and you want to hear music that will help you reduce your stress level as you drive home.

Songs Plotted on 3D Visualization Grid Cover Art courtesy of Magnatune

Now it’s a Question of Scale

Several other research teams around the world are experimenting with Music Information Retrieval (MIR) systems. One of the problems that all of these teams eventually encounter is the issue of scalability.

The Sun Labs system, for example, analyzes the features of the music frame by frame, measuring multiple attributes such as pitch, beat, instrumentation, and so on. Each frame represents a 40-millisecond slice of the music. In an average 200-second song there are 5,000 frames, which translates to, in the words of Paul Lamere, “a very non-trivial compute power requirement.”

Multiply the 5,000 frames per song times the number of songs in even a modest collection, and you’re talking about an enormous compute power requirement—more than even the most powerful server made by Sun could easily accommodate. The storage requirements are equally daunting—where do you archive the feature data of literally billions of individual frames? How will it ever be possible to scale up the system to work with very large collections of 2,000,000 songs or more?

“This is a case where being part of Sun really helps,” said Mr. Lamere. “First of all, Sun is pioneering a new computing model called grid computing that lets you plug into a huge network of extremely powerful computers and draw on their combined CPU power. It’s like electricity from a wall outlet or water from a faucet, only it’s raw compute power. The same model works for storage capacity. You can use it on demand, there’s plenty of it, and you only pay for what you actually use. This model is catching on at Sun and beyond, and it shows promise for meeting our processing and archival requirements and enabling us to scale up to work with very large music collections.

3D View with 1000's of Songs and Cover Art Cover Art courtesy of Magnatune

According to Mr. Lamere, a desktop system would take over six years to analyze a 2,000,000-song collection, whereas the same task on a 1000-node compute grid could be completed over a weekend.

“Equally important,” continued Mr. Lamere, “is the fact that Sun has a long track record of collaboration with other research institutions, standards bodies, and development communities, and this helps us scale our research efforts. We’re not about keeping the hood closed until we have a commercially viable technology. We want to share what we know with others, and collaborate to make our technology work better. That’s a philosophy that Sun has always encouraged and supported.”

For example, Sun Labs is actively working with a wide range of other researchers and organizations on its music search technology, including AST (Advanced Search Technologies), the HCI (Human-Computer Interaction) organization, researchers from the University of East Anglia, McGill University, and the University of Illinois at Urbana-Champaign, as well as researchers working on digital rights management (DRM) technologies, storage and archival technologies, and grid computing.

Of particular interest, Sun Labs researchers are leveraging the work by the AST group to do data mining of all the text that surrounds music—including lyrics and reviews—and using this information as yet another way to determine artistic similarity. Dr. Stephen Green, who leads the Sun Labs Advanced Search Technology project, is working to support artistic similarity evaluation directly in the AST search engine, which will open up all sorts of new possibilities in combining text and music search.

In addition, Mr. Lamere currently serves on the board of the International Music Information Retrieval Systems and Evaluation Laboratory (IMIRSEL), an organization that is developing standard collections and evaluations for MIR systems. Sun Labs contributed to the first official evaluation of MIR systems by the Music Information Retrieval Evaluation eXchange (MIREX), and contributed to M2K, the Java-based framework that is being used in the MIREX evaluations. Sun Labs also sponsors and participates in the International Symposium of Music Information Retrieval (ISMIR), the annual MIR conference.

Imagine the Possibilities (and the Impact)

Now it’s time to address the inevitable questions: what is the commercial potential of this technology? Will anyone really want to pay for computer-generated music recommendations? After all, what people “like” in music is undeniably driven by many intangible and subjective human factors, including memories, associations, acquired tastes and even moods—not just acoustic similarities among songs.

“My response is that first of all, this is not simply a music recommendation system,” said Mr. Lamere. “That is one possible use but there are many more, and we’ve just begun scratching the surface.”

For example, said Mr. Lamere, the technology could be used for audio indexing (i.e., to find the Maynard Ferguson trumpet solo in a long jazz piece). It could be used for thumbnailing (for example, extracting the chorus). It could be used for “fingerprinting” a song for copyright enforcement purposes. Movie producers could use it to quickly find mood music that works with a particular scene. Eventually, we could even get to the point where a person could hum a few bars and the system could identify the song and locate a commercial version.

Mr. Lamere also points out that music search capabilities could fundamentally change the music experience itself. “Think of what artists could build and create if they had quick, easy access to specific sounds and tones and rhythms from millions of songs,” he said. “Think of the new music communities that could arise and the new social interactions. I can picture classical music buffs listening to three different conductors’ interpretations of the Jupiter Symphony and then debating which is best.”

According to Mr. Lamere, music search capabilities could also alter the traditional “hit-driven” business model of the music industry and lead to broader exposure for everyone from garage bands to musicians in far-flung corners of the globe. “The Internet and the Web have already been huge in leveling the playing field for musicians,” he said. “Search Inside the Music could be the last piece of the puzzle in helping listeners find new artists’ music. You start getting recommendations based on the actual content of the music, not popularity bias.”

What's the next step for Sun Labs’ music search system? “We’re here to explore, to learn, to see where this takes us,” said Mr. Lamere. “I know that I get up every day thinking I have the coolest job in the world. And I think as long as there is energy and enthusiasm around the core technology, here at Sun and elsewhere, it will naturally develop and evolve and commercial opportunities will present themselves. And when that happens, I expect Sun will be at the forefront of an important emerging market.”

For More Information

Album art from the band Pizzle, available at