Listening to the sound of data

When you think of research you probably think of graphs, charts and spreadsheets, but what about sound? Professor Mark Ballora from Penn State’s School of Music is discovering the world of sonification and turning data into sound.

TRANSCRIPT

When you think of research, you probably think of graphs, charts, and spreadsheets.

But what about sound?

Professor Mark Ballora from Penn State's School of Music is discovering the world of sonification and turning data into sound.

Take a listen.

[ Faint beeping ]

When we think about science, we usually think about something we can see -- perhaps a model of an atom, or watching cells divide, or looking at a graph of data.

It's not often that we listen to science.

That's exactly what Mark Ballora, professor in The Penn State School of Music, does.

Welcome to the world of sonification.

♪♪

You're taking information, you're taking data points, and you are transferring them into a sound property of some type.

So, you might have a sequence of numbers that you convert into a melody -- a simple example.

[ Keyboard clacks ] I tell people it's a lot like visualization, conceptually, except you're doing it for your ears instead of for your eyes.

And people say, 'You sonify data? Why?'

One important one is accessibility.

One such scientist who uses sonification to analyze data is the astronomer Wanda Díaz-Merced, who lost her sight due to illness.

She worked with some scientists to create a program that rendered these graphs as sounds, as melodies.

And she says she's able to work at the same level that she worked at when she was sighted, by listening to the graphs.

And there was also an unexpected benefit beyond accessibility -- one that had benefits for her sighted colleagues.

She discovered that there were some electromagnetic resonances.

Nobody had detected these by looking at the graphs.

By listening to the graphs, she discovered that they were there.

So it's like, 'Wow, they made a discovery through listening that they hadn't been able to make by looking at the graphs.'

Mark himself would discover unique ways of using sonification while coming into contact with some unlikely collaborators.

During a sabbatical doing sonification work at Penn State's College of Information Sciences and Technology, Mark would come in contact with Mickey Hart, one of the drummers of the Grateful Dead.

I saw that he was working with George Smoot, a cosmologist at Lawrence Berkeley Labs who had been a co-Nobel laureate in 2006 for his work in cosmic microwave background radiation.

He had joined forces with George, and they were making this outreach film called 'Rhythms of the Universe.'

There was a note in it that said 'We're going to keep working on this film, and we will be exploring new forms of sonification.

And anybody who has data sets they'd like to contribute could contact Lawrence Berkeley Labs.'

Mark would end up contributing a number of sonifications to 'Rhythms of the Universe,' including a sonification on cosmic microwave background radiation.

[ Magnetic humming ]

I like to think of it as you're finding the music in science.

You're finding a way to make science musical.

That, to me, is going to be the key to future discoveries in sonification.

These kinds of sonifications brought attention from other scientists, including from renowned Penn State meteorologist Jenni Evans.

Over the last 20 or so years, my group has been among a now-growing group of people around the world looking at tropical cyclones that move out of the tropics -- like Hurricane Sandy, for example -- and change their structure.

Professor Evans' group developed the Cyclone Phase Space to do this research.

She asked Mark to create sonifications from this data, and also to combine it with other key dimensions of a cyclone.

And Mark actually combined those descriptors of the storm with the more traditional intensity and location.

Mark started with 11 storms, but then it developed into something bigger.

And then, she thought, 'What if we took a global view?'

And we looked at a whole map of the Earth, and we saw, as she put it, the spaghetti of storms that took place over the course of a 12-month period.

And she picked 2005.

That was a pretty active year.

This Spaghetti of Storms sonifications is what you see and hear in this animation.

[ Whirring ]

When you look at them, it might look like the center's in the middle of some curved band of clouds.

So, you think of the center in the traditional way for a tropical cyclone.

But as it's moving out of the tropics, it's actually off to one side.

The center's off to one side of that cloud.

And so, having the sound of the symmetry lets us know that.

Mark hopes to continue pushing the boundaries of sonification, where it will become an integral part of science.

There's also the hope that it will help students learn science more effectively.

I'm thinking of my kid here, who's going into the third grade, as he starts taking science classes, and they start teaching him how to listen to graphs as well as look at graphs.

As people of his generation grow up, they will simply expect that.