A spectrogram is a three-dimensional picture of sound – any sound. The three dimensions are time, frequency, and amplitude. Spectrograms usually look abstract, like successions of clumsy paint strokes or stills from Tron. They’re useful for sound engineers, but not all that good to look at. However, some software can also conduct spectral analysis in reverse, translating images into sound. In this case, the images are clear and the audio typically abstract.
The sound clip above, for example, was created from the picture(s) of me, moving from left to right and reading discrete clusters of color and texture as frequencies. The line of my thumb moves diagonally upward from finger to nail, causing its attendant sound to rise in pitch.
Photosounder.com, home of Photosounder, has a great demo video of how their program works with various images:
This kind of graphic sound creation has actually been around for several years (though newer programs are increasingly sophisticated), and musicians and sound artists have toyed with it extensively as a way to create strange noises that don’t otherwise exist.
I bring it up today not only because it’s fun, but because of the rhetoric of truth and discovery it inspires. People encountering this software for the first time quickly figure out that it can be used to encode information. As an experiment, you could take a screen shot of an email, turn it into “abstract” spectrographic sound, and then send it as an .MP3 to a friend. If your friend knew how you’d created the sound, she could then translate it back into an image, and read your original message. Likewise, you could embed translations of any image or text in a song or movie soundtrack as an ambient layer. The mind races at the probability that this has already been done many times, as a prank, an easter egg, a subtle political gesture, or even a means of legitimately covert communication.
Says one blogger about Photosounder:
I can see people … maybe hid[ing] messages in photographs or art work.
The hermeneutic exercise can, of course, encompass not only human communication, but universal structures as well. Several posters on Photosounder’s Youtube page imagine the applications of spectral analysis for science or pseudo-science:
The potential in this concept is far more than many people realize.
You could, for example, create hidden coded messages insides pictures/fractals, and/or DECODE incredible secrets of the Universe with very little modification.
It would be interesting to hear whether electronic voice phenomena (EVP) could be detected in these complex sounds.
These posters understand implicitly that visualization is the standard best mode of reading data, that sound as we hear it fares poorly against image in revealing patterns and broad trends. Thus, they assume that phenomena in the world (say, ghosts) may exist under our noses, present but undetectable until we invent a light that can shine on them.
For both the spiritual and science-minded, this suggests that natural sound could be worth divining spectrographically in search of patterns we haven’t been able to pick up with our ears. Sonic images that appear orderly invite claims of design, intelligent or incidental.
A recent photoessay of whale and dolphin sounds, rendered with a program similar to Photosounder and published in the London Telegraph, is a great example:
These images may look like just pretty patterns, but they are visual representations of songs sung by whales and dolphins
The sounds were recorded by American engineer Mark Fischer and transformed into visuals using a mathematical tool called wavelets
Mark used to work on US Navy sonar and software for defence and aerospace companies but he now records the underwater conversations between whales and dolphins and transforms the waves into art
Mark then uses a branch of maths called wavelets which creates these intricate structures
"To look at a spectrogram you will see a simple, boring blur with a few harmonics," he said....
"With wavelets, however, there was an image that displayed extraordinary structure. Something was going on with this sound, even if we are not quite sure what"
The effect is even more apparent when colour is applied and the graph transformed from rectangular to polar coordinates, forming a circular graph
Atlantic spotted dolphin, wavelet graph
Recent research shows that humpback whale songs differ in local dialects and contain complex grammatical rules, showing a higher level of communication than first thought
The risk of this approach, of course, is in romantically imagining that the software has no bearing on the data represented. The complex beauty of the pictures may lead us to forget that the computational processes used to render them were designed by people who probably share many of our own standards of beauty – formal symmetry, clear coloration, sharp lines, etc. Faced with images like these, we’re inclined to imagine that god, or evolution, or some other force, created a perfectly-patterned world, one that can ultimately be “read” and understood. But no matter who or what is in charge, that isn’t the case.
This website uses IntenseDebate comments, but they are not currently loaded because either your browser doesn't support JavaScript, or they didn't load fast enough.
Pingbacks to “Waves of Evidence: God, Like You’ve Never Seen Before”
10 comments to “Waves of Evidence: God, Like You’ve Never Seen Before”
Ben – very good. I used to work with this software quite a bit as an undergrad studying composition. I recently listened back to some of the sounds I rendered in those days (without the graphics that generated them). Ben, great observations!
Thanks, Neal. Was it Photosounder you were using?
This must have been done by someone somewhere already, but wouldn’t this be a fun way of "reading" graphical scores? (or regular old scores, for that matter…)
That's a great point, and I agree someone must have thought of it. If anyone has examples, please do share. Otherwise, I guess Nick has dibs on a very commissionable piece.
I would recommend checking out the following article on Rudolph Pfenninger's 1930s experiments in hand-drawn sound, which is germane to this topic as a historical precursor:
“Tones from out of Nowhere”: Rudolph Pfenninger and the Archaeology of Synthetic Sound
Thomas Y. Levin
Grey Room 12, Summer 2003 pp.32-79.
Andrew: Thanks for the reference. d/ling now.
I recently watched a documentary about an only-just-avoided nuclear apocalypse in 1983. One of the people featured was a top East German agent who was embedded at the highest levels in NATO, and used the opportunity to photograph secret documents which were then translated into sound bursts for transmission. Here's the film, the whole thing is well worth a watch:
http://www.youtube.com/watch?v=KpxbpeuSQyA
the great guy sherwin put it best – 'the only true relationship between the auditory and the visual is time'. but you're on the right track here regarding inferred truth (although i would disagree that visualization is the standard best mode for reading data – what kind?).
the interesting question to ask about inter-medial art (and particularly sonification) is why people believe that it does what it does; that's where 'truth' is to be found. the relationship inheres in the technology, and the sedimented and complex set of assumptions about sound, vision and the body (and often commerce) that each technological object, be it software or machine, represents.
that levin paper is a gem. the first two chapters of James Lastra's 'sound technology and the american cinema' really nails the historical context for this stuff as well IMO. i can also recommend – Hertz P (1999) Synaesthetic Art: An Imaginary Number? Leonardo, 32(5) pp. 399-404. Michigan: MIT Press – and Andrey Smirnov's research into the Russian work into this stuff in the 1920's and '30's is an essential contribution.
Hey Rob,
In saying that visualization is the standard best mode for reading data, I'm simply referring to graphic abstraction. Charts, graphs, and statistics are, in essence, reductions of events (which we always experience through multiple senses) into static pictures that we can study. Of course, it's been wisely pointed out that we can also reduce events through auditory representations, and that we can know the world at a remove through sound just as well as we can through image. But that was the basic idea.
In the meantime, I can't thank you enough for the references, which I'm excited to check out.