Archive

Hearing

Here’s a question about vision you rarely think of: Can your eyes create art? From color to eye movement, here are three fascinating examples:

Color to Sound

In perhaps one of the most awesome TED talks ever, especially for synesthesia-philes like myself, artist, composer and cyborg-activist Neil Harbisson speaks about hearing colors. Catalan-raised and Northern Ireland-born Harbisson is color-blind. After listening to a lecture on cybernetics and sensory extensions by Adam Montandon, he approached him afterwards and together they worked on a prosthetic device they call the “eyeborg,” which converts colors into specific sounds. In 2010, he founded the Cyborg Foundation which helps people become cyborgs.

Neil Harbisson. Image via Wikipedia.

Watch Harbisson’s TED talk below. My favorite parts were when he said he is “dressed in C major” and that he can “eat is favorite song.”

Eye Movement to Music and Art

EyeMusic is a project at the University of Oregon’s Cognitive Modeling and Eye Tracking Lab in which an eye tracker is connected to a multimedia performance environment to create computer music and interactive art based on eye movements.

Anthony Hornof practicing EyeMusic. Image via University of Oregon Cognitive Modeling and Eye Tracking Lab webpage

From their website:

While the eye is, in ordinary human usage, an organ of perception, EyeMusic allows for it to be a manipulator as well. EyeMusic creates an unusual feedback loop. The performer may be motivated to look at a physical location either to process it visually (the usual motivation for an eye movement) or to create a sound (a new motivation). These two motivations can work together to achieve perceptual-motor harmony and also to create music along the way. The two motivations can also generate some conflict, though, as when the gaze must move close to an object without looking directly at it, to set up a specific sonic or visual effect. Through it all, EyeMusic explores how the eyes can be used to directly perform a musical composition.

Eye Movement to Graffiti

It’s hard to talk about eye movement without mentioning the EyeWriter. American graffiti artist TEMPT1 was diagnosed with amyotrophic lateral sclerosis (ALS), which left him paralyzed except for his eyes.  The EyeWriter research project was born as a collaboration among TEMPT1, the Free Art & Technology (FAT) lab, the openFrameworks community, and Graffiti Research Lab (GRL), with support from the Ebeling Group production company, the Not Impossible Foundation, and the MFA Design and Technology program at Parsons The New School for Design, New York.The system uses cameras and open source software that allows the user to draw with their eye movement.

openFrameworks co-founder Zach Lieberman (right), with graffiti writer Tony Quan (TEMPT1) (left). Image via SVA Interaction Design.

Advertisements

Speaking at my first TEDx conference was a challenge. It was not just because it was a great platform for ideas, but because of the timing, design, and execution needed to pull off four interactive talks in one day.

TEDxNewHaven: The Art and Science of Happiness. Photo by Chris Randall

The theme for TEDxNewHaven was The Art and Science of Happiness. My goal for the talks was two-fold: to engage the audience in something interactive in the course of the day, and to enable them to view their senses as tools by which to achieve happiness. To achieve these, I had to produce 200 “sense kits” that contained physical, non-digital formats of my projects: HugPrints, Rorsketch, Smellbound, and EatPoetry.

Round One. Photo by Liz Danzico

First Talk: Touch

My first talk was about using our sense of touch. I presented HugPrints, my project where I am attempting to hug everyone in the world and getting visual feedback through a specially designed thermochromic vest. I asked volunteers to come up the stage to hug me.

Hugging on stage. Don’t you just love it? Photo by Chris Randall

Afterwards, I asked the rest of the audience to pull out the HugPrints cards in their sense kits, which contained hugging instructions so they can hug each other (if they wished).

Are you getting enough hugs a day? Photo by Chris Randall

Second Talk: Sight

For the second talk, I explained Rorsketch and first engaged the audience in a guessing game of what clouds looked like and then revealed what drawings I made. After doing this project for a while, I wasn’t surprised about how some people were calling out the same things, while other clouds had very different interpretations.

You know what my favorite Pixar movie is! Photo by Ruoxi Yu.

Yes, it’s a dragon! Photo by Chris Randall

I then asked the audience to pull out the blue Rorsketch cards and the Sharpies in their kits, and they drew their own interpretations on the clouds printed on their cards. To encourage them to draw, I did a live drawing session on a big cloud on stage.

Drawing on a cloud. The cloud I sketched on was taken by Nikki Sylianteng. Photo by Liz Danzico

One of my fellow speakers, Nima Tshering, sent me his cloud drawing. (Thanks, Nima!)

A fairy grandma with a baby by Nima Tshering

Third Talk: Smell, Hearing and Taste

I explained two projects for this post-lunch session: Smellbound and EatPoetry. First, I explained the connection between smell and memory and had the audience remove the Smellbound postcard which contained a printed smell. I showed them the book I made, An Olfactory Memoir of Three Cities. Afterwards, I read them After Apple-Picking by Robert Frost while they ate the apple lollipop inside their kits.

“Now please remove another envelope from your kits…” Photo by Chris Randall

Fourth Talk: Happiness and the Senses

For my fourth talk at the end of the conference, I summarized how we can use our senses to be happy. The key points were that we are all equipped with these tools; i.e. we all had the capacity to hug our loved ones, to interpret a cloud, to smell a memory, and to connect poetry with food, and that if we paid more attention to the world around us, it would promote our feelings of well-being and optimism. I also explained my own explorations with happiness by showing submissions from DrawHappy, as well as what I’ve learned from the project so far.

Photo by Liz Danzico

What Worked, and What I Would Do Differently Next Time

All the sleepless nights making sure all 200 kits had the right contents were definitely called for. So much craft and attention were given to every single detail—each postcard had to be sealed in an envelope so people can only view them as instructed, the envelopes had to match the main color of each project’s logo for easy recall, each postcard had to be branded and oriented in a specific direction, each kit had to contain envelopes in a particular order and needed both a Sharpie and a lollipop. Anything less would have taken away from the experience or would have confused people. Doing something onstage so that people can mirror my actions was also important, and getting all the props together also was something I had to keep in mind in addition to the actual slides.

I’ve also been onstage on different occasions and it was interesting to feel these variegations of audience contact. When performing for the Poetry Brothel in Barcelona, I had to inhabit a certain character and, depending on the piece we were doing, had to maintain a certain mystique. When doing poetry readings just as myself, there was a bit of conversation with the audience when I talk about the process of writing, but during the reading of the actual poem, it was just myself and the text. I felt this while reading the Frost poem to the TEDx crowd—the entire hall was dead silent in contrast to all the other times I was up there. (That was quite fun, actually.) But doing a talk that involved not just a Keynote presentation but an actual creative activity was another world altogether—it’s difficult to assess if everyone was enjoying the activity, although what was great about having a particularly open audience such as this one was that they weren’t afraid of trying new things.

For the next time I do something of this nature, I would have a card that had explicit instructions not to use the objects inside the kit unless asked. Although I anticipated this by making sure the envelopes were sealed, I did catch at least one person eating their candy before lunch. (We were running a bit late, and I suppose he was hungry.) One thing about having a sealed and long candy stick was that it affords a long eating session; they wouldn’t have finished it to begin with and they could just put the candy back in the plastic and in the kit. Hard candy was definitely the way to go.

Also, I would have split the third talk (Smellbound and EatPoetry) into two and let each project have its own 10 minutes. Thus, I would have placed EatPoetry for the fourth talk and followed it with a short summary of the senses. When I went up for the last talk and said that “Ok, I’m not going to make you do anything now,” I could swear I felt the disappointment of some people. If the schedule allowed it, giving each project its own time in the spotlight would have allowed the audience to absorb the concept more fully, instead of rushing to the next project right away. I can’t wait until I develop these projects more and more, and see what I will ask people to do the next time.

Finally

I have to say, I love this format of getting the audience to actually do something creative in a talk, instead of me just standing there telling them about myself. Thanks again to the audience for being open to these ideas, my friends and colleagues for helping me pull this off, and to the wonderful team at TEDxNewHaven who worked tireless to make it all happen!

Thanks to Kate Russell and Liz Danzico, my plus two!

As a poet, I will be among the first to say that poetry can be difficult. It can be abstruse, incomprehensible, and consequently irrelevant to most people. I’ve often felt that this was unfortunate, because I believe that poetry can be very powerful. As Susan Stewart, author of Poetry and the Fate of the Five Senses, wrote:

“Poetry makes tangible, visible, and audible the contours of our shared humanity.”

But what if we had a different gateway to poetry? What if we can explore verses not just through reading text, but through accessing it via an “easier” medium?

This is what is being explored in a project called EatPoetry. It is a dining event where we ask a chef to create a dish out of poems—literally translating the poem into food—so people can readily have an association between the poem and the dish.

Why food? Simply because taste is a sense that is both social and intimate. It might not be easy to have an opinion about a poem, but it is easier to have one about something you are eating. Eating is usually communal and social; thus, the idea of having a conversation around something you are consuming (whether this consumption is through hearing a poem or eating a dish) is an organic process. An event such as EatPoetry also triggers all five senses: the sight, smell, and taste of the food your are eating; the feeling of the texture of the food; the sound of the poetry being read.

This week, I’ve tried a simple experiment where I’ve gathered a group of people to find out their reactions about two different experiences: hearing poetry with and without food. The food were simple supermarket treats, just to see whether this was a concept that was compelling enough for people to enjoy and to find meaningful.

Supermarket treats for prototyping

It was a small group of six people who rated themselves on whether they liked poetry and how much they think they understand it. I had participants who rated themselves as a 1 (the lowest) on how much they liked poetry and how much they think they understood it. I also had participants on the opposite side of the scale; they liked poetry (5) and believed they understood it. One of them wrote poetry herself.

For the poetry alone, here are the poems I read:
1. “This is Just to Say” by William Carlos Williams
2. “After Apple Picking” by Robert Frost
3. “Figs” by D.H. Lawrence (an excerpt)

For the poetry paired with a dish, here are the poems I read and the treats that went with them:
1. “The Cinnamon Peeler’s Wife” by Michael Ondaatje paired with cinnamon buns
2. “The Emperor of Ice-Cream” by Wallace Stevens paired with mint chocolate chip ice cream
3. “Cake” by Gertrude Stein (from her book Tender Buttons) paired with soft chocolate chip cookies

For both cases, I’ve asked them to recall the poem and write down as many words as they could remember. I’ve also asked them to write anything that came to mind, be it their thoughts about the poem (or in the second case, the dish), or their own poem.

Here is what happened:

Poetry and memory

In general, participants were able to recall most of the poem, or at least the ideas in the poem. “This is Just To Say,” “After Apple- Picking,” “Figs,” and “The Cinnamon Peeler’s Wife” were visual and concrete poems; you could almost see the scene/s that were being written about. “The Emperor of Ice-Cream”, although visual, used more obscure words, as well as those that had nothing to do with ice cream. “Cake” was the most difficult, as it had a nontraditional syntax and the words didn’t seem to form sentences that “made sense.”

Expectedly, the ones that had the least recall were “The Emperor of Ice-Cream” and “Cake.” What was unexpected, and more interesting, were the comments, original poetry, and drawings that came out of the session:

Part A: Poetry, No Food

1. This is Just to Say by William Carlos Williams

Sweet, but cold
life it is.

Forgive me, but I’m not sorry.

And they were not far from
reach, these plums,
alive and red, burning
and freezing.
Forgive me, they were
delicious and cold.

Not really asking for forgiveness
Plum metaphor for person? Personality? (sweet yet cold)
Delicate, not long-lasting

Perched on the verge of summer,
I want to work on my thesis.
But thoughts of sweet, cold plums
Keep frolicking in front of my vision
It rather gets in the way
Of writing poems in code.
Doesn’t it?

I ate the strawberries
from the ground
when I was 3,
My mom got mad
at me.

2. After Apple-Picking by Robert Frost

Giving up is easy,
easy to say at least
But you know your heart won’t rest,
until it finds what it’s looking for.

I am done with apple picking now,
There isn’t the time,
To look at the shadows,
Or shadows the time,
Let tomorrows rise,
I let yesterday fall,
For that is all
there is to living small.

Reminds me of the apples
Waiting at home
To be made into apple crumble.

My desk is covered in Apples. Big ones,
Small ones, glistening, cold, smooth
Not unlike ones fresh-picked in Autumn,
But these you cannot eat.

3. Figs by D.H. Lawrence

A secret that each one must unveil,
A different one, for each,
A secret not quite
Revealed in a bite.

If I were to write a
thinly veiled sexual metaphor about fruit
in the form of a poem
I too would probably choose
a fig.

Societies make rules
It says this is good, this is bad.
Rules are objective &
Good/Bad is subjective
If your heart feels differently
break the rules!

Part B: Poetry with Food

1. The Cinnamon Peeler’s Wife by Michael Ondaatje paired with cinnamon buns

Sliced cinnamon buns with chocolate sauce

If I were a cinnamon peeler’s wife
my clothes drying in the courtyard
would bring many a wandering man,
back to my carousel; I
talking amidst them in the market
would proclaim, the love in my veins.

Is the cinnamon peeler’s wife a
cinnamon bun?
I feel awkwardly voyeuristic
But it’s so delicious,
I will just eat and eat some more.

World looks you through a lens
And the lens has a name.

2. The Emperor of Ice-Cream by Wallace Stevens paired with mint chocolate chip ice cream

Mint chocolate chip ice cream

The emperor of ice cream,
sits on a beam, with mint
chocolate dip and lots of cream.
The only emperor is the
emperor of dreams.

You know what’s colder
and sweeter than
a summer plum?
Mint chocolate chip ice cream
that’s what.

3. Cake by Gertrude Stein (from Tender Buttons) paired with soft baked chocolate chip cookies 

Two chocolate chip cookies

Automatic, non stop,
in a way again, come,
these thoughts, flat jab
as a needle in sea, the
sea, that hides, in me.

A tragic baking accident!
It was a little hard to understand
the words as they were
being read. I kept wondering
about homonyms. Maybe some
meaning was lost. I want to read with my eyes, next time.
But the cookies were solid, sweet, and decidedly
non-abstract. I wonder if they helped or
hindered by focus on the poem?

I also got some feedback on the process itself. Was it an enjoyable experience to hear poetry and eat at the same time? Did they like the poems? Did they want other poems to be read next time? Were they able to make associations between the food and the poem? Here are some; highlights are my own.

I don’t know many poems to make suggestions. I probably need to read the poems myself to understand them better. Never “listened” to poetry before. I wish I understood the ice cream poem, because the ice cream I ate was really good. 

I enjoyed the poems and the food, except for the cookies. Not sure if I was supposed to establish a correlation between the food and poems, or remember the poems better because of them. Perhaps that’s what you’re testing? I don’t now. Concentrating on the poems to remember them kept me from enjoying the poems—just made me anxious that I wouldn’t remember / test poorly.

I think more physical, sensual poems worked better for this experience. For the fig poem, I really wanted a fig. I’m not a big fan of G. Stein normally, but I could enjoy it if I were held still for a long enough time. For poems like that, I would maybe pair it with a slow-to-eat food, not something you’d wolf down. (A lollipop perhaps?) Her words take time, more so than W.C.W.s poem which is clear-cut and non-abstract. I really enjoyed it! Thanks, C.

Enjoyed this. Ice cream—favorite food. Robert Frost poem was my favorite, best captured a mood and a place. Although the food was delicious, it distracted from understanding the poems. Gertrude Stein—least favorite; embodied everything I dislike about poetry. (Note: This was from the participant who disliked poetry the most.)

The food was overpowering; the first two poems worked the best, mostly because of the length. Food made it difficult to read anything. Gertrude Stein was beautiful sounding words, which is why the words were all I remember. Shorter poems would be the way to go. Loved the concept as an artwork / performance piece.

Some of my conclusions:

1. Participants generally enjoyed the experience, and giving them a way to express themselves after the poem was read was great. It allowed them to express their opinions on the poem, unlike regular poetry readings where the poem was just read, and that was that.

2. Food was a deterrent in recalling the actual text of the poem; having to taste something distracted from trying to understand something being read. While the participants will likely remember what the ate more than the words of the poem.

3. While eating didn’t facilitate the recall of the poems’ text, what it did do was create an association with the poem. For example, while few understood “The Emperor of Ice Cream,” most enjoyed the ice cream served to them. I enjoyed the drawings a lot, particularly because it came from the participant who had a clear dislike for poetry.

4. It would have been better to be as literal about the food as possible. I served cookies for Stein’s “Cake,” mainly as a wild card, and because the words such as “mussed ash,” “two,” “dirty,” recalled something dark and earthy.

As with prototypes, there are some additional questions raised from this session. Should the poems be projected on the wall instead of just read? Should they be read first and then have the food served, since having them together increased the friction in understanding the poem? How can we facilitate the participants’ own creativity, as to me, one of the more interesting results was the marginalia that they scribbled about the experience? More to come, and soon.