TTYU Retro: The illusion of "natural" electronics interfaces, and the book

Ever since the advent of the iPad and iPhone I have noticed people talking about how "natural" the touch-screen interface is. I have seen people on Facebook posting video of their children trying to make a magazine perform like an iPad (with little success, obviously). Quite recently I have also seen advertising for new products that involve flexible circuits, where you'll be able to bend and flex your little phone or what-have-you in order to make it work, and how "natural" this is compared to everything that has gone before (I'm assuming, by implication, this includes the touch interface).

I would like us to take a little step back and examine this idea of what "natural" means. "Natural" means something that we do as part of our nature - but our nature is quite complex.  Humans are capable of all kinds of complex learned behaviors. When I examine different kinds of interfaces with electronics that we've used, what it commonly comes down to is a question of metaphor. Interestingly enough, the creation of metaphors is a very natural activity for human beings.

The original computer interface may not have felt natural, but really it involved learning a typed foreign language in order to interact with a user of that language (albeit a non-human one). This is something that has been done a lot, historically. It's not as natural as learning spoken language, but we all learn to write as children, and many people have the opportunity to learn foreign languages. The PalmPilot device allowed us to use handwriting as input...so long as we wrote it according to the language rules of the device. Would it be more natural if we could speak to a computer and have it understand us? Star Trek certainly seems to believe it is (and so does Dragon, not coincidentally subtitled "naturally speaking").

So here are a couple of possibilities for "natural" - one, that "natural" means we only need to learn something that can be done easily, and two, "natural" means that the computer should have to adapt itself to our modes of communication rather than vice versa.

The graphic user interface does something interesting. It moves the computer away from purely language-based self-expression into a visual mode of expression (though I remark this graphical mode is built on the basis of underlying layers of computer language). The visual interface takes advantage of the metaphor of art, expressed wonderfully by the painting "La Trahison des Images" (The Treachery of Images) by René Magritte:

The text reads, "This is not a pipe." And indeed, it isn't, just as a "button" on your screen isn't really a button, but an image to which we assign the name "button." Icons take the place of real objects, and they can be manipulated in the image-space.

What we use to manipulate them is another place where we might question "naturalness." The mouse, the touch pad, the tracking ball, and the tablet can all be considered less natural than the touch-screen interface, but what is "natural" here? Is it the indirectness of the interface? We work with indirect relations constantly without much trouble, every time we use tools. Think of cutting a piece of paper: tearing the piece of paper apart might be considered most "natural" but it's messy. Surely folding and tearing isn't particularly natural (at least in the "ease" sense of the word). The straight stroke of a knife down the paper is already starting to be indirect, though it is much simpler. And what of scissors? Would you call them natural or unnatural? A grasping action of the hand isn't at all obvious in its relation to cutting along a straight line, but we do it constantly, easily, starting before we even hit kindergarten.

The Wii was hailed as a much more "natural" way to play video games - instead of translating strenuous activities like fighting or boxing or skiing all the way down into a tiny joystick or keyboard arrangement, it used a wandlike apparatus that captured the momentum of the user's motions (and occasionally led to TV breakage). Now we have Xbox Kinect which apparently is able to take body motions within a particular region and translate them into video-game moves. This does seem to follow the pattern of making the machine conform more to our communication patterns (in this case, our physical movement patterns) than we do to its own patterns.

Three-dimensional theater is a slightly trickier example. It may be more natural to perceive images as three-dimensional, but it certainly is not natural to have to watch everything through a pair of glasses. Notice that I say that with authority, but I do bear in mind that I am among those people who (at this point in my life) observes reality through glasses all the time. Certainly we are able to perceive objects in two dimensions as easily as we are three.

So I'm going to go back to the question of whether the iPad interface or flexible electronics interface is more "natural." I'd argue that neither one  is, really, more natural than the other (if anything, the flexible electronics interface is less natural). The touch pad interface uses the metaphor of a flat desk space with folders laid out on it, where one can influence objects by poking them or metaphorically putting one's fingers into a small opening and pushing outward to expand that opening; the flexible electronics interface uses the metaphor of (from what I've seen) the yoke of an airplane, pulling or pushing to move inward or outward, etc.

Last, but not least, let's compare this to the book interface. I'm not sure anyone would argue that listening to the story read aloud would be a more "natural" way of experiencing it. Certainly that's the way very young children interact with stories - but are the pictures in their books less natural than those of their movies? Maybe they are, and that illusion of no intervening mechanism is what makes us perceive something as "natural." The book is in a sense a metaphor for the process that the writer went through in writing it, if less so now that so many of us don't actually write on real sheets of paper, but on metaphorical sheets of paper made out of light. Still, our eyes follow the visual representations of words along the same path that the writer used in writing them out. The stack of paper sheets becomes representative of story time, the magnitude of the story measurable in weight and in width, and the reader can judge "how far she's read" with a glance. It's just another technology, with another type of operating system. It's been around longer. Because it is a three-dimensional object that can be manipulated with hands, it has a naturalness that the electronic version does not have.

In the end, it's all a question of learned behaviors, familiar objects/tools and agreed-upon metaphors. The child who tries to make a magazine behave like an iPad doesn't think that the magazine "doesn't work," he merely observes that despite superficial similarities, the two don't work in the same way. Any judgment of the magazine's "failure" comes from the adult observing, not from the child himself. Replacing one metaphor with another doesn't necessarily lead to an increase in "naturalness." If we make the user experience simpler and more similar to things the user already knows how to do (like speak, or manipulate objects on a desk) then that frees up some cognitive effort to be used elsewhere. That might be valuable. Or it might be unnecessary - humans have been learning and automating complex behaviors for a very long time.

It's something to think about.