²

Part one; TUI & Metaphor In HCI

In the beginning of the semester, Joe and I identified our research cluster as tangible user interface. Our interest in this field was to look at new methods of interaction that bridge the gap between the digital and physical realms. We began with with researching tangible user interfaces, which are interfaces where users can manipulate digital information with physical objects, and because physical objects can have familiar properties, they are often metaphorical in nature. This method of interaction, and an element of personal interest, directed much of our initial line of research to the place of metaphor in interaction design.

We began by looking at the most common metaphor in interaction design, the desktop metaphor. The desktop metaphor has been a highly successful instrument in facilitating HCI, but in it’s ubiquitous state, it has become a burden on user interface design, prohibiting it from evolving into something more efficient by being inflexible to development. The desktop metaphor was created from the early 1970s work at Xerox's Palo Alto Research Center, and was not designed to address today's computing needs (Tristam, 2001).

Due to it’s limitations, there is obvious danger when using a metaphor for user interface design. Metaphors can often mislead the user and encourage false expectations (Mohnkern, 1997). Saffer (2005) admits the abstract object has more properties than the more concrete object does and, by using a metaphor, users may neglect the properties that aren’t reflected in the concrete object, therefore using it to comprehend one thing in terms of the other will obscure other merits of the concept. Given that computers are generally thought of as tools to assist work in the physical world, a focus on mimicking the physical world is appropriate, but to what degree determines the successfulness of the interaction.

Tristram (2001) argues that the desktop metaphor has become part of our cultural literacy, and that getting users to try new interface concepts is like trying to teach Esperanto, “a good idea in theory, but for most people not worth the trouble”. Because of this, we believe changing the medium allows us to surpass these techniques and therefore force us to explore something novel. As Tristram (2001) succinctly put it “the desktop metaphor is so tightly wedded in our minds to keyboard, mouse and monitor that unless the outside package changes, the on-screen presentation doesn't have much of a chance to evolve either.”

Part Two; Tabletop Computing

Originally we thought that the right medium for facilitating this exploration was tabletop computing, with less preconceived notions about the interface to impede it. Primarily, we looked to experiment with digital information that could benefit from enhanced tangibility and a novel interface. The technological basis for this design would have been to use a high quality webcam with its infra-red filter stripped off to track the reflection of infra-red light off of fiducals (markers that can be used to track position/rotation) and finger positioning on the touch surface. However, finding completely unexplored areas with tabletop design however proved to be a demanding, and furthermore, the cost of a wide angle projector required to build a touch table would have been prohibitive. Regardless of these barriers, the concepts we produced all had a considerable effect on our research and development.



Our initial concept was PuckUI, a basic heirachy navigation system, designed to break free from traditional methods of desktop interaction. We planned to use markers on the bottom of pucks that would be read by the computer and be used to control the interface in a novel way, using rotation and other movements to enhance tangibility. This was a satisfactory starting point, but it was too broad an application to execute. In addition to this, it also didn’t deviate enough from traditional desktop interfaces to truly explore any new methods of interaction.

Following this concept, we tried to create an application for a more specific purpose. The first was an Album Table, where albums would have markers on the bottom, and on being placed upon the surface, would be read and play the relevant album. Albums could then be physically manipulated to determine the order in which they are played, or moved to skip ahead or rewind tracks. We wanted to create a concept with music because of it’s large scale medium shift from physical to digital has meant a loss in tangibility, and perhaps even perceived value. However, we eventually concluded the concept itself lacked the ability to explore the intricacy of these ideas.

Our second concept was an email application that gave subtle hints as to it’s contents importance. The idea behind this concept was to visualize the intrinsic value of files which is lost in a digital medium. For example, an email with an attachment would be shown as a package, and a quick note as a small, thin, envelope. Other visual clues would include old emails becoming dusty and discolored if ignored for periods of time. We liked the premise of this application, however as it wasn’t something we could properly create, it would limit further development.

Our third concept was to create a personalized ID card, which when place on the table, allowed the user access to his or her most important sites or documents. This concept came from a desire to explore the idea that solid objects contain more data digitally than they do physically, and what it would mean if that could be accessed intuitively. This let us to an interesting research area, of ubiquitous computing and cloud computing.

Part three; Ubiquitous and Cloud computing

The concept of Ubiquitous computing was presented for the first time by Mark Weiser, in his text on Scientific American, September 1991. It describes a computer not as a self contained object, but as multiple objects that are everywhere. Ubiquitous computing represents a paradigm shift in the way we use technology. A move beyond desktop computing, where computing is embedded into everyday objects. It seamlessly integrates physical objects with virtual environment, and assists us from the most basic to most complex everyday tasks.

Cloud computing is essentially a forerunner of ubiquitous computing, which is already beginning to impact on technology today. Cloud computing doesn’t denote any new forms of technology, nor any substantial change in content. In many ways, cloud computing is simply a metaphor for the Internet, and the increasing movement of data into it. The fundamental difference is that cloud computing doesn’t represent the same paragdim shift as ubiquitous computing, it doesn’t suggest such a radical departure from traditional computing. Cloud Computing doesn’t put the computer everywhere but, instead, it gives access to it anywhere. As such, it served as both a more interesting and more pragmatic concept for our work.

This exploration gave a whole new set of depth to our ID card concept. If information, and the means to access it, were available everything, what does that mean for personal data and human computer interaction. One question we considered was what if it is you that becomes ubiquitous, as opposed to the computer. However, due to the aforementioned reasons, we chose not to explore this through tabletop computing. Furthermore, tabletop computing was not the right medium for exploring these ideas, as the technology employed needed to be more versatile and fluid.

Part four; Augmented Reality

This is where our research led us to augmented reality. Augmented reality technologies are by no means new, however they are not as pervasive as tabletop computing, and therefore provides more opportunities in creating new concepts. Augmented reality adds information and meaning to a real object or places. Unlike virtual reality, augmented reality does not create a simulated reality. Instead, it takes a real object or space and uses technology to add contextual data to deepen understanding of it.

One of the reasons we wanted to use AR is because it “bridges the real and virtual world and so objects in the real world can be used as AR interface elements, and their physical manipulation can provide a very intuitive way to interact with the virtual content” (Zhou et al, 2008) This seamless integration could benefit many domains, but for the most part, augmented reality has been largely limited to educational books and marketing. Another aspect of AR is that, being a fairly recently implemented technology, it encourages a renewed sense of enjoyment in it’s interaction.

Part five; Augmented Reality meets TUI/Ubicomp

Because of the nature of AR, it shares a relationship with tangible user interfaces. Tangible AR interfaces are those in which "1) each virtual object is registered to a physical object and 2) the user interacts with virtual objects by manipulating the corresponding physical objects" (Lee et al, 2004). Tangible interfaces are powerful because “the physical objects used have familiar properties, physical constraints, and affordances, making them easy to use” (Zhou et al, 2008). This TUI metaphor can be applied to AR interfaces “where the intuitiveness of the physical input devices can be combined with the enhanced display possibilities provided by virtual image overlays.”

Zhou et al (2008) provide a succinct explanation of the ubicomp and AR correlation, ubicomp "enables information processing capability to be present in everyday objects. Thus, it essentially extends augmented reality interfaces to daily objects. Augmented reality change the world itself into an interface, "taking advantage of natural physical affordances to achieve a heightened legibility and seamlessness interaction between users and information". The augmented reality technology we want to create aligns well with the concept of ubiquitous computing which looks to release the user from the computer and provide a rich ambient environment.

One concept we introduced with this medium, was augmented reality postcards, or AR Cards. The idea was you could give someone a marker which then accessed a flickr stream of photos from a specified location you were visiting, like an interactive postcard. However, apart from adding another layer of tangibility to photo viewing, the concept didn't provide much of an avenue for other interface methods. Furthermore, photo viewing was not a direction we wanted to look at in terms of cloud computing and it's implications.

Part five; Final Concepts

Consequently, we continued to explore the ID card concept within this medium, but the fundamental question we came down to was if this technology is best used for intuitively and interactively accessing information, why limit that to your personal information. This lead to the creation of the interactive business card, which was both more practical for the medium, and more interesting concept. The interactive business card is designed to show clients or friends the work you've done in a dynamic way. In some way's, the interactive business card could be argued as simply a portfolio website 2.0. However, the method of interaction is much different, instead of giving someone a link, you are giving them a tangible object which then turns into a device for navigation. This physical presence and purpose give more weight to the information it contains. We provide a visualization of this concept, but due to its personalized nature, it is not our working example.



Our final concept was the augmented reality music sampler, taking some past cues from our album table, but now in an appropriate medium with much larger social implications. This concept was a way to explore the use of cloud computing for allowing immediate and immersive access to digital information within a unique but intuitive interface. Band cards can be created with individual markers and then given to friends or music producers. The card then links to the 'cloud' to load the music to be played. Physical interaction with the card allows the user to play or pause the track, and the interface provides more information about the chosen band. Currently it's a concept which works mostly in isolation, meaning you need to load a webpage to first use he app. However, as augmented reality technologies become more pervasive, it could be accessed more naturally.



In continuing with the exploration of new interaction and tangible user interface, we wanted to enable the user to directly interact with the virtual interface elements using direct physical manipulation rather than indirectly through mouse and keyboard input. So in our presentation we removed these methods of interaction. Currently however, "various challenges in ubiquitous tracking make this interaction paradigm in its infancy" (Zhou et al, 2008) so although that is how we intend the technology to be implemented, the design allows it so that they can still be present for the interface to function.

The interaction elements of the AR music player currently include 3D tracking of the information and album art avatar, and occlusion dictating the song playing and pausing. One challenge of Tangible AR interfaces is to show users how to control the motions of the real objects to issue commands, and currently this is simply text direction. Future development would include making these hints more subtle, as well as creating band posters with the marker on them, which you could then use in conjunction with an iphone, or similar smart phone, to sample the music before deciding whether to attend the gig.

On technical side, we first intended to use processing with a library designed to track markers. However after a little research we found a flash based library known as FLARtoolkit took much of the heavy lifting out of the basic marker tracking class. Further development with the toolkit on various online blogs resulted in a class known as FLARmanager. Although this class is more powerful, handling multiple markers and various events easily it is also largely undocumented. However the final design is based on a combination of this toolkit and music handling/online data handling classes. Currently the marker with the software virtually links to an mp3 playlist stored online. Ideally this would be streamed straight from lastFM or Myspace to avoid any DRM issues, but simply hosting the mp3s on a secure server demonstrates the technologies potential.



During the course of the project we feel we succeeded in exploring emerging ideas of tangible user interface, augmented reality and cloud/ubiquitous computing. We believe we have created a basic interface which is able to address each of these components, and whilst the current implementation could definitely be developed further, it's an exciting amalgamation of technologies which has sufficiently demonstrated our theories. We have managed to overcome the technical limitations that arose due to current commercially available technology.