Augmented and Virtual Reality: Definitions and History [307038]

In my dissertation, I approach the field of Augmented Reality from different perspectives gathering definitions and delimitate it in categories of user engagement and not only. Being a new technology developing each year it is rather hard to have a fixed definition. But briefly AR can be described as a [anonimizat] 2D and 3D [anonimizat]. Augmented reality is different from virtual and mediated reality because the first one is creating a new whole world and the second is intervening changing the physical objects. [anonimizat]-generated images over the environment.

AR also has a [anonimizat]. But the first prototype created by Ivan Sutherland with his students in 1960 and the term “Augmented Reality” was coined by Thomas P. Caudell and David Mitzell in 1991.

Augmented reality facilitates new ways of expressing concepts revealing innovative ideas and in the field of art have its own definitions and categories as well subdivisions. Artists are approaching the new technology pushing boundaries and evolving being independent of the old curatorial proceedings and spaces.

AR applications have also different levels of interactivity and software which influence the way it can be manipulated by the users and creators. The five major categories of gesturing are: fiducial (fixed), planar, locative (GPS), environmental/spatial and embodied/wearable. The fiducial and planar are based on image recognition but the second one is used more for print. [anonimizat]. [anonimizat] a combination of both. The software is analyzing the real space and creates a 3D map with different algorithms and is applying the virtual object into the environment. And the device must recognize it in the digital map created onto the physical space. [anonimizat]-fiction films.

Artists nowadays are using digital media to protest and showcase their statements against injustices created by the elitist society and capitalism. ManifestAR was created by Mark Skwarek and John Craig Freeman and is one of the AR applications which gives everyone a way of expression with different movements. One example is #[anonimizat]. Another movement was occupy MoMA where different AR creators were attending their digital works into museums showing that art is not just for privileged curated artists.

[anonimizat] a [anonimizat]. There are different grades of immersivenesss and interactivity depending either on the artwork or the implementation of the application. Sensorial immersion is one of the most encountered and augmented reality art is almost always trying to embody and convince the viewer that the digital object is placed in the physical world.

The users can be simply performative observers who use just mediated devices to see the digitally augmented world where the interactions are happening inside the frame of a smartphone and not in the real world. But they can be engaged to participate interactively where they manipulate the narrative or even get involved to become co-creators and leave on the digital platform in different GPS location from different 2D writings to 3D sculptures.

But AR is not just for artists, nowadays technology became much more sophisticated and accessible for everyone. Social media platforms are using the technology for entertaining and we can see how augmented reality became mainstream in the past few years. From Instagram and Snapchat applications for selfies to Pokemon Go and Ingress games, AR is now more relevant and consumable than ever and it will grow exponentially together with the more and more evolved technology. Personal wearable AR is not so far from our reality and now is helping humans to learn and comprehend spatiality in different industries such as architecture, anatomy, mathematics and education in general.

My practical dissertation work contains five digital printed illustrations which are augmented through the Artivive application. Artivive is an active creation application from the perspective of consumption which let the users create their own content and share it on the platform. I named the project ARvisceral because of the process I choose to create the content. In need to express my subconscious, I was concentrated entering a zone almost as a meditation having my mind in the moment, every glimpse at a time, to create the movements of my avatar which experience different simulated forces of physics in the 3D program, Maya.

After that, I started to interpret them creating different elements and the kinetic environment in After Effects. The printed illustrations are as an explanation giving hints of what it will happen in the digital environment. In this case, the viewer is just a performative observer using a device to discover what will happen, the interaction being in the digital media between the camera and the recognize illustration by the software.

The translucent gleaming avatar is having a dual personality which is either tranquil or destructive. In the augmented illustrations “Cocoon” and “Creator”, the character is serene having slow movements floating into space and is fabricating digital matter. On the other hand, in “Eraser me, baby” and “01110011 01101111 00100000 01110011 01100001 01100100” which is translated as “so sad”, the avatar is impulsive and destructive crushing and erasing each element is in front of her/them. The centrepiece, “Cotton Candy” with the composition as a circle, represents the aftermath of this reality, the character is swallowed by a cotton planet transformed in different particles being the victim of entropy as all of us.

Augmented and Virtual Reality: Definitions and History

The phrase “augmented reality” is a modern creation, coined in the early 1990s by Boening research scent Tom Caudell. “For Gene Becker of Lighting Laboratories Augmented Reality is a technology, a field of research, a vision of future computing, an emerging commercial industry, a new medium for creative expression.” But it is difficult to have a fully formed definition because AR is developing technology so did not yet reach its full potential. Because of this fact, the definition of AR has gone under a lot of modifications depending on the method of implementation and context.

The AR system is enhancing the user’s perception supplementing the real world with 3D virtual objects making them interact in a way they seem to coexist in the same space as a “real world”. A short definition of AR, proposed by Azuma in his 1997 survey paper, must have three properties: to blend between real and virtual objects, in a real environment, to be real-time interactive and to be registered in 3D. And the AR to be in accurate alignment of the combination of real and virtual objects. The last property refers to the risk of compromising the “illusion” that the virtual objects exist in a real environment. This is why registration is extremely important and also a difficult topic of the continuing researches that are made. That doesn’t mean AR is restricted to display and add technologies, such as Head-mounted Display (HMD). Also, is not limited, AR can apply to all senses that include touch, hearing, smell etc. “For example, an AR visualization of a building that used to stand at a certain location would first have to remove the current building that exists there today.”

AR mediates between computers and humans, humans and humans and humans and computers. Humans interact with different media in different ways. In comparison with classic media consumption (read a book, watch a movie, listen to music), AR is appealing to many of user’s senses being interactive forcing you to engage in order to gain the experience that it provides. AR can be applied in art, medicine, entertainment, education, medicine and many more.

It already has been implemented and researched in various industries such as gaming and retail and has gained an interest in the tourism industry. It was created criteria for AR that the interaction is with the immediate surroundings, connect real and virtual objects and register them. And it was taken the concept further defined AR due to the ability to share and exchange location-based information in the immediate surroundings.

AR vs VR

Virtual reality places the user inside a completely computer-generated environment but augmented reality present information that is directly registered to the physical environment. With AR in the user’s perception, everything appears to become part of the real world. AR goes beyond mobile computing merging the virtual world and the real world, both spatially and cognitively.

Augmented reality is changing the vision of the physical world by overlaying the digital one over it. This can enhance and help the awareness of humans without altering their perception, but just having additional features.

Figure 1 “Reality-virtuality continuum”

Augmented reality can’t be a virtual reality because the second one is another whole abstract reality. Is made entirely into the digital world with the power of technology and computer graphics. And the user is usually immersed in this new world with all the senses. Augmented reality, on the other hand, is adding new virtual features into the physical world.

Augmented reality is not mediated reality because the second one is altering the physical reality by either covering or subtracting the real objects. AR is just adding new digital information into reality, but it’s never changing it.

-Classifications- Reality-Real Environment, Augmented Reality, Augmented Virtually, Virtual Environment

-VIRtual reality, AUGmented reality, MODulated reality (modified DIMinished )-MEDiated reality

Figure 2 “Mixed reality with mediated reality”

A lot of people are confused also by associating the visual combination of virtual and real elements with the special effects movies. Augmented reality appears in movies as well using the same technologies as the applications but it’s lacking the interactivity.

Taking in consideration the most widely accepted definition about AR where AR combines real and virtual, it is interactive in real time and registered in 3D, the user can at least experience an interactive viewpoint, and the computer-generated augmentations in the display will remain registered to the referenced objects in the environment. This requires three components: a tracking, a registration, and a visualization component. And everything stored in the information about the real world and the virtual one. The real world is a reference for the tracking component determining the user’s location in the real world. And the virtual one is the contented used for augmentation. They need to be registered in a coordinate system.

“While opinions on what qualifies as real-time performance may vary depending on the individual and on the task or application, interactivity implies that the human-computer interface operates in a tightly coupled feedback loop. The user continuously navigates the AR scene and controls the AR experience. The system, in turn, picks up the user’s input by tracking the user’s viewpoint or pose. It registers the pose in the real world with the virtual content and then presents to the user a situated visualization (a visualization that is registered to objects in the real world).”

Augmented Reality Devices:

– Personal computer-Webcam

-AR glasses and Head-Mounted Display

-Mobile phone, tablets based augmented reality

The past can predict the future. See-through lens design was created in the early 60s for AR which makes the user’s perception of the real world unmodified and displays the information of graphic augmented reality objects as an overlay on the transparent display, mirrors, and lenses. There are different classes of AR displays: helmet, head-up display (HUD), smart glasses, contact lenses. The AR is overlaying the data with all its graphics from the headset over the real physical world having a projective display, this technique is named projected AR.

The head-mounted display (HMD) is obstructing the view-blocking the real world but the user sees the view of the wold from a front facing camera in the HMD. This technique restricts the field of view and it’s the closest model of mixing graphics overlayed into a video feed on a 2D view camera.

Technological Realities

“Immersive reality is a multidiscipline multi-labelled and massively confusing collection of technologies, applications, and opportunities. It or they go by many labels.”

Reality can be:

Table 1 “Reality has many names”

It’s hard to sort out technologies and their conflicting names but it is mandatory for understanding to put a label, even though it will change shortly while technology is evolving.

“The Metaverse is a collective virtual shared space, created by the convergence of virtually enhanced physical reality and physically persistent virtual space, and is a fusion of both while allowing users to experience it as either. The term came from Neal Stephenson (1959) science fiction novel Snow Cras, (1992) where humans, as avatars, interact with each other and software agents, in a tridimensional space that uses the metaphor of the real world.”

We’re not quite at the step to use in everyday life augmented reality, virtual reality, and mixed reality head-mounted displays. But inventors, industries, and consumers recognize we’re getting faster than ever before and the way we interact with our devices evolved like never before.

New media makes possible of reinvented and original ways of expressing concepts. Augmented reality has a big potential to reveal new ideas with its own limitations and emerges with innovative methods to do so. Because online life can have consequences into the real one, now the digital world is overlayed over the physical one being converged. Nathan Jurgenson, a sociologist and a social media theorist proposes “an alternative view that states that our reality is both technological and organic, both digital and physical, all at once. We are not crossing in and out of separate digital and physical realities, a la The Matrix, but instead live in one reality, one that is augmented by atoms and bits”

With the rise in popularity of mobile phones and social media, the mobilization of the masses in physical spaces are increasing all over the globe. This new political movement is linked with augmented reality. One such example is Occupy Wall Street, where different artists used this technology to overlay their protest images whenever they could not do it physically.

However, Alex Gibson, a leading commentator in AR, does not believe in the theory that two realities are in a crossover, but rather that they are just influencing each other constantly. They are interacting but the atoms and bits are two different distinct forms in two parallel universes and they are, at best, just blended.

Our perception of seeing and using technology and the method of input is evolving. We moved from using a keyboard to the mouse, then to the touch. The interactions will evolve as well, becoming more complex. We will soon use our movements of the head, hands, our voices, or our eyes, as eye-tracking (technology it’s already being used for special cases of disability). AR will break these paradigms and invite the user to use these natural movements abandoning the touch screen in favour of virtual reality gesturing.

Categories of engagement with the user

Augmented reality applications and games have different levels of engagement with the users especially in the creative field varying from passing consumption to the active one:

In the passive consumption (user engagement none) the AR content can’t be changed by the users the physical object is predefined and has the same form for everyone. On the next step, the augmented reality application or game have also predefined content and real object but is changing the interface based on what the user desire. For example, it can be an application that it’s showcasing an art gallery but it’s experienced differently by everyone based on their own setups. In the third category, just the physical object is predefined but it can change when the user is watching it. One example is where every experience can be different depending on who is watching it. The software is memorizing the preferences and it’s personalizing the content for each individual. The fourth step on engagement is where the physical content is not entirely predefined and it can give new content in augmented reality. An example is an application where is placing your digital photographs between the physical ones in an exhibition. In the fifth, the content is still predefined but is constantly expanding with the help of users. For example, an application that owns an archive but the users can add new photographs to the server making them co-creators. In the sixth step, the content is predefined but the users can personalize and augment it. For example, it can be an application where the user is playing transforming forms in the AR changing and adding on the “physical” objects. On the seventh, the content is not predefined and it is changed by the users. For example, they can create their own work and expose it for everyone to see (Artivive).

History of AR

Even though it feels like augmented reality is just at the beginning of its journey, it has an interesting history worthy to mention. Henry Dircks invented in 1862 an illusion named “pepper’s ghost” and this technique is used even today with high-quality projectors and advanced technology. John Henry Pepper popularized Dircks’s technique and incorporated it in theatres. This is why the illusion remained with this name and not with its inventor. It was first used for Charles Dicken’s play “The haunted men” having a big success. It works by reflecting the original being lightened through a mirror. The reflection seems to be tridimensional and now is used to give the holographic futuristic effect (especially on stages) using led panels and projectors but it’s not actually one. A hologram has computer generated 3D parallax that allows the viewer to walk around and see it in 360 degrees.

Figure 4 “Illusion Pepper’s Ghost”

But conceptually AR maybe comes from ancient Greece, in the book of Plato where the notion of subjective perceiving of reality is given by a few men seeing shadows of people from outside’s world imagining what is actually happening there. In 1420 the engineer Giovani Fontana writes a book where he describes a magic lantern that can project fantastic images and not only on the wall. His idea was actually put in practice by Carolina Cruz-Neira being a pioneer of virtual reality and interactivity. She created CAVE in 1992, a virtual reality software system, an advanced technology and she designed an AR room with 3D virtual images on the walls that immerses the viewer into a surreal experience.

But the first AR prototype was created by Ivan Sutherland with his student from Harvard and Utah Universities in 1960. Several years after ARToolKit software appeared and it was available for everyone interesting. In 2001, NASA Ames Research Center finished their researched and created the biggest international symposium for the industry about Mixed and Augmented Reality (ISMAR) helping everyone involved talk about their knowledge and doing networking about the matter.

Augmented reality took much longer to evolve having much more technical technology difficulties than virtual reality. Because of the obvious reason being much harder overlaying a virtual matter into reality than creating an alternative one. Combining the virtual with the real world, using a tracking device for the position, the components used for creating an AR system remained more or less as in the 60’s: the tracker, display, the software and graphics.

In the 80s and 90s mobile devices were created and the Massachusetts Institute of Technology continued the research for AR. In the 1990s because the technology advanced and the devices were small wearable computers was introduced. An example is Apple Newton MessagePad introduced to the public in 1993.

In January 1991 the term of Augmented Reality was invented by Thomas P. Caudell and David Mitzell, two scientists working at Boening Computer Services Corporation at the department Research and Technology. They were working at developing a see-through, head-mounted AR prototype display.

Still, in the 90s, the technology wasn’t sufficient enough for creating an AR technology small enough for day to day basis. But GPS technology was invented and the navigational system helped the user to get to the destination with an audio guidance overlay. After this, the software engineers created an image overlay prototype for mobile with 3D graphics directional advice.

Because of augmented reality technology advanced and it was seen as a specific tech, in the late 90s there were some conferences about the matter: Designing Augmented Reality Environments Workshop, Symposium on Augmented Reality, the International Workshop and the organization of Mixed Reality System was created.

Augmented Reality Artworks and Projects

The AR art movement evolved with the technology which became over the years increasingly user-friendly. In the ‘90s the artist needed knowledge in coding and also equipment of googles and webcams. Now it’s easier than ever to generate your own artwork into the world with all the mobile and tablet applications including the geolocation AR-based media. After the commercialization of AR with beauty selfie apps, Pokemon Go and Google glass, now it has the challenge to be established in the art world and seen as avant-garde.

But at the same time, when we talk about augmented reality it is necessary to start mentioning first about its integration in art. AR is a visual statement having its own aesthetic in the digital world. And also it is a mechanical technological fairly new tool of expression used in different ways and branches of art. There are many types of AR art expanding in different approaches. AR artists are always trying to push the boundaries and find new ways to use technology. Augmented reality is constantly changing and evolving being explored and in our times is a turning point for all artists that are using it. What content humans are distributing through this new technique represents just the subjective view of how they are perceiving the world. Just a few decades ago digital art was taken in consideration, the art world is full with classic mediums of expression such as painting, sculpture, some installations, music, dance and dramaturgy where the viewer was just a spectator and nothing else.

AR artists can play with dynamic matter and also in their performance. The AR experience is not in galleries but on mobile devices so this is why it’s usually independent of classic curatorial proceedings being moved into the digital space and having much more freedom. “Augmented reality art, as a new media subset, distinguishes itself through its peculiar mechanics of exhibition and performative re-contextualization. It allows the artist to translocate the borders and constraints of the experience from physical to virtual, expressing the piece onto spaces in a way that is independent of physical or locative constraint, yet still tethered to the real world.” AR art is now more relevant than ever because of the complex ways of expression and also the easiness of finding and using it.

Categories and examples

The two most defining interactions of the public with AR artworks are the gaze and gestures. The gaze is the action of the viewer which is practically seeing the art manifestation through different devices. And the gesture is the way an augmented reality software is using the technology to showcase the work. There are five categories of gestures: fiducial (fixed), planar, locative (GPS), environmental/spatial and embodied/wearable. One application or artwork can inherit a multitude of gestures having combined more than two technologies.

The Fiducial is one of the earliest forms of technology in AR. It gives the dimension of the subject and the camera sees where to overlay the 3D object into the real world. One example is Reactable made in 2005. In a circle, there are AR buttons that react to the participant's touch manipulating the sounds. Another two examples of Fiducial AR is Miku Hatsame at Berlin’s Transmediale festival in 2016.

The Planar (print, poster) AR is much more advanced by scanning usually overlaying into a print media. This kind of AR is used usually in marketing campaigns because of the nature of print. One example is Esquire Magazine cover where the actor Robert Downey Junior is talking with the viewers presenting a 3D space into a 2D magazine.

The Locative GPS based AR is one of the most complex gesture because of the dynamic relationship between the user and location. One example is We AR in MoMA by Mark Skwarek, John Craig Freeman and others where they entered in the museum without permission. If you used the app inside, you could see the artworks overlapped with digital 3D media.

Environmental AR has a number of problems with spatial recognition because depends a lot of light, weather, time of the day or changing objects. This is why a lot of artists chose to do their work in closed spaces. Ascension by Richard Humann mixes environmental and planar AR. At the Venice Biennale, the user could head his phone to the sky and see the constellations interpreted as in mythology but in a contemporary way. Another good example is Exit Glacier Terminus by Nathan Shafer. In Alaska, the user can point its phone or tablet to a location and see how big the glaciers were and the change made from 1978 to 2013. He also uses augmented 3D glaciers to remodel the shape they had several decades ago. It is built on location so the user needs to go and see the glaciers because the creator thinks the geolocative form of augmented reality is cheating by downloading everything beforehand so it’s not experienced as it should be. Also, there is a possibility the GPS of the mobile not to work in such aeries because of the wifi connectivity.

Figure 5 “Exit Glacier (2001 Terminus) by Shafer (2013), five different AR versions of glacial termini were built on location at Exit Glacier, and this is the terminus from the year 2001”

Embodied AR evolved from the computer screen to phone device to the body and finally into space through contact lenses. This nowadays is seen more in dystopian films and television than reality. One example is the short film Sight by Eran May-Raz and Daniel Lazo from 2012. The users had implanted the AR technology in their body using it for everyday mundane tasks and also using applications for meeting other people. Being a dystopic genre it shows how someone can hack into your perception and brain and control your thoughts.

Project:

Early Projects and semi-AR installations

The earliest artwork that used digital media and location is “ I am sitting in a Room” created by Alvin Lucier in 1969. He recorded himself talking in a room and then plays it recording it over and over again until the sound has the frequencies of a room tone. The room became more of an abstract augmented concept than an actual location. This work can be seen as a promoter of the conceptual thinking of many AR artists and not only.

Another promising early installation is “You’ve got bugs” from 2006. Having just a screen and a camera broadcasting the viewers pixelated, every movement was tracked and overlayed with bugs. The viewer is interacting with the AR screen being seized by small black pixels on the screen multiplying following them.

Figure 6 “You’ve Got Bugs! Screenshot and a picture of installation with and without audience member”

MONOLITT is a sculpture that generates colours made by Eirik Haugen Murvold and Syver Lauritzsen. It can be interpreted as environmental AR even though is not using digital media in an obvious way. The colours that appear from the installation are influenced by the general mood on social media. It’s taking all the data from what people are posting, for example on Twitter, and it’s transforming it in pure paint in real time.

Quick & Easy Recipes for Disaster is an AR Installation made by Thomas Storey who uses 3D objects as food generating random recipes. The viewer sees the installation as an average cooking website blog. They are served virtually into a plate and you can take a picture and share it on social media. This artwork is as a response to social media, looking continuously at the screens, the individual forgets to enjoy the surroundings, interactions or simply how the food tastes. Nowadays we don’t need to eat anymore in real life to exist on social media, you can post a digital food picture and is as if it’s the real one.

Interactivity in AR

AR using an intermediary apparatus between the artwork and user and in this way is having its fourth wall of viewing being firstly mediated by a technology in which the image is altered. So we can name the viewer just as “performative observers” because the slightest interaction is happening actually inside the frame of the virtual space, and he can just move the device to observe it in different ways. But the way artists are using this technology can become interactive engaging with the users. I will display in this chapter from the least interactive with the viewers to the most:

Mapping Ararat uses Augmented Reality to commemorate the past through digital monuments and also uses digital maps to show the path taken by Major Mordecai Noah which wanted to found Ararat but failed. Kaplan and Shiff displayed what his intention would look like through image augmentation and the viewer can see synagogues, memorials, banks, graveyards, different shops etc. in Grand Island. This can be problematic because it is shown something that it could have been but never was because all the repression and never having the opportunity to settle throughout history. “Thus, the user-visitors performative actions reflect a personal sequencing of the story in piecing these images together. Ultimately, this is a visual story-telling project that can have unlimited numbers of sequencing. Each visitor walk can be told or visually viewed differently.” The project is designed as a tour for a utopian vision of the present.

Aurasma is an application which can accept fractal forms and the software also use image recognition and localization for the augmented objects. Dragoș Gheorghiu and Livia Ștefan choose this platform to mark prehistoric places with specific patterns like in the ancient culture Vădastra. When the user is pointing the phone on the specific patterns in a specific location, a 3D prehistoric house and characters appear. The viewer is experiencing a virtual tour about the history and the character is doing different things such as decorating a pot with the same prehistoric pattern. The project also includes real objects scanned in 3D such as the Vădastra patterns and vases. The creators combined science and art to illustrate the vast prehistoric history from the location.

Crisis 22 is a participative narrative work where the user is tracing the trajectory to see the story throughout the city of Ottawa. But besides being participants and explore the events happening into the cyber world, they can’t change the narrative. The only engaging event is searching for a specific space.

Another example of an augmented reality artwork that tried to use interactivity is The Haunted Book by Camille Scherrer. He used a book, a camera that was connected to an augmented device and projected onto it the digitally animated work. The user could turn the pages and see different short pieces of AR but wasn’t engaged fully performing as a participant into the artwork.

At the Venice Biennale from 2011, Will Pappenheimer came with an interesting approach: He transformed his bufo toads into psychedelic ones through Vita Flaneurazine which is a potent programmable psychotropic drug used in Second Life. Their skin had colourful imagery and you could found them everywhere into the city and growing in number around the pavilions in the Giardini. If you touched them on your screen, they were releasing the VF effect on the entire display.

Figure 7 “Colony Illuminati by Will Pappenheimer/Virta-Flaneurazine (2011). Augmented Reality. Colony group on Giardini main concourse”

Sky Petition City is an application created by Will Pappenheimer and Zachary Brady on Manifest.AR platform and lets the user to draw or to write on the sky with a cloud-like line. Pappenheimer explains: “It has become evident that Washington institutions no longer listen to the voice of the people. We need something we've never tried before.” Everyone can express what they think on important national institutions from Washington. This is an informal political statement that allows civilians to express their feelings on the blue sky. Every old message once in a while are replaced by new ones.

Pheonix Toews engages his users through the app Pyrite with sculptural digital installation to a new another whole universe. Everyone can participate by creating and find other sculptures anywhere in the world breaking down the fourth wall. And the application Manifest.AR does the same: you can upload virtually your artwork in any gallery you wish. Skwarek destroys the idea of privileged art and artists and invites creators willing to display their work in such “sacred” art spaces. This can be defined not only as interactive participative AR artworks but also co-creation installations because the public work together to create new digital worlds into the real one. But this depends on the willingness and knowledge of the participants to interact with the applications or not because of the difficulty augmented reality technology can give.

Feminist AR

Riot Grrrls was a feminist and queer movement from the 90s which is still influencing the art world today. Now it became an augmented reality application with the same name created in 2017 by the School of Art Institute’s Virtual Installation under Claudia Hart’s supervision. Their goal was not only to augment some of the works but to elevate the movement through new media and make the viewers think analytically about it.

One of the works “Hedda Gabler” by the Francesca Udeschini is reinterpreted by Molly Zuckerman. She is making it black and white with bitmap dots. The dots are transparent and the original piece can be seen through it.

Beier Zong chooses Metropolitan by Mary Heilmann and he tried to create the 2D piece into a tridimensional one. The user can see the augmented 3D work moving in different geometrical lines and forms adding to a sphere which is almost breathing. There are dispersed four forms interpreting the six black and white squares with the red-pink lines integrated simultaneously and each of them is having different styles.

Figure 8 “2017 Rendering of Mary Heilman augmentation by Beier Zong over Mary Heilmann’s Metropolitan, 1999. Oil on canvas; 75 × 60 in. (190.5 × 152.5 cm). Collection Museum of Contemporary Art Chicago”

A love letter to a Violet it’s been augmented by Ellen Burkenblit giving a different and introspective meaning. The viewer can see different females taken from old films fighting and it has audio with different speeches. Jonatan Martinez gives homage to Tomma Abts augmenting bright colours coming from her work propagating to the entire space lines of bright blue, red and green.

The students wanted the viewer to spend more on the female artist’s art pieces with this augmentation than the usual of several seconds per work. Claudia Hart comments on her student’s work and her effort “Riot Grrrls App Show was, to me as conductor and choreographer, an improvisation—both a pleasure and a surprise. I discovered that augmented-reality apps are a kind of museum in a bottle, a device to make curatorial connections, open up different worlds, and create dialogues between the art and the visitor and among museum visitors. I couldn’t be more pleased with the result and more grateful for the experience.”

Google Glass Art

There are different types of immersion in AR which are bearing away from the VR ideals. For example, sensorial immersion is one of the things AR art tries to embody with the real world altered with a digital one. The viewer can be immersed and almost convinced with a really good integrating augmenting software that the digital element is there. Social immersion, on the other hand, is where people are connected to place virtual objects with meaning.

Samantha Katz created Gallery Glass where she gathered thirty artists from Brooklyn to perform while wearing Google glasses. After, on the Youtube channel with the same name posted edited videos where the viewer can see through the point of view of the artist while creating. Over it added interviews with them where they explain about themselves.

Molly Crabapple created a drawing with Stoya while wearing Google glasses. The name of the work is “Glass Gaze” because she was looking at the model live through the glasses and draw her. This is an example where the artist is using AR devices to create art via classic mediums.

David Datuna used the application BrickSimple to create an interactive installation. From the distance, it looks like the American Flag made with small pieces of glass. But as you get closer and see the work in detail you realize that actually are small cameras filming you. The installation live-streamed the visitors through the small cameras. This can be interpreted as a comment on how art can watch back and also about how small cameras from our devices can be turned against us.

Figure 9 “Viewpoint of billions by New York artist David Datuna at Art Basel Miami (DatunaGlassfeed”

Immersion-AR seen more as a concept

Integrating digital form into the physical one, AR can be seen also rather conceptually than just a form of technology. As a user, you can interact as the physical form with the digital one become a part of the performance, some artists that convey such immersive works are Klaus Obermaier and Gideon Obarzanek.

Kim Vincs, John McCormick, Alison Bennett and performer Steph Hutchison did a dance scene, Crack-Up, where they used a Kinect to show the depth of the image filmed in real time of the performance. This digitally augmented modified image of the skin was displayed in the background and the performers could interact with themselves but in the form of bits.

Alison Bennet also did a performance project Shifting Skin having a series of photographs with parts of bodies having scars and tattoos. When the viewer sees it through a smartphone device they see an overlay image of the photograph but scanned and highlighted coming out from the surface. The user can move the device to see the photographs in different ways.

Figure 10 “Untitled (camera) from the series ‘Shifting Skin’ 2013 by Alison Bennett”

John McCormick created Recognition and it was performed by Steph Hutchison. As in Crack Up the performer is interacting with the 3D almost liquid iris eye were is following its movements. And with the help of artificial intelligence, the object is doing its own choreograph by learning past motions. So after, the software generating the 3D image can perform alone creating also newly improved forms having already all the information processed and is adding and reproducing the knowledge every time is performing.

AR and ‘classic’ mediums

Usually, digital work can be constantly modified or changed entirely as time passes by the artist compared to the classic mediums. This gives a new concept challenging the classic interpretation of art, now the creator can have the whole control over the digital medium even after if it was already showcased to the world. AR can be also displayed in classic mediums such as painting and sculpture. Hidden Realities is a series of six paintings created in 2013 by Vladimir Geroimenko using 2D overlayed images in the application Layar and 3D ones in Junaio. All of them represent different styles and techniques as well as the digital forms overlayed. AR artists use print or sculptures to be tracked by the applications having palpable content.

First work is named “What lies underneath?” being a tribute to analogue black and white photography being made digitally in an impressionist technique. The second one, “The Half kiss” is the first AR artwork ever sold, the price was 19.99 pounds. “This is not a phone” is a digital photo with an iPhone given an allusion to “This is not a Pipe” by René Magritte. And where the viewer is seeing it through the camera a 2D image with a Swiss army knife appears. It can be a comment of the real purpose of a smartphone and also the dualism of the idea that the device is used to see a representation of itself. The fourth painting name is “Augmented Quote” having just half of a text and seeing it through the device is completed in an ironical way. “Four keywords lost in augmented reality” is presenting a virtual environment where just with the AR the viewer can find the fourth key. The last artwork is “The hand of Moscow”, in the 2D digital painting there is the Red Square of Moscow and a 3D hand is emerging from it as a representation of the cold war.

Vladimir Geroimenko realized also augmented reality sculpture, one example is “Enterprise Jigsaw”. It was created in 2011 and is in the United Kingdom, Plymouth in the City Jigsaw Garden. The viewer can experience the sculpture tridimensionally all around and the puzzles have such forms that can be read as an enterprise.

Another “classic” art that was introduced in augmented reality is graffiti. In its essence, it can be a form of AR adding creative images onto the buildings. B. C. Biermann organized the “Heavy Project” with original illustrative murals. Using Re+Public AR application the viewer can see the initial image modified being animated and is spreading all over the environment moving into the tridimensional space. Biermann uses interactive works because the forms can vary, the viewer having different experiences each time and also some pieces can be seen in the dark, being really hard for the majority of applications software to give such experiences.

Figure 11 “Heavy Projects, How & Nosm mural augment (2012). Full view [left], screenshot [right]” (Gwilt, 2018)

Shannon Novak suffers from synesthesia, he can feel and see colours and sounds where they are not actually there. He uses real places and objects to overlay his AR works as he perceives it in real life. He overlays animations with geometrical objects and colours over them with music. “Manhattan Phrase” is one of his work were into an orange door he displays lines and forms as a comment of history of music in that area.

But the not only artist can experiment with augmented reality doing digital graffiti. Now everyone can use the app to draw on buildings. There are several applications that invite users to do their own kind of work on different monuments and important building as a signature. Autography is one of them, unfortunately, it works just with the Cathedral from Florence but it’s a fun way to leave a mark in a place you visit or you live. Everyone can see it and explore it.

Artists:

John Craig Freeman

John Craig Freeman is an artist that uses technologies to influence individual’s lives and communities. He is one of the founding members of Manifest.AR and has and showcased artworks around the globe.

“Things we have lost” it’s a project based in Liverpool and Los Angeles created by John Craig Freeman which introduces 3D objects with the help of GPS coordinates around the city. He created a database after asking random participants “What have you lost?” They were connected to an EEG-reading brainwave sensor and the software detects their answer and then the virtual object appears in front of them. You can see around the town different lost items or concepts such as trains, empires, women’s rights, keys, boats, hello kitty stickers, the time, and a dodo bird. This project is interesting because the answers are the most sincere possible with the help of the technology used.

Frontera de Los Muertos is an AR piece showcasing every Mexican immigrant that died trying to pass the border in Arizona. In each play where human remains were discovered, he overlays digital skeletons to remind people how hard can be the process of trying to pass on the other side of the border. And Mexican people are still doing it wanting a better future for them and their families. The user can download the app Junaio and it will give exact locations through the GPS.

Figure 12 “Border Memorial: Frontera de Los Muertos by John Craig Freeman, On the road to Ajo along Highway 86, Arizona, 2013, Augmented reality public art”

In West Belfast, the Catholic and Protestant communities were in conflict so this is why they have “peace” lines which are actually walls that were first built in 1969 after the riots in Northern Ireland. At first, they meant to be just temporary but they are still up. John Craig Freeman tried to unite the communities through his nine AR works situated in different locations along the line. “The doorways are integrated into the physical location as if they existed in the physical location.”

Virtual Russia and Virtual China is a two-part project created in 2017 which is using geolocation in Sankt Petersburg, Wuhan at National Historic in Massachusetts. The viewer can see through a metaphorical portal and “teleport” themselves in China or Russia. They see digitally just mundane normal events. Freeman comments about his work: “During its early history, the Port of Salem conducted trade with both the Baltic and China. This history is relevant today as the world struggles to reconcile the discord between globalization and the rise of nationalistic protection and isolationism. We tend to think of globalization as if it were something new.”

In the USA there is a big problem with gun regulations and with the school shootings. There is an impressive number of incidents and deaths each year. J.C. Freeman did an AR artwork in front of the United States Capitol at Wahington D.C. “School Shooting eMorial”. In it, you can see the Sandy Hook School sign, 20 backpacks representing the deceased students and 6 apples being the professors. As you approach the AR you can hear the phone ringing as the parents started to call their children in panic without knowing they were already dead. Public discourse has been relocated to a virtual space that encourages exploration of mobile location-based media in public. Moreover, public space is now truly open, as artworks can be placed anywhere in the world, without prior permission from government or private authorities —with profound implications for art in the public sphere and the discourse that surrounds it.

Paseo Portal, Securing the Virtual Border was created for the exhibition ZERO1 at the Paseo de San Antonio Plaza. Through it, the public can see people they would not normally encounter on a daily basis. “ Paseo Portal, Securing the Virtual Border acts as an access point where the public can immerse themselves in virtual reality experiences documenting gentrification, the housing shortage, and working class flight in the Bay Area generally, and homelessness, displacement and migration globally.” The portal is immersing the viewer into the world of homelessness and poverty. The viewer is having a meaningful experience as a privileged citizen it can be shocking and eye-opening to encounter such digital augmented images especially that nowadays this is seen as normal.

Sander Veenhof

The patent alert is an application designed for HoloLens. It’s an anti-patent because after it’s scanning the environment and by discovering an object usually is telling the user how to avoid, how dangerous can be or not to use that object. This is a comment of all patents that restrict individuality by blocking people to use for example devices and applications from competitors. And instead of helping to broaden the freedom and imagination using technology, sometimes it’s doing the opposite. The future can look very gloomy if this will continue to influence our daily lives.

Sander Veenhof made a call internationally: “Whether you are in Australia, Asia, Europe or the Americas, anyone in the (connected) world is invited to join a globally performed synchronous dance on October 7th. A new genre of collective experience, now possible thanks to mobile technology and augmented reality: a distributed flashmob, “dancing alone together”. The artist through Global Choreography is trying to influence users to move in the physical space coordinated by a digital object. They are doing a choreographed flashmob even though they are not together in real life by following the augmented cube with their phone in the digital space. As a user, you could see how many people and from where are participating being part of a bigger picture on the entire planet, but from the outside real world each individual looked like they just are looking at their phone gradually turning around. This is a good example of how augmented reality can influence the real world and people globally.

Figure 13 “Global Choreography (Veenhof and Vogels 2012)”

Cyborg Dating is a project created by Sander Veenhof, Paul Siegmann, Rosa Frabsnap, Rudolf Burma and Paul Siegmann which allows the user to be a robot just for a moment. The application needs to be played by two people, one of them is using the smartphone being from the present and the other one is using a VR device being the cyborg from the future. They are both immersed in a story communicating but one of them is being told what to do and where to walk. After the person with the smartphone is asking a question, the other one is being told what to say by the platform. In the middle of the game after they start to know each other, the users are changing their roles each of them understanding what is happening.

Also, he created a digital augmented traffic light on the bike path with the idea if enough people will know there is a digital traffic light, they will stop and wait for the green light. This could be another example of how artists can influence humans with “invisible” objects which just some have access to them or know about it.

The quantified self is another project that helps the user to concentrate. Using Google Glass or HoloLens the technology is connected to the heart rate of the viewer detecting when is not concentrating. The augmented reality shapes will appear as a reminder that they need to stay present and focused. It is useful in lectures and meetings and not only.

Nathan Shafer

Borealis takes part from Manifest.AR being created by Nathan Shafer and Christopher Manzione. With the huge problem of pollution, the aurora borealis is not possible to be perceived with the naked eye like before in the dark months, especially in the city. So the co-creators made a way to animate it so the user can see it where it supposed to be with the device using geolocation.

Figure 14 “Borealises by Shafer and the Virtual Public Art Project (2011), digital/animated AR version of the northern lights, displayed”,

Non-Local is an AR project where the viewer is integrated into science-fiction short digital stories that together form a bigger one. It’s using the Layar application in Seattle, Anchorage and Skidegate and in New York, it’s working with Junaio. But here it’s just one story about two boys encountering an alien.

Mark Skwarek, Tamiko Thiel, Conor McGarrigle-AR in Activism

“When injustice becomes law, resistance becomes duty” by Thomas Jefferson

Activism is an old notion, the first dated protest was made in ancient Egypt when workers just stopped any activity until they got paid. In France in the 19th century started a movement which spread all over the world of occupying the streets. The mobility of vehicles or lack of it was extremely valuable politically because of the consequences in the economy. Back then was seen as something radical but from the 60s, it is a liberal practice. And as society evolved, the tools used to prove a point got as sophisticated with the time.

Augmented reality is an emerging technology which takes internet activism with its physical and digital worlds combined. The artists can place their message anywhere in the world and make different statements. Some of them started to use AR as an instrument of social change. “Technology will never be a fix-all solution to fight injustice, but it can aid activists to help level the playing field against the corrupt elite.”

Everyone nowadays can make their own statement having access to free or low-cost tools spreading their message through their smartphones mobilizing thousands of people just with a hashtag via social platforms-it is much more effective to be heard in a digital world. And now technology is expanding like never before even in developing countries, the majority of people worldwide will have access to the internet via smartphones devices.

This is why AR applications on mobile give the artists the ability to spread their perspective because there is no border that separates the viewer from the art in this digital era. AR doesn’t replace the reality, it is just another tool of expression starting conversations and it is displayed in public spaces to engage people.

The AR artists are finding real-life objects (logos, images, advertisements) targeting corporations and transforms them giving a new whole interpretation. One example of an artist who wanted to inform the population is Mark Skwarek's media artworks with the app arOCCUPY (2011). AR can make you see a lot of things that the naked eye alone can’t catch- for example, the amount of money used as a bailout for Wall Street companies which created the recession.

Mark Skwarek is an artist and activist who works to make the gap between the virtual and physical reality much tinier. He was one of the creators organizing ManifestAR, We AR in MoMA and arOccupyWallStreet movements.

The known protest Occupy Wall Street was forbidden to happened and closed by the police on the main street, most of them being detained and treated by the FBI as terrorists. Therefore the activists were forced to camp on Zuccotti Park. The AR helped the movement to spread digitally even reaching the White House. There were more than twenty-five artists working together for the #arOWS and was organized by Mark Skwarek proving that a protest can be present with the help of technology even though the government spent millions of dollars through FBI and police to stop it.

Their objective is to reveal the corruption behind the big corporations by giving an AR message over their advertisement or logo. So the purpose is already met when the viewer can be informed with what is happening behind the scenes, this statement has the name of logo hacking. Basically, the camera sees and recognize the trademark image and overlays the digital one. In 2010 was still legal this kind of use with AR but now a lot of laws have passed to regulate the Augmented Reality experience with copyrighted images, logos, maps, sequences, patents etc.

“Many culture jams are intended to expose apparently questionable political assumptions behind the commercial culture. Common tactics include re-figuring logos, fashion statements, and product images as a means to challenge the idea of ‘what’s cool’ along with assumptions about the personal freedoms of consumption”

On April 20, 2010, after an explosion was the largest marine oil spill in history in the Gulf of Mexico. After this environmental disaster, a big reaction of the activist happened, “The leak in your home town” was created by Mark Skwarek and Joseph Hocking as a response against British Petroleum. In the AR image, you can experience on your smartphone the 3D broken pipe over the logo of the oil company. It was the first activist work made with a mobile app. “Basically turning their own logo against them. This repurposing of corporate icons will offer future artists and activists a powerful means of expression which will be easily accessible to the masses and at the same time will be safe and nondestructive”

Figure 15 “Mark Skwarek and Joseph Hocking, the leak in your hometown, Augmented BP logo” (2010)

The Separation Barrier is a wall which divides Palestine from Israel. People can’t pass through easily, they need to go to different military checkpoints with official papers and usually wait around over an hour or more. The project made by Mark Skwarek, Daz Chandler, and Ghassan H. Bannoura erase the barrier and the viewer can see through it. Erase the Barrier used satellite maps images as documentation and produce 3D models and for some locals, it was the first time when they saw what is actually beyond the barrier.

After the war and still ongoing rivalry between North and South Korea, Mark Skwarek manages with the help of Augmented Reality to remove “weapons, checkpoints, fortifications, barriers, walls, and all reminders of the ongoing conflict from the Korean landscape” at their borders. This is a vision which shows a better alternative for the two countries, enforcing the message of peace, by tempering with reality.

NAMA (National Assets Management Agency) is an Irish government agency created after 2009 that bails Irish banks by removing property loans with the target of improving the economy. This attempt failed and the banking system next year still collapsed. The agency kept secret all the information from the public’s eye. Conor McGarrigle created his AR project taken from an unofficial database, “NAMA Wine Lake” with all the information about NAMA’s properties. The project tagged 120 locations of buildings in Dublin exposing for the viewer what the government tried to shield. You can identify the place by seeing through the screen the Monopoly Man figure.

Figure 16 “NAMAland in operation on the iPhone, Conor McGarrigle” (2010)

One interesting work is “Reign of Gold” by Tamiko Thiel where you can see golden coins falling from the sky when you reach NYSE with your camera. This is a reference after Abbie Hoffman and Jerry Rubin, two activists that entered NYSE and threw real money inside the trader’s stock building. Comparing both protests we can say Reign of Gold even though you need to be at that place to have the experience, it can be seen by thousands of people, the second one was behind a closed door and only the brokers took part.

Another project from 2013 which is still growing by Tamiko in collaboration with Will Pappenheimer using Manifest.AR application is Boomers Skelters. It is created to put users at ease by creating a natural environment around them. It is connected to a heart monitor and depending on how the heart rate increases, the augmented fauna is started to grow around the user. You can see the town in a natural botanical environment. The users can choose between natural fauna from that specific location or the exotic ones. The viewer sees the flowers growing or appearing each time differently, having each time a unique experience each heartbeat transforming the digital images constantly. The user becomes a participant having an indirect role in changing the digital universe. They stated in 2016: “It is in this sense that our work becomes an interventionist space for critical thought. Virtual augmentation, therefore, is not utilized to enhance or commodify objects or space, but rather to reveal problematics of the public or institutional site and memory. The virtual artwork, integrated into the actual Cartesian environment that claims a specific functional or ideological territory, reveals what is otherwise hidden, functioning not merely as a technological apparition but also as an index of suppressed social objects or strata of allusion.”

Figure 17 “Player of Biomer Skelters, and her ‘biome’. Image courtesy of the artists, Virtuale Switzerland 2015”

Gardens of the Anthropocene created in 2016 can be seen as an addition of the Biomer Skelters project. He is using Layar and geolocation to exhibit different types of animated plants. They can either be attached to buildings or streets and even on the water in the ocean. They are weird looking coming from a science-fiction movie and are inspired by plants which will resist in our polluted future. The plants are remembering the viewer if we are not thinking with the nature this dystopian future is not so far away as we think.

Applications in the Art world

Artivive is an active application created by Sergiu Ardelean and Coding Popescu in 2017 which sees augmented reality as the future for artistic manifestations. “With Augmented Reality, your canvas has the power to become a story. With Artivive your art has the power to transform before the viewer's eyes. It evolves, it warps, it reshapes itself as the moment of expression evolves. You are no longer sharing a single moment of yourself. You are telling a story.” This platform is designated for artists who want to enhance the experience of their own works and not only. The user can add three visual animated layers and six images and also sound. It is user-friendly, you can use the chat for every problem you encounter. And the viewers just need to download the free application to experience the works and also to save it and to share it in eight seconds.

SketchAR in a virtual reality application created by Andrey Drobitko and Alexander Danilin which helps users to sketch tracing over 2D images including famous artworks. It has step by step models where beginners can trace it after the phone into the real paper. But also different professional artists are using this app with Google Glasses and HoloLens to help them innovate their works: Audrey Kawasaki, Demeski J, Natalia Rak, Spok Brillor and others. You can search for different templates for beginners learning to draw or you can upload your own image.

Nowadays artists and curators can create and organize their own exhibition seeing the real space through their device in a 3D virtual form from their own home homes. The application that makes this possible is Taking the Artwork Home. The user can select each artwork and arrange them in different ways and also name each of them as the exhibition. They can share it and the viewers can zoom-in to see also the details of each artwork.

TARX is an augmented reality application which contains an archive with history photographs given by the museums for the users. The app is using geolocation and the viewer can interact by visualizing them over the actual place. Everyone can upload historical photographs into the platform to enrich it and also they need to add a description, the date and the location. Besides the co-creating feature, the users can explore cities interactively and see the past through augmented old photographs. It has a map where the visitors can explore it, they can download the photographs and the information about it and also share it on social media.

Another platform that maybe works the other way around is Blippar. The augmented reality application recognizes over 2 thousand landmarks on the entire planet. The user can reach with their camera the unknown building and on the screen, it will appear the name and the information about it. Because it’s using just image recognition and not geolocation, you can use it also in photographs to find what you are searching for. The CEO of Blippar, Ambarish Mitra explains his perspective: “Everyone is focused on placing things on top of things with AR. But computers need to really, semantically, understand reality.”

At the Mixed Reality Festival, Conrad Gleber and Gail Rubini created the application NEWzzzzz. It’s using geolocation and also recognition tool to overlay red propositions from the newspapers of Alaska’s countryside. It’s a funny way to use random words taken out of context and display them in a feed and locate them in different places.

Augmented reality entered also the world of fashion. With the help of holographic hardware now people can imagine how they will look like in different outfits. Haphazardme is an application that uses the real-time scan to create digital forms and texture over the models. The viewers can use HoloLens to see fashion shows. Also, it can be helpful for fashion designers when they want to preview their work digitally to see if it works or not before buying all the expensive materials. We can see a future where the artists can combine the real textures with the digitalized ones and express themselves beyond the logic of physics.

Figure 18 “Left: Hyperfabric, Marga Weimans in collaboration with Augment and Weimans (2013); right: Haphazardme HoloLens app (Ensemble de Solution 2017)”

AR applications in industries

AR in applications and games is a growing business especially because technology becomes more sophisticated. “Augmented reality is not only an exciting new technology but a business opportunity with numerous applications. Beyond the world of gaming and the movie industry, augmented reality is poised to become a thriving stand-alone industry. “ We can see it everywhere in our daily basis:

Mini AR experiences for everyone

Developers and artists are making filters that are accessible to everyone giving them an immense exposure. “There are museums that incorporate digital art in the form of AR, but this is the more accessible way for everyone to not just try those tons of augmented reality mini experiences, but also to share it through the application and even see it on the web.

Social media platforms such as Instagram, Snapchat or Facebook use the AR for mobile applications. There are some limitations that every AR applications still struggle with: lighting, viewing angles and no dynamic lighting having problems with powerful shadows. Face tracking is well developed and has really good accuracy this is why it’s available on a lot of platforms. From face swapping, gender or age change and make-up to 3D overlayed animations and face morphings made by artists.

This filters can get problematic when it comes to showcasing different stereotypes. Meitu is an application that makes the user way cuter with lipstick, bigger eyes and whitened placing him in a nicer environment. This application even though has the intention to beautify it is borderline racist but being a Chinese developed program it shows the sociological situation between lower/middle-class and upper-class.

Going further over the filter-generated world, a lot of researchers believe the personal wearable AR technology can have an immense potential in the industry. AR could satisfy the user more naturally in the future for everyday use. Some apps could overlay reality with messages, reminders, and GPS guidance. It can be used in a lot of work context where you can take notes and write with AR. For navigation, there are some prototypes that demonstrate it can have a better result in driving having fewer GPS errors. But they are just concepts and prototypes, it will take time for the technology to evolve and to be used by the general population.

We can experience AR in different ways. The latest generation of the headset from Microsoft is HoloLens were introduced this year in February. Facebook confirmed they are making their own AR glasses. With the technology evolving we can see lens displays that could work for AR. Screen devices are much more available for the users, with the use of an application every mobile phone can be transformed into an augmented reality device. The LCD walls combined with a camera can display an AR experience having bigger proportions. Holograms using a pyramid are an effect that can revolutionise the augmented reality industry.

After 1998 during American Football the viewer could see an augmented reality line on the field which was delimitating the stadium. But American football is not the only one utilizing AR, the Fox-Trax system is widely known for hockey, swimming, racing games. Another example for AR is ARQuake being a game where the player is fighting in real environments virtual enemies. SkyInvaders is another mobile augmented reality game where online players try to avoid real human runners.

Pokemon Go being released in 2016 is a good example that the general population is ready for AR in entertainment. As a user, you need to chase virtual pokemon characters in the real world. It was a world-wide phenomenon being an almost a hypnagogic experience with everyone trying to “catch them all”.

But this kind of application goes back in 2012 with Ingress, which developed by Google where the users try to catch portals all over the important places (such as monuments, parks, museums etc.). You can gain points by linking the portals together. This transmedia game makes the users choose between being from the group of the “enlightened” or the “resistance” having a big battle between them. When a user links two portals between them creates a field owning it for the team. They have different augmented shapes and forms over the real map and the users started to create designs which were representative for each group of individuals that might never be met. Some examples are forms of flowers, Christmas tree, dinosaurs, and butterflies. Also after the Boston marathon bombing, both teams decided to create a design n memorial for the deceased policeman.

Figure 19 “Field Art created with Ingress. a Flower in Germany. b Christmas tree in London is over two miles long c Butterfly in Heidelberg, Germany d Woodpecker in Germany”

The layer is an application that lets users geotag locations with texts, sounds, images and others. It is an interactive way to share information and discover new ones. Initially, a locative technology evolved also in a soft that recognize images and add the digital feed over it.

Vuferia, on the other hand, is basically based on image tracking technology. And an interesting thing is that the digital 3D augmented object is still present for the user to experience it even though the camera is not pointing at the initial tracked target anymore.

AR evolved considerably since the beginning of the XXI century and it will still grow exponentially in the next years principally in image recognition. And with this, all improvements in software, cameras and improved devices, augmented reality will have new breathtaking experiences for users.

Architecture, Urbanism and Construction

Digital images used for architecture evolved in the past few years because of the modelling image and visualization algorithms software. AR programs are overlaying 3D models of buildings over the 2D urbanistic plan. The user can navigate both inside and outside the model but still has some problems if it’s used on the actual site because of the recognition system where it depends on the weather and lighting.

In construction, there is a huge development in augmented reality devices that helps the architects, engineers and workers. Just in the United Kingdom, there is allocated around 1.000 pounds just for developing AR technology in this field. A lot of companies are using Building Information Modeling to make 3D plans. After uploading them into the software the engineers can see with a mobile, tablet device or glasses the full-scale model plan.

MLM Group is using Waking App to show the AR projects beyond the plan creating 3D models over the original from several to half an hours. The CEO of the application Matan Libin explains: “AR technology is providing clients with more control and understanding before the first nail is ever hammered into their construction projects.” Augmented reality application gives the opportunity to visualize tridimensionally everything. SmartReality is another mobile application which uses 3D models where it scans the plan and overlays the virtual design over it and the user can see it all around.

The communication between the architect and construction workers becomes easier because everyone can see exactly in real size how the structure needs to be made and the measurements are extremely precise saving time and money. Everyone can have access to the platform in the team changing information about materials, layouts and equipment. AR can also help for safety, some application showcase the construction workers how to prepare and check for safety hazards in the real-life scale environment they will start to work.

AR can help architects also with the relationship with clients. It is really hard for someone who doesn’t have a spatial vision or doesn’t know how to read a blueprint to understand their future homes. So architects are using 3D models to showcase their designs and AR can improve this relation and expands the client’s vision. They can walk around the empty area and use their smartphones to adjust their building because they actually comprehend the plan. Instead of trying to figure out how big is a room from numbers or a computer generated modelling scale through a screen, they can experience it in real life. It can be an emotional moment as well as seeing your future home for the first time before it was built.

Figure 20 “Augmented Reality for Architects and Building Designers”

One problem with this technology can be the illumination. This element can influence the integration and realism of the 3D object in the real world. A group of professors and students from Spain research the technology which makes the augmented object to adapt to the real world’s lighting. From the 1990s until now there are some developments and research that try to apply the lighting conditions from the outside world into the digital one. There are three lighting techniques can improve the quality: adding the shadow from real to virtual, illumination mixing the two realities, and shadows from the virtual to the real world. But besides the three simple elements, depends on the intensity of lighting and shadows as the location of the source. Also, it should be taken into consideration the reflection of some objects and the interactivity between augmented objects and real spaces.

At Virginia, Tech analysts develop a wearable AR technology. This exoskeleton AR can help the workers to perform safely and efficiently. And by 2030 in this industry is expected the builders to have machine suits and AR will be a part of it. The devices will help the constructers see life-size plans, having more safety and can help everything go smoothly without errors.

Education

“AR has strong potential to provide both powerful contextual, on-site learning experiences and serendipitous exploration and discovery of the connected nature of information in the real world.” Augmented Reality provides the necessity of the children to learn in a playful way while engaging directly with different physical artefacts preserving it. The users can interact with the applications in different ways. Here are some good examples of applications that broke the old conventional educational system:

University of Northern ColoradoAR developers, in educational games, the task is to make the user engaged to learn to have a new approach and it is proven to be really useful even though we are at the beginning of such endeavour. Augmented reality will soon influence all the educational system like all new media technologies introducing new methods to make the information comprehensible. In our days 80% of students have smartphones, and the possibility to use it augmented reality applications is a big opportunity. Students can be far more motivated especially for the ones that understand better visually.

AR can render difficult objects that can be hard to visualize by the student and translate the theory, especially in mathematics, into images. In Canada, there is an augmented reality wall in a gym class where the kids can hit the forms with the ball making the physical activity fun. Also in Portugal uses AR for the Polytechnic Institute and the students find the app useful and easy. The application Google Translate also has an augmented reality mode where you can use it live to translate foreign languages overlaying the text exactly over the original one. It is perfect for self-education and also for tourists.

AugThat founded by Adam Newman is an application that has 360-degree environments with lessons and their mission is to engage the student with the technological experience. Instead of trying to memorize the subject, they can firstly visualize it in an interactive way having videos, images and text. Students can scan interactive sheets or images and it will reveal digitally videos and explanations.

Elements 4D is a free application that helps the user, especially middle school students, to understand chemistry. They learn about the properties of the elements by combining them together into a 3D cube and when they touch each other start to react as in real life. They can also click on each element to learn more about it. The same developer, Daqri has another app named Anatomy 4D which help students to see the augmented human bodies and organs. You just need to reach the 2D sheet with the camera and the application will overlay 3D animated elements from the human body, also you can choose between learning just about the organs, skeleton or muscles.

Gulliver’s World is a multi-user interactive mixed reality game having a multilevel structure and people of all ages can play in this 3D augmented universe. In it you can choose between digital film production, theatre and you can be either a spectator, an actor, director, stage designer or director. Environmental Detectives and Mystery at the Museum are two educational interactive augmented reality games having stories, activities, giving roles engaging children into learning without even knowing it.

Mystery at the Museum is an AR game created by MIT Teacher Education Program. In this game, you can be a technologist, a biologist or a detective and you will work with your teammates to solve crimes. There are different ways to elucidate the mystery, such as adding the clues together, analyze samples having virtual instruments and interview virtual characters. It’s an interactive digital game where you can learn about the museum and how detectives proceed in the real world.

Environmental Detective is a multi-player augmented reality simulation game where the teachers can instruct the designers to develop it for a specific location and the students to investigate it with the purpose to learn about it. Being customizable the situations can vary from class to class emulating the needs for each field. The students need to gather data to see potential environmental problems given by the game.

Playing with the artwork is an augmented reality game that engages children to learn while they use their creative side. Usually, during school trips, they are given dull homework and they can’t actually enjoy the artefacts. This application is given to children different puzzles from artworks that they need to complete into the real works. Another feature is to colour the black and white pieces and after they also need to find which buildings, sculptures or painting is from. If the colour doesn’t match with the work the app gives more information about the work they are searching for. After they identify it the application starts to give information in a text or/and audio about the artwork. And also in the device, the kid can see how it blended into the piece giving a feeling of reward. In the end, it can give interactive quizzes where the students are more engaged to learn in a playful dynamic way.

Figure 21 “Playing with the artwork: user colours a given contour. After finding the corresponding artwork the patches from the contour wrap onto it creating a unique personalised version of the artwork”

Nathan Shafer created augmented Inuksuit Project in 2012 in Alaska. It’s an educative AR project where the kids use 3D legos to build different forms such as an igloo. The kids worked together to create a new form from inuksuk which represents landmarks made by the Inuits in the past.

Construct3D is an efficient application for geometry and mathematics in general and can help future engineers to comprehend this world. The developers try to maximize the abilities to comprehend through a platform that is easy to learn to improve spatial skills. It’s much more user-friendly than an architecture/construction application. And the spatial skills learning tool includes five distinctive categories: spatial visualization, spatial relations, spatial orientation, spatial perception and spatial rotation. Now the student can see a tridimensional object instead of imagining and calculate it on a 2D platform. The application has the possibility to take measurements, a multitude of geometrical 3D forms and also 2D ones.

ARvisceral

My dissertation project is ARvisceral, a series of five illustrations which are augmented by 3D and 2D animations. Each of them has its own story but all of them are coming from the same digital world and in the middle of it it’s always an avatar mostly creating the events.

I wanted to express the subconscious because is linked to another “universe” full of surrealist and incoherent dreams and thoughts. And the digital world can somehow be linked to it because it’s different from the physical one. I choose this title, “visceral” because of the kind of process I did to create my works. It was rather simple for me from a young age to draw or to write entering a zone where I wasn’t thinking when the next line or word will be. So every time when I did that I produced works, and exactly as dreams, it took me several weeks to actually decipher them myself. The zone is the state of mind for the brain which manifests exactly as mediation by practising yoga or tai-chi. The brain is maximizing the concentration over the movements and it’s not thinking about other things such as the past or future, just about every glimpse at a time.

But for my dissertation work, I knew because of the technique it would be much difficult to fully enter that zone on the entire process. So I choose just the first steps to be produced in such manner and the rest to be as an interpretation “explaining” the product in a minimalistic way. The first step was moulding and manipulating the 3D avatar in Maya. I call it as my avatar because unintentionally I choose a blue translucent gleaming woman to represent my unconscious. The only representation of the “real” me was in the first work “Eraser me, baby” which is actually another moulded old character but it is quickly erased as a process of cleansing the mind for the next works. In the 3D world from Maya, the avatar is experiencing different encounters with forces which can simulate physics but usually, they are being ignored. The character has all the power to erase, destroy, create and float but it’s also absorbed by the elements around it.

In the next step I used Photoshop and After Effects, seeing the actions of the rendered character, I constructed an environment which would be perfect for the manifested action. I choose bright popping colours because I envisioned the digital universe having a totally different full scheme and also I was inspired by glitch art and cyberpunk’s colours. I also manipulated the 2D layers to be integrated into the frame, just the 3D ones escaping it into the world.

The third step, after finishing all the layers which are separated one from another, is to create the illustration which will be printed and recognize by the software. This step is the most analytical because it is part of the physical world. I choose to sum up the beginning of the experiences giving clues of where the augmented videos will go. And at the same time, the illustration can stand alone as works which can tell a story without the augmentation.

In the final step, after editing the videos in Premiere and After Effects, is the integration in the application. I used Artivive which is perfect for artists, it has an option for 3D where I could upload a maximum of 3 videos and 6 images. Because of the limitation, I always take in mind which elements are important to be seen in movement besides the 3D objects. Because in this universe there is also vibration translated in sound, I also used foleys and ambient editing them in Audition to immerse the viewer more into the augmented works.

Figure 22 “Eraser me, baby” (author’s collection)

They are in two categories referring strictly to the content reflecting dualism in the avatar: destructive, tranquil, and one aftermath. From the first category are two short and impulsive works where the avatar is creating the destruction. In ”Eraser me, baby” (Fig. 4) the two avatars translated in moods are destroying by erasing the real character with two lasers coming from their heads. One of them is conscious of what is happening: to inflict the destruction intentionally and the other is almost harmless having a child-like character and actually smiling when she sees the aftermath. It is like now they have all the liberty to manifest themselves in their true forms without having the consciousness limiting them. In the background, the brushes which remind the viewer of the old physic painting techniques are distorted into liquid lines until they almost disappear.

The second destructive work is “01110011 01101111 00100000 01110011 01100001 01100100” (fig. 5) which is translated from binary in “so sad”. In the work, binary writing starts to appear like someone is controlling every aspect of the avatar through programming it. In a way this is true, the viewer is reminded that the work is created by bytes with technological languages. The character now is having its own digital aura translated in a circle giving a divine connotation. It’s choosing to destroy the object appearing in front of her, a heart-crushing it in thousands of glass shards.

Figure 23 “01110011 01101111 00100000 01110011 01100001 01100100” (author’s collection)

A tranquil work is “Cocoon” (fig. 6) having the other side of the avatar living and trying to repair the glass shards in the multi-colour environment. It is floating into space in the core of the event, the DNA in the background actually representing the symbol of infinite being one of the only self-replicating material. It can be seen as a continuity of the events; even though they are not meant to be seen as a chronological series with a story, but each of them has its own narrative being from parallel time-lapses. Cocoon paradoxically has the most dynamic composition complementing the actions.

Figure 24 “Cocoon” (author’s collection)

The second one is “Creator” (fig. 7) making the avatar much more complex, she is not only destroying and repairing but also she’s actually the creator of the world she is living in. Levitating is making spherical particles to appear and then slowly disappear, having the character of a child she’s surprised to see her own making. Also, the three elements which the avatar is always controlling are surrounding it going through her ephemeral translucent body without a problem.

Figure 25 “Creator” (author’s collection)

The centre-piece is “Cotton Candy”(Fig. 8), here it is the only action which the character is not controlling, she is running from the Cotton ball which ends up swallowing her. The composition I made from a circle not needing any other element in the background. In the augmented background the viewer can see a space-like environment and stars floating appearing and disappearing. The user is having a clue of how reality is coming back swallowing the digital augmented avatar through a cotton-swirl-candy ball. In the print media, I added the “ghosts” from the avatar’s past and future actions. She is transformed into one of the background yellow star being a small particle in an entire universe as all of us.

Figure 26 “Cotton Candy” (author’s collection)

Conclusion

With the technology evolving and the software becoming increasingly sophisticated, new forms of digital realities emerge. Augmented reality implants digital objects into the physical world and the users can experience it without leaving it as in virtual reality. AR is not just an innovative element in the creative field, but it can be the precursor of an emerging amalgam of realities in the art world. The technology is constantly evolving but to be defined as an augmented reality it needs some distinct components: To use tracking, either on the 2D or 3D image or/and location from the real physical world, to be registered in the virtual one activating the visualization into a device used by humans.

The Artivive application is having a maximum level of engagement of consumption because the content is not predefined and can be changed by the users giving the opportunity to showcase their own works and display them publicly. The gesturing of the application I used is planar because it’s working on printed media, the software is using image recognition and the digital animations can be displayed into the tridimensional physical space.

In my project, ARvisceral, the viewers are using the intermediary apparatus to see the augmented printed illustrations. The interaction is happening inside the frame of the device in the virtual space, the AR software from the Artivive application is recognizing the image and is activating the animated visuals. People attending the vernissage become just performative observers where they can just move the camera to discover the work from different angles and perspectives in the physical space.

The project emulates truthfulness coming from the process of achieving a mental state similar to meditation. And this is reincarnated in an avatar, with a complex and counterintuitive personality, and its actions in a tridimensional world. The result is translated into five distinct illustrations: “Cocoon”, “01110011 01101111 00100000 01110011 01100001 01100100”, “Cotton Candy”, “Eraser me, baby” and “Creator”.ARvisceral is using a classic context such as a vernissage with printed illustrations to display a digitally augmented world by adding computer-generated images over the environment.