But, I wonder about some of the non-obvious ways ways these games are different.  Today, as I was walking to the coffee shop to get beans for our espresso machine, I was thinking about the ways games enforce rules, and how board games and card games are fundamentally different than computer games.  With board and card games, rule enforcement is left up to the players;  they know the rules, and they abide by them.  Even when there is nobody around to “catch” them people naturally follow the rules;  to cheat removes the fun.  My dad played solitaire for hours to unwind after working shifts in an auto plant, and as far as I know, he never cheated.  What would be the point?

Computer games, on the other hand, encourage players to do anything the game allows to win.  Because the system is closed and the rules are enforced by the computer, finding ways to get around the system is part of the fun for many players.  If the game let’s you do it, it must be ok!

But, this presents a problem, which I’ll put this way:  will players treat hybrid computer/board games (or computer/card games) as board/card games or as computer games?  Will they play along and follow the rules, even if nobody is there to “call them on it”, or will they do what they can to win?  When there is a high score board, and achievements to unlock,  will players be content to take what the deck gives, or will they stack the cards?

This question is more than academic, because it impacts the kinds of games we can create.  Consider Sony’s Eye of Judgement, which is designed assuming players will treat the game as a computer game, and thus is structured to allow the computer to enforce the rules.

Click here to view the embedded video.

For those of you who have played the game, you know how awkward it is.  After playing for a while, you realize that you are playing a relatively complex card game (ala Magic), but one where the computer’s main job is to enforce the rules. The computer is used for little else, beyond adding eye-candy to justify playing the game in awkward space of the PS Eye camera.

But beyond the awkwardness, one can’t help but wonder what the game might have been like if the players were trusted to follow the rules and enlisted in the gameplay.  What kind of games could we create if the player was allowed to move cards fluidly, was allowed to manipulate their environment naturally, and not treated as a hostile, untrustworthy interloper?  What if the rules were presented and it was assumed people would follow them?

Imagine a single player augmented card game that starts with the player shuffling a deck and laying out 9 cards in a grid.  No checking by the computer, no proactive rule enforcement.  BUT, like traditional solitaire, the placement of the cards determines much about how the game unfolds.  Would such a game work?  It’s hard to say.  If a player approaches it like a card game, where the goal is to enjoy a pleasant diversion for a few minutes, and challenge yourself to solve a puzzle, perhaps such a game would work.

But, will players even consider that approach?  After years of being trained by computer games to take any advantage offered by the computer, will they “cheat” and chose a card layout designed to give them the best start to the game?  Will they even consider this “cheating”?

I’m not sure, but it certainly impacts the kinds of games we can create.

Yet, whenever I read about someone’s technical game degree program, I’m always left wondering about the jobs we’re pointing these students at, and the “unwritten pack” we make with students we accept into our programs.  I’ve thought a lot about this, because the topic of a “game degree” comes up occasionally.  I’m also teaching 3 game classes this semester (a game-oriented capstone, a game prototyping lab, and an augmented reality game design class), so I’m very focused on the issue of game education.  Yet, with plenty of course and interest in games here at Georgia Tech, we haven’t created a full-blown game degree;  our game education activities are folded into a number of other degrees that offer a broader education beyond games.

This article isn’t about our choices at GT, though.  Rather, I interested in the opinions of others who might read this.  I believe there is an implicit suggestion that, if we have a focused degree program in a technical area, it’s educating the students in preparation for actually doing something.  With a liberal arts education, the goal is to give a broad education, and the students understand that there aren’t “liberal arts jobs” per se.  But, I suspect that students don’t generally get a degree in Mechanical Engineering, Information Security, or Pre-Med just because they want to broaden their horizons and open their minds;  they get these sorts of degrees (presumably) because they want to work in these areas after they graduate (or move on to other degrees, in the case of pre-Med or pre-Law).

So it is, I think, with a computer science or technical game-oriented degree.  Which brings me back to the topic of the post.  I wonder how many “game degrees” are being created because the school honestly believes that there is an industry need they are fulfilling (industry has unfilled job positions and needs to have more folks educated to fill them) or because the university has a need they are fulfilling (the student enrollments are dropping and they need students to fill up the classes).  For a number of years, CS enrollments have been down at many schools (we’ve been doing OK at Georgia Tech, and if you include our cross-over degrees like Computational Media, we’re actually doing better than OK, both at attracting students, but also at attracting students who aren’t young, white boys).  Over the past few years, a number of schools have created game degrees that have attracted a lot of attention, not least because it appears that they attract a lot of students and because their graduates get to go and work in the game industry.  That’s all well and good;  a few schools (USC’s MS program in the School of Cinema TV, CMU’s MS in Educational Technology, our combined undergrad and grad degrees across CS, CM and Digital Media, for example) have a great reputation with the game industry, and the students coming out of the program have generally had good success in getting the jobs they want.

But, how many schools are just putting lipstick (and new name) on a pig (their dying CS degree programs) to attract new students?

Overall, my sense is that there aren’t that many great jobs in the game industry, at least not in the numbers that are needed to employ an increasing number of game program graduates.  Especially the jobs many students seem to dream about (game design, game engine programming, etc.); here, I’m talking about technical and design jobs, I’m not talking about testing and QA, or level design, or content-oriented jobs (a few schools, like SCAD and RISD and so on, are doing a great job educating those students).

Unfortunately, when I read articles like Mike’s, I’m reminded of the increasing trend of CS departments to offer Game Degrees (note:  I’m not saying this is what USC is doing;  Mike has a long history of game education, going back to the Naval PostGraduate School, and actually came to USC to create a game program in the CS department).  A game degree is a great fit … for the school.  There is virtually no area of CS that isn’t applicable to the technical side of a modern game, especially the blockbuster console games;  these games require everything a major CS degree offers, and then some.  Taking your existing CS courses, faculty, labs and infrastructure, putting some new makeup on it, and calling it a game degree can be a great way to attract students.  And (based on anecdotal evidence) for many schools, it seems to be helping.  Helping them, that is, attract students.

My worry, though, is that as more and more schools offer game degrees, we’re going to turn out a generation of pseudo computer science students who can’t get the jobs they want.  After all, how many jobs are there?  I was chatting with a game industry exec at a recent conference, and he joked that, given the low turn-over in the good jobs, the good opportunities number in the hundreds … not in the thousands or more that will be needed to place these students.

So, I’d love to hear other folks thoughts on this, or get pointers to hard data about jobs and graduates.

In general, I agree with Jarrell, although I am probably a bit more optimistic that he appears to be;  for those who’ve read my previous posts, that may seem surprising.  I think his comments in the first few sections (“Augmented Reality Glasses are not Viable in the Near Term”, “GPS and Compass is not Enough”, “Sensor Fusion is the Answer”) are pretty much dead on; if anything, I’d say he doesn’t emphasize enough just how impoverished an experience “GPS + Compass” limits designers to! In both our work at Georgia Tech, and in my company, we’ve experimented with what you can do with the GPS+Compass combo, and it’s hard to come up with non-trivial, compelling experiences.

The problem with the GPS+Compass is that, on any affordable device, these sensors are of such poor quality that they are almost useless for AR.  Because the position accuracy, in particular, is so poor, you really can’t “put stuff in 3D” … you end up treating the world as a sphere around you on which you paint content that lies in a certain direction (which is what all the so-called “AR browsers” are doing;  putting icons up for content that lies in roughly a certain direction).  By explicitly acknowledging this limitation, that the AR world is really just a sphere around you, we designed and released a small application, SantaVision, on the iPhone just before Christmas;  in this application, you put 2D stickers on a virtual sphere around you, ignoring the full 3D nature of the world and letting you decorate it from one location in space.  The application is fun, but the limitations of the compass are fairly obvious.

As Jarrell points out, sensor fusion (combining these sensors with computer vision), is clearly the answer:  this has been known for a while, highlighted by his reference to Ron Azuma’s PhD work in the late 90′s!  (Ron is now doing AR work at Nokia’s Hollywood Lab, btw).

Of the 5 platforms he points out, I’d say the only two important ones are smartphones and tablets;  the others are more practical now, because of the greater computing power they bring and because the constraints of their deployment make them easier to target.  However, handheld devices offer a unique first-person perspective that I believe is essential to leveraging the full power of AR (this perspective is also offered by HMDs):  by coupling the display with the camera (in video-see-through applications such as these) the technology creates the illusion that the display is being looked through at the world.  This coupling facilitates a direct, natural interaction metaphor, that cannot be achieved with the other technologies mentioned.

Overall, I think his final point is key, which I would paraphrase as follows:  the success of AR is tied to creating usable, useful, fun or entertaining applications and experiences.  Novelty and gimmick’s will wear off soon, but fun experiences that take advantage of the unique nature of the technology (and thus can’t be achieved without it) will earn AR a place as a viable and lasting approach to as a human-computer interface.

I’m quoted in the article as saying

“When the graphics effectively align with specific parts of the real world and we get this tightly accurate form of augmented reality, people start to really relate to their AR surroundings [the 3D images of the town and characters],” he claims. “They talk about the mixed space as if it actually existed and was theirs, rather than just a gaming platform.”

which is very odd sounding to me;  what I actually would have said is something like

“When the graphics effectively align with specific parts of the real world and we get this tightly accurate form of augmented reality, people start to really relate to their AR surroundings [the 3D images of the town and characters] as if they really were out in the physical world,” he claims. “They talk about the mixed space as if it was actually one combined space, rather than just being inside the gaming platform.”

While the later is fairly straightfoward (“when the graphics are tightly registered with the physical world, people can start to behave as if the graphics are in the physical world”), the former quote is somewhat thought provoking.

Could we ever create an AR experience where people really started to believe the virtual content is in the world (not just behave as if it were)?  I do not believe that can happen;  if you know that the content is fake, then you know it is fake.  You may experience the beauty and fidelity of the holodeck, but in the end, you know you are in the holodeck.

If you are interested in this, we’ve done a huge project exploring this very issue, which you can read about here.

Can you explain what augmented reality is?

Augmented reality is a technique for presenting information to a person by blending it with their perception of the world around them. Typically, this has meant using see-through displays to merge 3D graphics with the physical world; it can also mean using spatialized audio to put sound out in the world. The see-through displays could be real transparent displays, such as the Microvision virtual retinal displays; more typically right now, the see-through displays are created by combining a camera with a normal display (such as the camera on the back of a smart phone) and adding graphics to the video before displaying it. When you do this in real-time, these “video-see-through” displays give the illusion of being see-through.

The key is that the information is “registered” (i.e., aligned) with the view of the physical world. There are many interesting things that can be done with location-sensitive information, such as doing location-based search via google-maps, or telling stories or creating games based on the place or time the game is played. These can be fun and useful, but I would call these “mixed reality” or “context-aware computing.” Augmented reality is a subset of these concepts, that adds the idea of perceptually merging the content with the world.

To overlay graphics (or sound) correctly on the view of the world, the computer (or phone) needs to know precisely where the world is in relation to the camera. The more accurately the position and orientation of the device is known, the more accurately the graphics and the physical world can be combined.

On a smart phone, you can try and use the GPS and other sensors (e.g., compass, accelerometers) to estimate the position and orientation of the phone relative to the world; the problem is that the limited accuracy of those sensors restricts the capabilities of these applications. In talks and presentations recently, I have been showing a video I made of the Yelp Monocle system, where I look around the courtyard in front of my building, and none of the labels align with what they are referring to. This is not a bug in the Yelp iPhone application; it’s a limitation of the accuracy of the GPS, and is common to all GPS-based AR applications on mobile phones today.

Alternatively, computer vision can be used (with smart phones, or with web-cameras on computers) to find known objects in the world (e.g., the little black and white square markers used in a lot of AR demos) and determine the position of those objects relative to the camera. Since this can be done very accurately, these systems can put graphics very tightly on these objects. If you look at the videos of our AR games (i.e., on our youtube channel at www.youtube.com/aelatgt) you’ll see that we mostly do these sorts of techniques.

How is this technology syncing with today’s electronics on the portable side?

The growing proliferation of smart-phones with cameras, GPS, compass and accelerometers means that primitive AR applications can be created now. There is a wealth of geo-located data available on the web (e.g., all the location-searchable content on Google and Bing, plus Flickr images and soon Twitter tweets). So, it’s straightforward (though not entirely trivial) to create a “browser” for one of these information sources. As the technology gets better, and the sources of geo-located data grow and is refined, there will be a natural progression of quality and capability of these applications.

The real question, of course, is “Are any of these geo-located information-browsing applications useful?” Time will tell. I think there is some utility over the top-down 2D view, but there are a lot of negatives imposed by the limited quality of the sensors and the crude quality of the data (see my comment about the Yelp application above). So, right now, there is excitement generated by novelty, but eventually these applications will need to do more.

The next big breakthrough will come when there is enough infrastructure in place to support combining these sensors with computer vision outdoors, away from known markers and objects. This is the focus of significant research in universities and companies right now, and will lead to some advances as early as the next few years. When vision-based tracking can be used outside, there will be a dramatic change in the kinds of applications that will be able to leverage AR.

And on the stand-alone side?

Consider desktop computers and consoles.  The technology is here to do really interesting things. Companies like Imagination (in Austria), Total Immersion and Metaio have great technology, which hasn’t really be leveraged non-trivial ways yet; most of the uses have been pretty simple advertising sites, or gimmicky applications like the Topps baseball card augmentions (which are cute, but of dubious long-term interest). Some applications (like the Lego kiosks) are actually pretty nice, since they solve a problem (even if only a minor one) at the place they are deployed (“what’s the model inside this box look like?”).

If you look at Sony’s EyePET game, you can start to see what will be possible in the desktop camera space. The interesting thing there is that the PS3 is actually a pretty weak machine by today’s desktop computer standards (compared to a souped up gaming PC or Mac, for example), so there is a lot of potential for interesting applications.

The big problem with the desktop is the relationship between the camera and the screen. Over the many years I’ve worked with AR, it has become obvious that the first-person perspective created by see-through displays (e.g., head-worn displays or handheld smart-phones) is far easier to understand, far more compelling and useful than the “magic mirror” effect you get when you have a camera that is not attached to the display. When using a web-camera or something like the PSEye and a computer display or HDTV, where the camera is either looking out at you or something you hold in your hand, the merged world you are “seeing” on the display is not connected to where you are physically looking. While people can learn to use the systems, and they can still be fun, they are nowhere near as natural or immersive as the first-person ones.  Think about the difference between playing tennis (looked at your opponent) and playing Wii Tennis (standing beside your opponent, with both of you looking at the screen).  It’s similar in some ways, but not the same.

You need to try both sorts of AR experience to understand the difference. When you look at videos on the web, they look the same; but the experience is completely different, and the videos are often staged to hide these differences. For example, in the EyePET videos, the kids are often looking at the little monkey-like creature on the ground in front of them; however, in the game, they would be looking up at the screen, to see the merged view, and they would see themselves looking at the screen, not at the pet.

We’ve been able to do these sorts of “magic mirror” AR things for years, but it is the handheld applications that have really sparked peoples imagination.

How do you see this technology being used for videogames?

I see it being used as an extension of what we are currently seeing. In the near future, we’ll see a lot more of the Sony EyePet and Sony Invisimals kinds of games, that use fixed cameras on consoles or use markers and game-boards with the handhelds.

Handheld consoles combined with game boards and markers are the area where I believe the most exciting things will occur, and the limits of what is possible is largely dictated by the underlying tracking technology. The current GPS plus compass plus camera setup on current mobile phones will not allow any really good games to be built, in comparison, but once we can use the video to do precise outdoor tracking, we will start to see amazing things on these devices as well. Imagine taking our ARhrrrr! game outdoors, where you are fighting zombies on the street, or shooting at them from your windows. We will be creating an outdoor AR game this year, using some experimental tracking software one of my students is developing with Nokia. Our goal is to have a multiplayer outdoor game on Nokia N900′s, with the same kind of tight registration between the physical and virtual worlds you see on our other tabletop games.

What impact do you see augmented reality having on entertainment?

In addition to games, I see lots of potential in the social applications of AR, akin to all of the little applications you see in Facebook right now. We ran a class last year, where we asked the students to imagine “mobile AR facebook” applications, and the prototypes they created were very exciting.

Similarly, I see lots of little casual “AR toys” that let kids mix virtual and physical content. We have a project where we are taking MIT’s Scratch programming environment and adding AR to it, and hope to have a “player” for it running on smart phones this year. The idea is to let kids create little AR games and toys and share them with each other. When the next generation of mobile camera-based handheld game devices come out (e.g., the next DS or the camera-enable iPod Touch) we can start imagining kids creating their own AR experiences.

What are some examples of what your lab has done in the gaming space?

You can see videos on www.youtube.com/aelatgt and on our lab web site.

There are three main games that are interesting, that we’ve learned interesting things from.

Bragfish (the paper on this was published last December at ACE2008) was the first one we created that we actually studied. It’s a two-to-four-person “around the gameboard” game, meant to explore the social experience of playing a combination of a board game and computer game. The key things we found are that when the graphics are really tightly registered with the physical world (the gameboard in this case) people start to treat the handheld as a window into the play space, which is “on” the table. This is important, because people then start being able to use all of their perceptual and physical skills to interact and to understand what the other players are doing. The second thing we found is that, not-surprisingly, the experience of playing these games was very different than other multi-player computer games, because the players felt like they were “in” the same space.

In Art of Defense (published this summer at the SIGGRAPH Games track), we looked at the same issues in a collaborative game. We also included a lot more tangible props and game pieces. We found that the shared understanding of the space carried over to collaboration and that people really could smoothly collaborate by leveraging the physical props, rather than having to refer to the small screens. Again, the experience was very different than other collaborative computer and handheld games, according to the players.

Finally, my students and I created the ARhrrrr! game with Tony Tseng and his students at SCAD-Atlanta (and with help from NVidia and Daniel Wagner at Graz University) to explore a much richer, engaging experience. This game points to one future category of table-top handheld AR games, where a 3D world is superimposed on a table, and people can play around it. We hope to do a lot more in this space over the next year, and I can see building many commercial games based on the ideas we’ve had.

What do you think we’ll be seeing at CES in January that will help propel augmented reality into the mainstream?

I don’t expect much revolutionary to appear, honestly, but I do expect some obvious evolutionary steps.

I expect new head-worn displays to appear (companies like Vuzix have been promising that), which will generate some excitement, but not to have a huge impact yet because they won’t be that well integrated into the platforms nor have much in the way of compelling experiences.

I expect a flood of new phones, based on the high-end chipsets (e.g., the Tegra, the Snapdragon, the OMAP3, etc). The real question in this area will be “will those phones have the right APIs and sensors to support AR experiences that can fully leverage the platform?”. We could build AR games like ARhrrrr! on the iPhone right now, if we could get at the camera video stream efficiently in real time; the limit is not the hardware, but the OS. And so it will be for the new devices. Another example: Android doesn’t allow efficient access to the video for computer vision, so the Droid may not end up being the breakout AR platform it could be.

We’ll also see a host of new UMPC style devices that start to look like smartphones, but have better technology. It’s unclear what this will mean though, since they typically aren’t aimed at kids.

The big question will be in the space of devices aimed at kids. If Apple opens the video API, and releases a camera-based iPod Touch, this will be a game changer because of the size of the market. Kids don’t own iPhones; the phone is cheap, but the plans cost more than $1000/year, which most kids can’t possibly afford. But any kid could have an iPod touch. If Apple doesn’t do it, perhaps someone else will. Will Sony get behind the PSP with camera, and do more games like Invizimals? The PSP and DSi are pretty weak platforms, so they games are limited, but what will the next portable look like? Will Microsoft release a XBox GO based on the hardware they are developing for their phone, akin to the iPod Touch/iPhone dynamic?  Who knows.