Living in an Augmented Reality

I was just reading Mike Zyda’s article in the December CACM on games and computer science education.  It discusses the technical game education program they’ve created at USC in the CS department, and gives a nice overview of why they are doing things the way they are.  Seems like a reasonable degree.

Yet, whenever I read about someone’s technical game degree program, I’m always left wondering about the jobs we’re pointing these students at, and the “unwritten pack” we make with students we accept into our programs.  I’ve thought a lot about this, because the topic of a “game degree” comes up occasionally.  I’m also teaching 3 game classes this semester (a game-oriented capstone, a game prototyping lab, and an augmented reality game design class), so I’m very focused on the issue of game education.  Yet, with plenty of course and interest in games here at Georgia Tech, we haven’t created a full-blown game degree;  our game education activities are folded into a number of other degrees that offer a broader education beyond games.

This article isn’t about our choices at GT, though.  Rather, I interested in the opinions of others who might read this.  I believe there is an implicit suggestion that, if we have a focused degree program in a technical area, it’s educating the students in preparation for actually doing something.  With a liberal arts education, the goal is to give a broad education, and the students understand that there aren’t “liberal arts jobs” per se.  But, I suspect that students don’t generally get a degree in Mechanical Engineering, Information Security, or Pre-Med just because they want to broaden their horizons and open their minds;  they get these sorts of degrees (presumably) because they want to work in these areas after they graduate (or move on to other degrees, in the case of pre-Med or pre-Law).

So it is, I think, with a computer science or technical game-oriented degree.  Which brings me back to the topic of the post.  I wonder how many “game degrees” are being created because the school honestly believes that there is an industry need they are fulfilling (industry has unfilled job positions and needs to have more folks educated to fill them) or because the university has a need they are fulfilling (the student enrollments are dropping and they need students to fill up the classes).  For a number of years, CS enrollments have been down at many schools (we’ve been doing OK at Georgia Tech, and if you include our cross-over degrees like Computational Media, we’re actually doing better than OK, both at attracting students, but also at attracting students who aren’t young, white boys).  Over the past few years, a number of schools have created game degrees that have attracted a lot of attention, not least because it appears that they attract a lot of students and because their graduates get to go and work in the game industry.  That’s all well and good;  a few schools (USC’s MS program in the School of Cinema TV, CMU’s MS in Educational Technology, our combined undergrad and grad degrees across CS, CM and Digital Media, for example) have a great reputation with the game industry, and the students coming out of the program have generally had good success in getting the jobs they want.

But, how many schools are just putting lipstick (and new name) on a pig (their dying CS degree programs) to attract new students?

Overall, my sense is that there aren’t that many great jobs in the game industry, at least not in the numbers that are needed to employ an increasing number of game program graduates.  Especially the jobs many students seem to dream about (game design, game engine programming, etc.); here, I’m talking about technical and design jobs, I’m not talking about testing and QA, or level design, or content-oriented jobs (a few schools, like SCAD and RISD and so on, are doing a great job educating those students).

Unfortunately, when I read articles like Mike’s, I’m reminded of the increasing trend of CS departments to offer Game Degrees (note:  I’m not saying this is what USC is doing;  Mike has a long history of game education, going back to the Naval PostGraduate School, and actually came to USC to create a game program in the CS department).  A game degree is a great fit … for the school.  There is virtually no area of CS that isn’t applicable to the technical side of a modern game, especially the blockbuster console games;  these games require everything a major CS degree offers, and then some.  Taking your existing CS courses, faculty, labs and infrastructure, putting some new makeup on it, and calling it a game degree can be a great way to attract students.  And (based on anecdotal evidence) for many schools, it seems to be helping.  Helping them, that is, attract students.

My worry, though, is that as more and more schools offer game degrees, we’re going to turn out a generation of pseudo computer science students who can’t get the jobs they want.  After all, how many jobs are there?  I was chatting with a game industry exec at a recent conference, and he joked that, given the low turn-over in the good jobs, the good opportunities number in the hundreds … not in the thousands or more that will be needed to place these students.

So, I’d love to hear other folks thoughts on this, or get pointers to hard data about jobs and graduates.

For those of you interested in AR, Jarrell Pair’s recent post on the state of AR is a pretty good summary.

In general, I agree with Jarrell, although I am probably a bit more optimistic that he appears to be;  for those who’ve read my previous posts, that may seem surprising.  I think his comments in the first few sections (”Augmented Reality Glasses are not Viable in the Near Term”, “GPS and Compass is not Enough”, “Sensor Fusion is the Answer”) are pretty much dead on; if anything, I’d say he doesn’t emphasize enough just how impoverished an experience “GPS + Compass” limits designers to! In both our work at Georgia Tech, and in my company, we’ve experimented with what you can do with the GPS+Compass combo, and it’s hard to come up with non-trivial, compelling experiences.

The problem with the GPS+Compass is that, on any affordable device, these sensors are of such poor quality that they are almost useless for AR.  Because the position accuracy, in particular, is so poor, you really can’t “put stuff in 3D” … you end up treating the world as a sphere around you on which you paint content that lies in a certain direction (which is what all the so-called “AR browsers” are doing;  putting icons up for content that lies in roughly a certain direction).  By explicitly acknowledging this limitation, that the AR world is really just a sphere around you, we designed and released a small application, SantaVision, on the iPhone just before Christmas;  in this application, you put 2D stickers on a virtual sphere around you, ignoring the full 3D nature of the world and letting you decorate it from one location in space.  The application is fun, but the limitations of the compass are fairly obvious.

As Jarrell points out, sensor fusion (combining these sensors with computer vision), is clearly the answer:  this has been known for a while, highlighted by his reference to Ron Azuma’s PhD work in the late 90’s!  (Ron is now doing AR work at Nokia’s Hollywood Lab, btw).

Of the 5 platforms he points out, I’d say the only two important ones are smartphones and tablets;  the others are more practical now, because of the greater computing power they bring and because the constraints of their deployment make them easier to target.  However, handheld devices offer a unique first-person perspective that I believe is essential to leveraging the full power of AR (this perspective is also offered by HMDs):  by coupling the display with the camera (in video-see-through applications such as these) the technology creates the illusion that the display is being looked through at the world.  This coupling facilitates a direct, natural interaction metaphor, that cannot be achieved with the other technologies mentioned.

Overall, I think his final point is key, which I would paraphrase as follows:  the success of AR is tied to creating usable, useful, fun or entertaining applications and experiences.  Novelty and gimmick’s will wear off soon, but fun experiences that take advantage of the unique nature of the technology (and thus can’t be achieved without it) will earn AR a place as a viable and lasting approach to as a human-computer interface.

While I usually reserve this spot to comment on the AR stuff other folks are working on, I thought I’d take this opportunity to comment on something we just released.

My colleague Maribeth Gandy and I create a small startup this year called Aura Interactive;  while we are doing a variety of things with the company, ranging from consulting to product development, our main goal is to create some fun AR games and toys.

And so I’m happy to share that we’ve just had our first iPhone application (done in collaboration with our colleague Kim Spreen) go live in the iTunes App Store, a little toy we call SantaVision (click here to go directly to the app in the iTunes Store).

Like many people, we’ve struggled with the inability to “touch” the live video on the iPhone, and the corresponding inability to do accurate, fast 3D tracking, of the sort we leverage in our research prototypes (e.g., the Zombie shooter ARhrrrr!).  At the same time, we’ve want to avoid creating yet-another shoot-the-<xxxxx>-around-you compass-based game, a couple more of which appeared today.  We think we have come up with a fun middle ground, leveraging the accelerometer and compass to let people immerse themselves inside small AR Christmas scenes they create themselves, using the “stickers” (static and animated images), sound effects and music loaded into the application.  And, using the GPS, you can share your geo-located scenes with the world.

If you try it, let me know what you think.  It’s not perfect, but we’re having fun with it!

I’m not quite sure what this means, but it is probably a good thing:

Zemoga, a pioneering digital marketing and innovation firm, today announced the launch of its augmented reality (AR) practice, designed to further the company’s work in this groundbreaking technology that has taken the marketing world by storm.

[...]

“We built the prototype of the human heart and could visually demonstrate in a dynamic 3D environment the effect a new drug would have on the heart,” explained Zemoga CEO and co-founder, DJ Edgerton.

I wonder if they are teaming up with an more experienced AR folks (if they are in New York, Steve Feiner’s group at Columbia seems like a good choice), to more rapidly leverage the research that’s been done over the years?  Regardless, though, it’s interesting.

Most people who know me know that I remain unconvinced of the usefulness of wearing projectors around your neck, despite all the cute fake mocked-up demos that people do.  But, if you are going to do something with projectors, that isn’t going to work “in the real world” anyway, why not something cool like this? (As an added bonus, this is done with a pile of those new Nikon cameras with the projectors built in.)

If I look at the misquote in this article, I’m struck by what a difference a small concept makes.

Read the rest of this entry »

Over the last few months, I’ve talked to countless reporters, and answered questions via phone and email. It occurred to me the other day that it might be good to start sharing some of these questions and answers here, so more people can see them; if I’m going to the trouble to explain AR and my views on it, why never put those ideas out there?   So, without further ado, here are some current thoughts on AR.

Read the rest of this entry »

Looking back over the summer, it’s almost comical to re-read the sometimes-frenzied excitement at the possibility that the iPhone will support AR with OS 3.1. The possibility was exciting, even to those of us not trying to keep a small company afloat;  the idea that there would finally be a platform that would let us get our ideas, games and products to millions of eager customers was an unfamiliar feeling for those of us who have been doing AR for a (very long) while. And for the dozen or so small AR companies, surviving on contract work for movies web sites and eye-candy web advertising, it must have been intoxicating.
Read the rest of this entry »

Ok, this is cute.  I wish I thought of it.  A use of FLAR (Flash AR) for something … almost useful?  And, most ironic of all, it’s by the US Postal Service!  I’m almost giddy!

It looks like someone may actually make an AR game for the DSi;  the GhostWire game that someone created on the Nokia phone is now going to be ported to the DSi.

Some folks would say this is not “really” AR, since the graphics (ghosts) are not tightly, accurately, spatially registered with the world.  Normally, I would agree;  as is obviously from my work and other posts, I think it is important to distinguish between different kinds of systems that relate the physical and virtual worlds, because different approaches have different affordances and capabilities.  I tend to call “AR” things that tightly register graphics with the physical world, because it creates experiences that afford certain things (e.g., allows people to leverage their spatial and perceptual abilities to remember where virtual things are, refer to them, perform actions in relation to them, collaborate, and so on). Conversely, I usually refer to experiences like this, that crudely position graphics in a space, as mixed reality (MR).

However, the ghost theme is the one theme that is amenable to “faking AR” on low end devices.  Over the years, we’ve done a lot of ghost-themed experiences, first on Auburn Avenue (an historic area in Atlanta) and later at the nearby Oakland Cemetery (where we had students do class projects for years in our Augmented and Mixed Reality Experience Design class).   One of the reasons ghosts “work” is that we all “know” how ghosts “behave”, and one of these behaviors is that they don’t stick to the real world, they float and shift and even appear to go through objects or walls.  Which is exactly what happens when the graphics aren’t registered.

So, the upside is, I’ll look forward to this game coming out, and like to see how it works.  There are lots of questions that remain to be answered before one can say if this will be good or not, but the idea is interesting and promising.