An In-Depth Analysis of the Nintendo 3DS

This week, Nintendo announced a successor to their wildly popular handheld gaming device, the DS, called the 3DS.  While details are sparse Nintendo did announce that this device would be able to display 3D images without the use of special glasses.  Nintendo hasn’t elaborated on how this technical feat is accomplished, rather they will defer details until its unveiling at the upcoming E3 on June 15th.

In term of innovation in gaming, Nintendo itself is a very aggressive company; from the design of the original d-pad on the NES/Famicom, to the analog-stick in the N64, to the touch-screen in the DS, to the motion controls in the Wii.  Nintendo has always been a company that experiments with new ways of interacting with technology, they have also had several lesser known, lesser successful, endeavors like the Famicom Disk System, Virtual Boy, 64DD, and the Satellaview that brought online gaming in the mid-90s.

However, Nintendo has not been a company that develops the key enabling technologies from scratch, rather they are a company that uses existing technologies and reinterprets them for entertainment purposes.  For this reason we can examine existing technologies and analyze how Nintendo may be adding 3D display technology to the 3DS.

Autostereoscopic Displays

The technology to display 3D without glasses has existed for close to a decade.  Sharp first released a cellphone in 2002 called the SH25LiS for NTT DoCoMo in Japan that came with a 3D display, this was one of the first models with a display switchable between 2D and 3D modes.  In 2004, Sharp announced a 15″ 3D monitor called the LL-151D, and they have has also released laptops with 3D displays as well.  Phillips until 2009 even sold a 42″ WoWvx TV.  In 2009, Hitachi released the Wooo H001 cellphone that also had autostereoscopic technology, and most recently Fuji has started selling a 3D camera and picture frame both of which have 3D displays.  However, the technology in its current form has its nuances and limitations that better suit smaller screened devices.

To understand these limitations we first need to understand how this technology works.  The visual ability to perceive the world in three-dimensions comes from the fact that each our eyes are receiving two slightly different images; this is called stereopsis.  The technical challenge is to insure that each eyes sees different images from the same display.  With 3D glasses, we can filter the image directly before it reaches the eye, however without glasses this presents a challenge.

There are two primary methods that this is accomplished.  The first method is through a lenticular lens screens, which has small micro-lenses that are put above each pixel, these lenses direct light from alternating pixels to different eyes.  The Phillips WoWvx uses this lenticular lens technology, and most recently, Sumimoto-3M, has also shown a lenticular lens display prototypes that utilize this method.

Sumimoto 3M Lenticular Lens 3D display

The second widely used method is the use of a parallax barrier that is placed in front of the LCD screen.  The images for the left and right eye are placed on alternating pixels; the parallax barrier acts to block out the light from the alternating pixel so that only the corresponding image for each eye is shown to the viewer.  The parallax barrier itself is merely a transparent monochromatic LCD that is placed above the main LCD display.  For this reason, parallax barrier itself can be completely turned off to display a conventional ’2D’ screen.  Sharp, Hitachi, and Fuji use this method for their 3D screens.

The biggest issue with either of these methods is that the 3D image can only be effectively viewed within a narrow spatial window.  For this reason, the viewer needs to be a set distance from display to properly get the 3D effect.  This is also the reason why autostereoscopic TV sets have not been pursued as aggressively for 3DTV sets; multiple viewers and variable viewing distance can quickly ruin the 3D effect.  Also, the larger the screen, the more effort is necessary to keep your eyes within that 3D sweet-spot, and prolonged viewing can quickly lead to fatigue.

For handheld devices, the viewing distance is a lot more predictable; the viewer’s eyes would be between a comfortable viewing distance to no further then an arms length away.  Even for a person with very long arms the actual viewing distance would be fairly consistent.  Furthermore, a handheld gaming device is much more personal, the concern about multiple viewers is also reduced, and the getting yourself in the 3D sweet-spot should be less of a problem with a portable device compared to a fixed monitor or TV set.

Faux-3D.

Another option is giving the user the illusion of 3D while using a conventional 2D screen.  As was mentioned on Kotaku, as well as here, there are various methods that would give the player the illusion of 3D.  The “iPhone method” is through the use of an accelerometer, where titling the device would produce a different field of view.  The problem is that the field of view is based off of the orientation of the device, not the viewer’s eyes.  For that reason the 3D effect is minimal, and accelerometer alone cannot effectively produce a faux-3D in all but the most limited environments.

The more effective faux-3D method is by using eye-tracking technology.  As Kotaku has pointed out, a similar technique is used by DSi game, 3D Hidden Picture, which uses a less sophisticated face-tracking technology (the video below demonstrates the efficacy of a face tracking on a Mario doll).  In that game, the camera is used to adjust the image orientation to hit the viewer’s field of view.  There are also commercial monitors with mounted cameras that use this same principle to produce a 3D-like effect.

Eye-tracking is well suited for portable gaming devices since the user would be positioned less then a meter away from the handheld device and camera.

Nintendo 3DS may use Natal-like technology

If a eye-tracking approach is used a crucial component in this capability would be the camera.  However using a conventional camera to do eye-tracking could be a problem for a portable device.  After all, we should expect that the 3DS would be used in  a large array of lighting environments, on a bus or a train, with constantly variable lighting, at home, on the toilet, or even in complete darkness.  For this reason, a CMOS camera using a normal image sensor cannot be relied upon to eye-track the user in all environmental conditions.

The solution to this, and the prevalent technique which most eye-tracking is done, is through the use of near-infrared TOF (time-of-flight) cameras.  This is the same technology that Microsoft is using to capture full-body motion in the Natal.  The basic principle of operation is that a diode sends out infrared light, and that light bounces off the user and captured by an infrared sensor, and the ‘time-of-flight’ of that infrared light can be calculated.  The main benefit of a TOF camera is that it operates irrespective of visible light.  It can even operate in complete darkness, and mated with a normal camera can cover a large range of environmental conditions.

There are key differences in how the Natal would implement this technology and how the 3DS would.  First and foremost is the fact that 3DS would capture the player less then a meter from the camera, for this reason the infrared diode and camera would be designed to meet these conditions.  Natal, comparatively needs to capture the entire body of the player, and in some cases multiple players.  From that data, Natal creates a 48-point skeletal model of the human player based of that data.  The 3DS would only need to do head-tracking or eye-tracking, and possibly hand-gestures.  Technically speaking, the 3DS would not have the processing requirements that Natal would need if they used a TOF camera.

TOF (Time-of-Flight) cameras can capture images in three-dimensional space that can be utilized in eye-tracking as well as gesture control

What would Nintendo do?

Asahi is reporting that Nintendo will be using Sharp’s displays for their 3DS.  Which means that Nintendo will be using the parallax barrier method to display 3D.  Asahi, we should assume, to be very reliable.  Asahi is a news organization that has a 130 year history in journalism, and are generally considered to be fairly conservative with strict editorial oversight for their business related news.  Sharp, also has a long history of being a supplier to Nintendo; including the DS.  Moreover, Sharp has invested in 3D technology since the early 90s, particular through their European arm.  If Nintendo where to choose a partner for 3D displays, Sharp would be a primary candidate considering their existing relationship with the company as well as their current technical prowess of this technology.

Another thing we should consider is that parallax displays and eye-tracking is not mutually exclusive.  There has been research in mating these two technologies, where eye-tracking could be used to adjust the efficacy of the parallax display.  But as we add technologies we need to consider the cost of these technologies as well.  Nintendo has traditionally shied away from high-cost gaming devices, and instead have relied on the elementary economic principle of supply-and-demand.  From this rationale, we should expect the Nintendo device to be within the ~$200 range, give or take, and we should not be expecting the addition of technology without direct purpose.

As for the parallax display, Sharp has had a difficult time promoting this technology due to the lack of content (its difficult to sell a 3D cellphone when only the menus are in 3D).  Teaming with Nintendo and their developer network gives Sharp dedicated content and a market for their technology.  To add a parallax barrier itself only requires a transparent monochromatic LCD.  What deal that Sharp has with Nintendo is still a mystery, or whether both screens with have 3D functionality.  However, we do know based on the Asahi report that the screen should be below 4″.  Based off of iSuppli, the 3.5″ within the Apple iPhone 3GS costs $19.25.  For this reason we should consider the BOM (bill-of-materials) for the dual LCD screens itself for the Nintendo 3DS to be over $40 without the 3D capability assuming they have a similar resolution.

In term of adding eye-tracking, from a cost perspective, a TOF camera’s CMOS sensor is not necessarily that different from the cameras already on the DSi.  A CMOS/CCD sensor has the ability to capture light in larger range then human’s perception of visible light, and for this reason most camera sensors go through the effort of keeping captured light only in the visible range.  Capturing excessive IR light in normal photography produces fake-colors and is considered an undesirable trait, but there have also been some specialty cameras like the Fujifilm S5 Pro that captures IR, as well as UV, images.  In the context of volume which would be necessary for the 3DS, an addition of a TOF camera would meet sufficient economies of scale, and the cost to implement it should not be significantly more then the two-cameras already present on the DSi, however it will still be an additional cost nonetheless.  Now, if Nintendo will implement parallax display and/or eye-tracking is still speculation.  However adding either or both should still be a significant cost to Nintendo.

Force-Feedback 3D Controls

Nikkei is reporting that the 3DS will have “a 3-D joystick and a force feedback mechanism that will let players feel the collisions of a game character, for example. It had already acquired related patents at the end of last year.“  The Nikkei we should also consider to be a very reliable source, they publish Japan’s primary stock market index, and is generally very accurate on what they report.  This “3D joystick” may only mean a traditional analog-stock, however, the patents Nintendo has acquired last year are of a different nature, so the Nikkei may be referring to something completely different.

Nintendo has traditionally avoided adding graphical enhancements without backing that technology up with a corresponding control scheme.  If we think back to the N64, Nintendo added an analog stick to move characters in three-dimensional space, and added 4 yellow “c” buttons to control the camera angle.  Mario 64 was a revolutionary game for the time, and it was the first true-transition from 2D gameplay into a fully realized 3D gaming experience.  Those “c” buttons has evolved into a second analog stick, but the original concept remains.

When we think of Avatar-like 3D, we expect objects coming out at the viewer in three-dimensional space.  Instinctively, we reach out to touch these visual objects to only have our hands pass through them.  Its an experience I’m sure we have all had when we first saw 3D at the theaters.  So we need to re-investigate the semantics; is Nintendo going to make ’3D controls’ in the vein of what currently exists in the form of analog sticks?  Or is Nintendo going to create ’3D controls’ that exist in three-dimensional space?  Assuming that its the latter there are several technical methods that this could be accomplished.

If Nintendo is going to add a IR TOF camera, the easiest method would be to do finger-tracking.  As the video below demonstrates, a user’s fingers can be made into a 3D mouse and be used to type on a virtual keyboard.  The force-feedback could be given via rumble in the device itself when a corresponding gesture is made.  While Nintendo implementing IR cameras is still speculation, if they are implementing the faux-3D approach via eye-tracking, this control mechanism could be added for little to no additional cost.  Again, much like eye-tracking, finger-tracking can be accomplished without the TOF cameras, however given the variable lighting conditions that the portable would be subjected to, a TOF camera would insure accuracy and consistency.

Another method force-feedback 3D controls could be implemented is through a stylus.  Nintendo has patented related technology, and these patents may be what Nikkei was referring to.  Within this patent, Nintendo refers to a game apparatus that is akin to a stylus, and when used in a gameplay environment, for instance attacking an enemy, vibrations of that gameplay action can be felt directly in your fingertips.  The benefit of using a physical device like a stylus is that it can have buttons on it as well.  If this approach is taken, the overall size and how such an input device would be powered is another question.  It might be a stylus-sized Wii remote, or it could be something completely different.  But there are examples of similar technologies:

Nintendo patent demonstrates force-feedback on a portable gaming device

In the video above, the news reporter is startled as the stylus device he is holding in his hands is mysteriously pulled towards the screen, he laughs as the stylus device is then pushing his hand away from screen.  The scientist explains that there is a small motor within the stylus, and the spinning of that motor in calculated directions can give force-feedback sensations.  The reporter then tests out the pinching and pulling of this 3D ball; he explains that there is a distinct sticky sensation as he moves his stylus.

Conclusion

The technologies presented here may, or may not, be used in the 3DS when its finally showed in June.  What we need to understand is relative to Nintendo’s announcement there is a limited number of options that can accomplish what Nintendo is claiming.  The announcement of the 3DS saw Nintendo stock surge almost 10%, all based on short PDF announcement without much detail.  There are still a lot of unanswered questions, but what we can count on is Nintendo is full of surprises.