Visuo-Haptic Augmented Reality

Human perception is multi-modal: the senses of touch and vision do not operate in isolation, but rather closely coupled. This observation has inspired systems that allow users to see and touch virtual objects at the same location in space. The movie on the right shows one of our systems, which allows users to see and touch a virtual car. Using a head-worn display, the user can see the virtual car (the small inset on the bottom right of the movie shows the user's view). The user can also feel the car with his right hand through a haptic device called Phantom.

An easier way is to use passive haptics: overlaying graphics on physical objects. The movie on the right shows a user holding a milk carton (prepared with visual markers) in his hand. As long as the virtual objects coincide with real objects, the illusion works. We can also display slight variations in shape; however, major changes to the shape require a different approach.

Using our visuo-haptic platform, we have implemented a painting application. Users can paint with a virtual brush on a virtual, earthenware teacup (our teacup is a traditional Japanese teacup, called chawan). Our goal was to make this interaction as easy as possible. We decided to let the users control the virtual teacup with a graspable object. Additionally, we have invented a new interaction technique to make color selection from real objects very easy.

Most recently, we are refining this system in three ways: 1) we are using Canon's newest head-worn display (VH-2007) and their newest Software Platform (MRPlatform IV). To our knowledge, there are only two labs that have access to these. 2) We are collaborating with Tokyo Institute of Technology's Sato Lab. We will soon receive a brand-new prototype of their SPIDAR haptic device. 3) We are currently performing sub-millimeter calibration of the Phantom with a CNC Mill.

Canon's VH-2007 head-worn display.

Calibration setup for Phantom and CNC Mill.

Sato Lab's SPIDAR device.

People

	Michael Csongei
	Ulrich Eck
	Liem Hoang
	Kenny William Nyallau
	Matthew O'Loughlin
	Dr. Christian Sandor
	Peter Weir

Publications

Copyright Disclaimer: This material is presented to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each authors copyright. In most cases, these works may not be reposted without the explicit permission of the copyright holder.

The colors next to the publications denote the lab in which they were created:

Conferences

2007
		Sandor, C., Uchiyama, S., and Yamamoto, H. Visuo-Haptic Systems: Half-Mirrors Considered Harmful WHC '07: Proceedings of the Second Joint EuroHaptics Conference and Symposium on Haptic Interfaces for Virtual Environment and Teleoperator Systems, pages 292--297, Tsukuba, Japan, 2007. PDF | video | bib
		Sandor, C., Kuroki, T., Uchiyama, S., and Yamamoto, H. Exploring Visuo-Haptic Mixed Reality Technical Report PRMU2006-199 IEICE, Tokyo, Japan, 2007. PDF | bib

Patents

2006
		Kuroki, T., Sandor, C., and Uchiyama, S. Information Processing Method and Apparatus. Japanese patent 2006118446 (Filed 5/2006 as P2007- 293453A). May, 2006. bib
		Sandor, C., Kuroki, T., and Uchiyama, S. Information Processing Method and Apparatus. Japanese patent application P2007-293412A (Filed 4/2006) April, 2006. bib
		Sandor, C., Kuroki, T., and Uchiyama, S. Information processing method and device for presenting haptics received from a virtual object.Japanese patent 2006117732 (Filed 4/2006 as P2007-293413A). Patent in China, Europe, and US G10035198 (Filed 4/2007 as 2007/0247393). April, 2006. bib

Demos

2011
		Swoboda, M., Nguyen, T., Eck, U., Reitmayr, G., Hauswiesner, S., Ranftl, R., and Sandor, C. BurnAR: Feel the Heat Demo at IEEE International Symposium on Mixed and Augmented Reality, Basel, Switzerland, October. Best Demo Award, 2011. PDF | bib