Kinect in the OR: What Happened to Touchless Surgery

Here is the short answer. In the early 2010s, hospital teams in Toronto, London, and elsewhere adapted the Xbox Kinect so surgeons could navigate CT and MRI images with gestures from the sterile field. The idea was right. The projects stalled because the sensor was a consumer gaming product that was later discontinued, each system needed custom software for each viewer, and telling a deliberate command from ordinary movement proved hard. Today the same goal is met with a standard RGB camera, software that speaks mouse and keyboard to any viewer, and patented intent detection.

What were the Kinect operating room projects?

When Microsoft released Kinect in 2010, researchers saw what gamers did not: a cheap depth camera that could let a scrubbed surgeon control a screen without touching it.

• Sunnybrook Health Sciences Centre, Toronto. A surgical team and engineers built a Kinect system for navigating MRI and CT stacks during surgery. Sunnybrook reports that it grew into a company named GestSure.

• Guy's and St Thomas', London. Microsoft Research and clinicians developed a gesture and voice system for image-guided vascular surgery, described in the Communications of the ACM article "Touchless Interaction in Surgery."

• Many academic prototypes. A 2019 systematic review in the Journal of Medical Internet Research cataloged studies using off-the-shelf sensors such as Kinect and Leap Motion for gestures in surgery.

Surgeons liked it. The problem they were solving, leaving the sterile field to look at a scan, is just as real today. We cover it in how surgeons review imaging without breaking scrub.

Why did touchless surgery stall?

The hardware disappeared. Kinect was built for living rooms. Microsoft ended the consumer sensor, then Kinect for Windows, and officially discontinued Azure Kinect in October 2023. A clinical workflow cannot depend on a sensor with a gaming product's lifecycle. The full story is in Azure Kinect is discontinued: what now.

Every system was a custom integration. Most prototypes were written against one image viewer at one hospital. A new viewer, a new version, or a new site meant new engineering. That does not scale past a research grant.

Unintended gestures. This was the deepest problem, and the researchers said so themselves. The CACM authors describe "gesture transition" errors, where the movement into a gesture fires a command, and the difficulty of separating a deliberate command from a surgeon simply talking with their hands. Pause-to-activate schemes failed because surgeons naturally pause to look at the image.

The gesture vocabulary was small. Many systems supported next, previous, and zoom. Real image review needs window level, series changes, measurements, and 3D rotation.

What has changed since then?

• The camera is ordinary. Modern hand tracking runs on a standard RGB camera. There is no single-vendor sensor to be discontinued. Where a room needs depth or infrared, that is a choice, not a dependency. Through our partnership with Orbbec, AirTouch also takes advantage of 3D depth sensing and IR cameras when a deployment calls for them. See the Orbbec page.

• No viewer integration. AirTouch gesture control and hand tracking software works at the operating system level. Gestures become mouse, keyboard, and touch input, so any PACS client or visualization package responds as if someone were at the desk.

• Intent detection. Our patent covers the problem the Kinect era could not solve: identifying the person who means to control the screen and staying with them until they finish. Incidental movement is filtered out in most setups.

• A full input vocabulary. Fifteen gestures map to any keyboard shortcut or mouse action, saved per profile.

• Gloved hands. The models are trained for gloves, including colored surgical gloves.

Where did Kinect still have the edge?

Depth sensing handles dark rooms well, and full-body skeleton tracking enabled research that a hand tracker does not attempt, such as rehabilitation and movement analysis. If your project is body tracking, a depth camera with a body tracking SDK is still the right tool. Kinect's depth technology lives on in Orbbec's Femto line.

Is touchless control in the OR real now?

AirTouch is deployed in clinical settings today. It is computer input software, not a medical device, and we make no clinical outcome claims. Edge cases remain, which is why medical evaluations start in your room with your gloves and your viewer.

Book a demo and we will test against your environment, or start a free trial to try the interaction yourself.

Sources: Sunnybrook Health Sciences Centre, "Xbox Kinect in the hospital operating room"; O'Hara et al., "Touchless Interaction in Surgery," Communications of the ACM; "Use of Commercial Off-The-Shelf Devices for the Detection of Manual Gestures in Surgery: Systematic Literature Review," Journal of Medical Internet Research, 2019.

Related: voice, foot pedal, or gesture control in the OR, touchless control in the sterile operating room, and gesture control for medical environments.