Maverick Engineering Social Media

Aiming to Help the Blind “See”

Electrical Engineering doctoral student Ninad Thakoor has embarked on an ambitious project – providing the blind and visually impaired with a wearable, interactive system that will give them better information on their indoor and outdoor surroundings.

For decades, researchers and tinkerers have created a broad, but often impractical, assortment of assistive devices to supplement or replace the cane as one of the primary sensory feedback mechanisms for the blind and visually impaired. Now, thanks to advanced technologies and clever thinking, Ninad and a team assembled by Computer Science & Engineering’s Dr. Jean Gao have created a working prototype of a device that makes a giant leap in sensory information for the blind and visually impaired. Other members of the team include Electrical Engineering’s Dr. Sungyong Jung; Richard Srebro, M.D., of the Ophthalmology Department at UT Southwestern; and Dr. Youngjoong Joo of the Electrical Engineering Department at Arizona State University.

The team calls their system IntelligentEyes. The system consists of four major components: eyeglass frame-mounted cameras providing real-time pattern recognition and distance information; a wireless, ultra-wideband transceiver sending video and voice signals to and from a reconfigurable processor; a battery-powered image processor analyzing information from the cameras; and voice recognition and generation software providing ease-of-use communications from and to the user. The prototype contains all but the voice capabilities of the system.

Ninad, a research assistant in the Computer Science & Engineering Department’s BioComputing and Vision Lab, has been working on the system for several years using some off-the-shelf hardware components and others created specifically for this purpose. He also assisted in the development of the video analysis and other software powering the system. Ninad began this work after being solicited by Dr. Gao, the lab’s director and who initiated this project.

“The IntelligentEyes system won’t replace the cane,” said Ninad. “It would be additional source of important information for a blind or visually impaired user, information on things beyond the length of the cane.” Wireless signals from the eyeglass-mounted cameras are processed to provide several types of information about what is in front of the user: A general description – a flat surface such as a wall, a person, an unknown object, a change in surface such as a step or curb; the distance to that object or surface change; its movement, such as an approaching person or animal, and the rate of speed of that movement.

This information would be processed through the voice generation software, creating a vocal response sent to a speaker mounted in the eyeglass frame or an earpiece and informing the user what was in his/her vicinity. The user would be able to tell the processor how frequently information should be updated and presented.

One enjoyable and productive step in the development of the IntelligentEyes system was a visit by Drs. Gao and Srebro to the Dallas Lighthouse for the Blind, a non-profit organization providing social services and employment opportunities for the blind and visually impaired. “I needed to learn directly from these individuals the kinds of information and feedback they wanted from an assistive device,” said Dr. Gao. “During our visit, I was continually amazed by the groups of workers assembling different products and mechanisms. It was also interesting to see how they made use of and depended on the senses they had to do their work and move about the facility.” Dr. Gao also visited the American Foundation for the Blind’s model home for blind or visually impaired people.

Though Ninad and the others have been developing the different components of the system for some time, it is still in its infancy. The image processing has been refined to target single objects, much as a sighted person’s vision, while capturing peripheral objects, disregards them to concentrate on whatever is the center of attention. And though the wireless signal transmission is working as planned, the individual units are not packaged in a manner suitable for consumer use.

“We’ve been supporting this project with departmental funds,” said Ninad. “We’ve approached some organizations for funding, but they said that while our project, if successful, was certainly groundbreaking, we may be a little ambitious in attempting to achieve so much in a small package. But that’s what makes it such an exciting project, one that we think we can achieve, one that will really make a dramatic improvement in the lives of the blind and visually impaired.”

In addition to conducting research and teaching students, Ninad has found time to write 16 journal articles and conference papers. Two of his papers were selected for presentation at the world’s most highly-recognized and sought-after conferences: The IEEE International Conference on Computer Vision (2005) and the IEEE CS Conference on Computer Vision and Pattern Recognition (2008). Two of his articles appeared in IEEE Transactions on Image Processing, a prestigious publication with stringent submission and acceptance standards. An additional four articles have been accepted for publication by respected journals and he is waiting for acceptance on one more.