SBIR-STTR Award

Most computer programs require both keyboard data entry and Point-and-Click (PAC) operations with a mouse or trackball device to operate efficiently. In a moving environment such as an airplane, car, bus, or armored vehicle, PAC operations become more difficult. Speech recognition can replace tedious data entry, and with domain specific grammar corrections it can ensure spoken data and dictation are recognized and entered properly. Adding a secondary device to complement speech input, which could perform all the PAC operations efficiently and require no steady hand movements, would permit full use of a computer in moving vehicles. The use of a multiple-controller approach enables the interaction to be tailored to the task and environmental constraints, as well as user preferences. A natural language software-based speech recognizer in combination with eye-tracking, facial gesture, brain wave and neuromuscular sensor technologies are to be configured in a variety of ways to provide the required functionality. Unique to this proposal is the development of an integrated set of human-computer control suite technologies using standard Application Program Interfaces (APIs) in the commercial Windows and Unix environment that are compatible with the Army Technical Architecture for efficient hands-free operation of computer systems. Some PAC needs could be met by using a combination of eye movement, facial muscle, and brain wave bio-potential combined technologies detected at the forehead, to generate computer inputs. Both the frequency bands and electrical signal content levels derived from these sensors have shown progress in the selective mind and voluntary neuromuscular control of a computer display pointer and clicking on a displayed selection. Projective task analysis, driven by a decision support activity scenario, will be conducted for various combinations of interface technologies to derive an effective and feasible PAC conceptual design integrated with natural language. Quantitative performance and qualitative human interaction analysis will be documented and provided with the conceptual design. This research would be helpful for the military in controlling combat simulations and interacting with command and control consoles, it would also make a useful commercial gaming device and afford many handicapped individuals, who cannot efficiently use a mouse, trackball, or keyboard; the opportunity to access the next generation internet.

Soldier performance in a modern battlefield environment is becoming increasingly demanding on the human sensory facilities and more lethal due to threat of biological, chemical, and other agents. The logical defense mechanism is to encapsulate and try to isolate the soldier from the hostile environment. However, when the human head is enveloped in a protective sealed helmet or the soldier is confined to a sealed vehicle; he becomes disoriented because of the denial or mismatch of ordinary sensor cues. The resultant effect of this disorientation causes the soldier to lose the vital situational awareness and the feeling of well-being needed to react to changing threats and to carryout his mission. The Point and Click/Speech (PAC/S) research opportunity is a solution to this and other human computer interface problems while maintaining local surrounding audio cue orientation and situational awareness. Unique TERI Natural Language Speech, integrated with enhanced speech recognition and language understanding, is to be seamlessly incorporated with tactical operation aids, 3-D audio cueing and eye-tracking, providing a cooperative modality that is user friendly and intuitive. Universal interface standards will be integrated among the PAC/S components, with application to Objective Force Warrior, Homeland Security Architecture, and Combat Decision Aid System, to offer an Army Technical Architecture and commercial compliant solution. The integrated PAC/S performance will be demonstrated using a "plug-in" architecture by a soldier to test OFW portable Multi-Modal Interface (MMI) suitability and to quantify benefits. Incremental deliveries of the enhanced MMI components are scheduled for delivery every six months. Tactical speech recognition research at TERI has made great strides in the past few years in terms of noise robustness and language processing, and is gaining acceptance as a keyboard replacement. A hands-free mouse (trackball) replacement that can be operated accurately in a moving vehicle or by a soldier in the field, will enable the warrior to navigate through menus, select items, and control C3 information technology components. A MMI low power lightweight PAC/S device will be delivered, providing unique tactical operation aids for an intuitive and natural feel. Operator training will be minimized, and reaction time will be improved. OFW soldiers using this MMI device will gain increased lethality and improved survivability.

Keywords:
Plug-In Architecture, 3-D Audio Awareness, Tactical Operation Aids, Natural Cultural Language Understanding, Natural Language Speech, Multi-Modal Inte