Due to the necessity of operating alongside humans piloted vessels in open oceans and busy littorals, Unmanned Surface Vehicles (USVs) need a means to effectively communicate with such vessels when existing methods of safe navigation, including Automatic Identification System (AIS) and COLREGS, fail to fully address non-standard maneuvering scenarios. To achieve this, TDI Technologies, Inc (TDI) proposes Contextual Observations for Natural Speech CharacterIzation On Unmanned Ships (CONSCIOUS) to give unmanned vessels the ability to intelligently understand and interact with human piloted ships over VHF bridge-to-bridge radio. While many maritime inter-vessel interactions can be safely maneuvered using COLREGS standards, a large set of these interactions, including multi-vessel interactions, is not fully prescribed. When faced with these gaps in COLREGS specifications, mariners often turn to VHF bridge-to-bridge communications to negotiate safe passage. This is a challenge for USVs as they have difficulty understanding context and meaning in unstructured data, including spoken word. In order to successfully operate for long durations with minimal or no human intervention, it is critical to close this technology gap, and give USVs the ability to convey maneuvering intent to and understand the intent of human piloted vessels using Natural Language Processing (NLP). The CONSCIOUS NLP engine is tailored to accurately identify the intent of a human speaker by (i) training its linguistic models with a maritime focused collection of texts (e.g. boating forums, boating handbooks), known as a corpus, and (ii) developing a Long Short-Term Memory (LSTM) classifier that captures conversational context by combining the critical information from previous transmissions with the most recent transmission. After a transmission, if warranted, CONSCIOUS generates a response to convey the USVs intentions and converts it to spoken word using Text To Speech (TTS). After each conversation, CONSCIOUS computes a qualitative metric, using ship movement data (synthetic in Phase I and captured live from AIS in Phase II) that describes how well it understood human intent and how well it conveyed the USVs intent, which allows the CONSCIOUS team to see the effects of software modifications and focus efforts on poor performing TDIs existing work on the Navys Prototype USV (PUSV), make it well suited to understand the software interactions and interfaces on the Navys USV efforts. With this knowledge, the team will develop CONSCIOUS using Unmanned Maritime Autonomy Architecture (UMAA) standards and Data Distribution Service (DDS) for message handling as prescribed by current and future Navy USV efforts, allowing for a streamlined transition to the Fleet.
Benefit: As we transition this technology to Phase II, we will work with the Unmanned Maritime Systems Program Office (PMS-406) to ensure that we are incorporating their long-term visions for the roles and responsibilities of USVs as part of the U.S. Navy Fleet. During a Phase II program, the CONSCIOUS team will work to make this system more feasible for operational use, including the ability to synthesize more human sounding responses as well as the ability to perform on-board VHF triangulation. On-board VHF triangulation will allow the USV to know which ship a voice transmission comes from even if they do not identify themselves. Furthermore, this technique will help to prevent spoofing (e.g. a malicious actor on land posing as if they are a ship to redirect the USV). During Phase II, TDI will seek to partner with a USV autonomy provider to allow for an on-water demonstration with a USV, of a relevant scale, where the USV interacts with neighboring ships via VHF. As the program transitions to Phase III, TDI will work to modularize the CONSCIOUS system into a single hardware and software package that can be inserted into a standard server rack. At this time, we will seek to expand the potential customer set for this technology. TDI has existing work with NIWCs Electronic Harbor Surveillance System (EHSS), which monitors ports and harbors for dangerous or malicious activity. The CONSCIOUS package would be a useful additional to the existing sensor suite in that it can autonomously identify anomalous VHF conversations or potential traffic issues within a busy harbor. We will also seek to apply this technology to commercial shipping providers such as Maersk and Rolls-Royce, who are developing their own USV capabilities. In fact, there exists an alliance of shipping companies, the Advanced Autonomous Waterborne Applications Initiative (AAWA) that aims to have autonomous ocean-going vessels by 2035. Meanwhile, the global merchant fleet currently stands at 53,000 as of 2019, even if 25% of those vessels becomes autonomous in the future, that represents 13,000 vessels that need a method to communicate with human piloted vessels. At this point in development, it is expected that CONSCIOUS will be a plug and play solution that is mature enough for the commercial market.
Keywords: speech to text, speech to text, Unmanned Speech, Speech Recognition, Natural Language Processing, Voice to text, VHF
