The limits of current AI need to be tested before we can rely on their output
Dr. Craig Martell, Chief Digital and Artificial Intelligence Officer of the United States Department of Defense, has made an unusual call at DEF CON 31 in Las Vegas. He invited the audience to go and hack large language models (LLMs), a request that is rarely made by a government official. But why did he issue such a challenge?
Throughout the Black Hat 2023 and DEF CON 31 conferences, artificial intelligence (AI) and the use of LLMs have been hot topics. This is not surprising given the hype surrounding the release of ChatGPT just nine months ago. Dr. Martell, who is also a college professor, provided an interesting explanation and a thought-provoking perspective that engaged the audience.
He started by highlighting the concept of predicting the next word in a sentence. When a data set is built, an LLM’s job is to predict what the next word should be. For example, in LLMs used for translation, there are limited options for the next word that are semantically similar to the prior words. This is not a new concept, as we are accustomed to seeing predictions on the internet, such as when purchasing on Amazon or watching a movie on Netflix. Both systems offer predictions for the next product to consider or what to watch next.
In the context of building computer code, predicting the next word becomes simpler because there is a strict format that code needs to follow. Therefore, the output of an LLM for coding purposes is likely to be more accurate than for generating normal conversational language.
However, the biggest issue with LLMs is the occurrence of hallucinations. Hallucinations happen when the model outputs something that is false. Dr. Martell provided an example that happened with ChatGPT. He asked the model “who is Craig Martell,” and it replied with an answer stating that Craig Martell was the character played by Stephen Baldwin in the movie “The Usual Suspects.” However, this is not correct, as a simple search engine query would quickly verify. The concern arises when people accept the output of AI models without fact-checking or validating the information. Dr. Martell referred to those who don’t check the output as lazy, emphasizing the importance of validating all output using another source or method.
The presentation raised the question of how many hallucinations are acceptable and in what circumstances. In situations involving life and death decisions, such as a battlefield scenario, zero hallucinations may be the only acceptable answer. However, in the context of a translation from English to German, a certain level of hallucinations may be tolerable. Determining the acceptable number of hallucinations is a critical challenge.
Currently, human validation is necessary alongside LLMs. A human being should be involved in the validation process, and one LLM should not be used to validate the output of another. Human validation goes beyond logic and reasoning. For example, if someone sees a picture of a cat and an AI system identifies it as a dog, they can easily recognize the error. Humans have innate abilities, such as recognizing faces and understanding hunger, that surpass the capabilities of current AI systems. However, not all humans are aware that AI outputs need to be questioned. Many accept AI’s output as an authoritative answer, leading to significant issues depending on the context in which it is accepted.
In conclusion, the presentation emphasized that although the technology has been released to the public and is seen as an authority, it is still in its infancy and has much to learn. This is why Dr. Martell challenged the audience to “hack the hell out of those things, tell us how they break, tell us the dangers.” The Department of Defense has created a project for providing feedback, which can be found at www.dds.mil/taskforcelima.
The limits of current AI need to be thoroughly tested and understood before relying completely on their output. The potential risks and biases associated with AI technologies must be addressed to ensure that they can be trusted and effectively utilized in various contexts.