Why AI security matters

- [Instructor] Science fiction books and movies are full of general knowledge AI, also known as AGI or artificial general intelligence, hyper-smart, independent systems that can think on their own and even repair themselves if a malfunction occurs. The reality is that while AI can accomplish a dazzling array of sophisticated tasks, they're not yet sentient, even though conversations with large language models like ChatGPT, Claude, or Gemini might make it seem like they are. And unless some element of self-healing functionality is built in, they're not able to fix themselves if something breaks. That means it's up to us as builders and defenders to analyze where and how things can go wrong within AI systems. This lets us know where and how to build in resilience and security protections. An important first step to understanding how AI can fail is via threat modeling to determine what can go wrong, and risk assessments to understand the impact and likelihood of something bad happening. As security professionals, you probably already know how to do threat modeling. But it's important to realize it may look a little different with AI due to emergent properties related to the way that AI systems work. For example, one of the most important considerations is the level of trust that people may place in the output of an AI model. Today, many of us would have a healthy skepticism if we were in an autonomous self-driving vehicle that attempted to run a stop sign. But it's not very likely that the average person would manually check a complex calculation performed automatically within a spreadsheet. And that's where the problem lies. Once we feel comfortable with autonomy, we shift focus. Consider someone in a self-driving car. Rather than keeping an eye on the road, their intention may shift to reading or watching the news while the car is driving. In this example, the rider might be distracted and not even notice if that self-driving vehicle failed to stop when it should. AI systems are already being used autonomously, sometimes, with very limited human interaction. AI is being used to make investment decisions for robo-advisor management, retirement portfolios, to assess credit risk, and to parse resumes and identify which candidates will move to the interview phase. These are deeply important decisions that can impact people's quality of life. In business, AI systems are now investigating cybersecurity threats, providing customer support, and even autonomously coding software. Failures in these systems, changes, or exploitation of them by attackers can mean greatly increased business and operational risk. The trust that we put into these systems can also be exploited. For example, consider the potential for accidental or purposeful amplification of misinformation or disinformation. As our trust in these systems and the AI used to make those decisions increases, the responsibility to ensure they operate as expected with integrity also goes up. Another extremely important aspect of security and privacy in AI is the need for proper governance of data. If you've heard the phrase, "Data is the new oil," you may already be thinking about this. ML-driven AI requires huge amounts of data during the training and testing phases. Finding the right data that's clean and ensuring they are inclusive and not biased is a huge, critically important part of the training process. AI-enabled agents and assistants can have access to business critical and proprietary information, so they need to be hardened against leaking sensitive or proprietary information or purposeful extraction of data by attackers using techniques like prompt injection. Since many traditional apps don't need huge amounts of data for training, understanding the risks for these datasets may be a new consideration for developers and security testers. And the fact that these systems are new in the market means that there aren't a lot of people who have experienced designing and building them securely. This consideration is compounded by the fact that we may not always know where or even if AI is present. If, for example, you are creating an app that uses services and functions via third-party APIs, such as a financial offering that pulls credit scores from an external credit provider, you may have hidden AI in your composite application. To ensure the overall reliability of AI, you'll need to assess the security of the AI and any services or functions it calls. This is often more involved than evaluating just one or two integrated services or functions. Agent-based systems, for example, might be able to call dozens or even hundreds of individual services, data stores, tools, functions, or APIs. Lastly, we should be extremely diligent with security and resilience in AI due to the potential for negative impacts. Risk is often described as a measure of likelihood times impact, and the impact of bias or malfunctioning AI could have severe negative consequences on people's lives. In some cases, this impact can extend to the most critical safety considerations. Imagine, for example, an AI system used to review medical images to determine if malignant tumors are present. A misclassification of a tumor as benign, when in fact, its stage 4 would be disastrous. Or imagine a bank's AI customer service agent providing homeowners with incorrect mortgage payment information, leading to delinquent payments, and possibly, the loss of their home. Human resilience and the accuracy of AI outputs is why it is so important for builders and security teams to perform proper threat modeling and to design AI systems with security and reliability in mind.