TIPS #5: LLMs in software development: the rewards are clear, but what about the risks?

Welcome to the fifth edition of #ForgepointTIPS (Threat Intelligence Portfolio Spotlight), where we examine the latest cybersecurity trends and threats and provide actionable insights for industry decision-makers. Today we explore #LLMS: how they’re changing the game in software development and how companies can reap the benefits without undue risk. Helpful reading? Subscribe and tell a friend…

Issue: Large language models (LLMs) are being rapidly integrated into the development cycle- but companies lack a full understanding of the technology and risks.

ChatGPT. Github Copilot. These are just two of the most popular large language models (LLMs) developers are leveraging to enhance development, design, delivery, and user experience in enterprise products. A June 2023 survey by Github found that 92% of US-based developers currently use AI coding tools both at and outside of work.

The productivity gains from using LLMs are significant. Developers can use LLMs to generate code, automate repetitive tasks, write scripts, and debug code more efficiently. LLMs are also helpful learning resources that can offer insights on code databases, solutions to problems, and recommended best practices.

However, LLMs are constantly changing, making it difficult to comprehend technological minutiae and associated risks. At the same time that companies rapidly integrate LLMs, many development and security teams lack expertise with the technology.

Impact: Reckless adoption risks security vulnerabilities, code sprawl, and declining proficiencies 

Using AI effectively requires an expert understanding to craft the right prompts, interpret outputs, and avoid high-risk use cases. Companies that adopt LLMs recklessly face a number of risks.

There are outright security risks- like prompt injections (bad actors manipulating AI behavior), data poisoning (bad actors manipulating AI training data), and data leakage (AI revealing sensitive information)- which must be understood to use LLMs safely.

There’s also the issue of AI output reliability. LLMs aren’t known for their high level of accuracy (particularly if they are trained on unverified data), introducing the potential for errors and vulnerabilities in code. LLMs are also prone to hallucinations (referencing non-existent functions, routines, or resources), which can go unnoticed by less-experienced developers.  If a developer relies upon a misinformed LLM-generated code solution, it can cause ripple effects in the end products consumers use.

In addition, LLMs have the potential to compound existing issues around huge codebases. Companies have struggled to manage and secure growing codebases for years. LLMs may make this problem worse if used to generate high quantities of code without limitation.

Overreliance upon LLMs has a potential long-term risk as well. Teams who lean too heavily on LLMs may lose their subject matter proficiency over time, leading to downstream security risks including a growing skill gap and the inability to contextualize LLM outputs.

Action: Position your company to use LLMs safely and effectively

1. Cautionary adoption and embedded security

LLMs are here to stay and can improve business efficiency and product development. Make sure your company understands any new AI technology- how it works, its underlying security risks, and its most relevant use cases- before implementing it.

Secure by design and secure by default principles apply here: the way your company uses LLMs in the product design, development and delivery process should prioritize consumer safety and business security.

2. Automate static application security testing (SAST) in your software development cycle

To counter LLM hallucinations, practice SAST in your continuous integration and continuous deployment (CI/CD) pipeline. This ensures any LLM-generated code is effective and secure before it’s live in your products.

3. Upskill and get outside of your team’s bubble 

There are a lot of great shared resources, best practice recommendations, and case studies (good and bad) your company can learn from. Follow industry guidelines, consortiums, and leading practitioners who share information about LLM utilization.

For example, ReversingLabs recently wrote about OWASP’s efforts to create a top 10 list of LLM vulnerabilities to educate security and development teams. These and other emerging awareness efforts help companies address salient risks and bridge skill gaps in development and security teams.

Well-informed teams will be equipped to handle the ever-changing nature of AI and its integration into the enterprise. Companies like Secure Code Warrior help developers learn secure coding skills.

4. Secure your software supply chain 

As businesses increasingly leverage AI, third party software, and open source code, it’s more important than ever to have a transparent view of the numerous components in your products and how secure they are. Companies like Whistic help you automate vendor security assessments.

Create a SBOM to illuminate the underlying relationships, potential risks, and overall scope of your software. Companies like ReversingLabs give you visibility into your software supply chain to secure your products and customers.

Thanks for reading Forgepoint TIPS! Please subscribe and share with a peer. Have feedback or a cyber threat or trend you’d like us to address? Get in touch.

***This blog was originally featured on our Forgepoint TIPS LinkedIn newsletter. Read the original post on LinkedIn here.***