Automating Technical Privacy Reviews Using LLMs

In the world of Trust-by-Design, technical privacy and security reviews are essential for ensuring systems not only meet privacy standards, but also integrate privacy from start. However, as companies grow and diversify their technology, the process of conducting these reviews becomes more challenging and expensive.

This challenge is particularly evident in agile environments, where frequent releases of small software components need to be reviewed in a timely manner. Scale worsens this challenge, where the number of reviews from thousands of developers and microservices can easily overwhelm a small team of privacy engineers.

The quality of documentation also plays a significant role: poor or incomplete documentation can result in wasted efforts on review that present little privacy risk, or even worse, can result in overlooking serious privacy concerns.

The challenge of identifying low-risk items that don't need a review (false positives) and high-risk items skipping the review (false negatives) becomes a critical task for maintaining privacy-by-design effectively across the organization.

This presentation will explore how Uber and Here Technologies have worked to improve efficiency of their review triage processes via automation. Large Language Models (LLMs) are suited to assess the completeness of technical documentation and classify a feature/project into high or low risk buckets, due to the textual representation of information and the models being trained on privacy and security concepts. We will demonstrate how we have adopted LLMs in the triage phase and how we identified that LLMs are not suited to perform full reviews and remediate issues without supervision, as they struggle reaching factual and logical conclusions.

Attendees will learn how AI can enhance efficiency for privacy engineers, and the most effective technologies and strategies, such as policy writing, dataset validation, prompt engineering with detailed decision trees and fine-tuning. The discussion will also cover the balance between performance (e.g. model accuracy VS human labeling, false negatives) and cost (e.g. workload reduction, computational expense). As an example, we will show how using gates (decision-trees) in GPT4 prompts allowed us to reach accuracy rates up to 90% but with high costs.

The talk will conclude with a discussion of the limitations of this approach and future directions in this area.