Stefano Bennati, HERE Technologies; Engin Bozdag, Uber
In the world of Trust-by-Design, technical privacy and security reviews are essential for ensuring systems not only meet privacy standards, but also integrate privacy from start. However, as companies grow and diversify their technology, the process of conducting these reviews becomes more challenging and expensive.
This challenge is particularly evident in agile environments, where frequent releases of small software components need to be reviewed in a timely manner. Scale worsens this challenge, where the number of reviews from thousands of developers and microservices can easily overwhelm a small team of privacy engineers.
The quality of documentation also plays a significant role: poor or incomplete documentation can result in wasted efforts on review that present little privacy risk, or even worse, can result in overlooking serious privacy concerns.
The challenge of identifying low-risk items that don't need a review (false positives) and high-risk items skipping the review (false negatives) becomes a critical task for maintaining privacy-by-design effectively across the organization.
This presentation will explore how Uber and Here Technologies have worked to improve efficiency of their review triage processes via automation. Large Language Models (LLMs) are suited to assess the completeness of technical documentation and classify a feature/project into high or low risk buckets, due to the textual representation of information and the models being trained on privacy and security concepts. We will demonstrate how we have adopted LLMs in the triage phase and how we identified that LLMs are not suited to perform full reviews and remediate issues without supervision, as they struggle reaching factual and logical conclusions.
Attendees will learn how AI can enhance efficiency for privacy engineers, and the most effective technologies and strategies, such as policy writing, dataset validation, prompt engineering with detailed decision trees and fine-tuning. The discussion will also cover the balance between performance (e.g. model accuracy VS human labeling, false negatives) and cost (e.g. workload reduction, computational expense). As an example, we will show how using gates (decision-trees) in GPT4 prompts allowed us to reach accuracy rates up to 90% but with high costs.
The talk will conclude with a discussion of the limitations of this approach and future directions in this area.
Stefano Bennati, HERE Technologies
![](https://www.usenix.org/sites/default/files/styles/speaker_photo/public/bennati_stefano_200x230.jpg?itok=tXVgx1a0)
Stefano is Principal Privacy Engineer at HERE Technologies. He holds a PhD in Privacy algorithms, and authored several scientific publications and patents in the location privacy domain.
At HERE, Stefano provides technical guidance to product teams on building privacy into products. He also builds privacy-enhancing technologies for internal use cases, e.g. data lineage, as well as external use cases, e.g. Anonymizer, HERE's first privacy-focused product.
Engin Bozdag, Uber
![](https://www.usenix.org/sites/default/files/styles/speaker_photo/public/bozdag_engin_200x230_1.jpg?itok=9PXA8Lp-)
Engin is Uber's Principal Privacy Architect and the team lead of Uber's Privacy Architecture team. He holds a PhD in AI Ethics and authored one of the first works on algorithmic bias. He also helped create ISO31700 (the world's first standard on Privacy by Design) and OWASP AI Security and Privacy Guide. Engin has gained extensive experience in diverse organizational settings, cultivating a privacy-focused career that has evolved over the course of a decade. Throughout his journey, he has assumed multifaceted roles, encompassing legal expertise, privacy engineering,engineering management, research, and consultancy in the realm of privacy.
![](https://www.usenix.org/sites/all/modules/usenix/usenix_files/images/usenix-locked.png)
author = {Stefano Bennati and Engin Bozdag},
title = {Automating Technical Privacy Reviews Using {LLMs}},
year = {2024},
address = {Santa Clara, CA},
publisher = {USENIX Association},
month = jun
}