Towards Robust Experimental Design for User Studies in Security and Privacy

Background: Human beings are an integral part of computer security, whether we actively participate or simply build the systems. Despite this importance, understanding users and their interaction with security is a blind spot for most security practitioners and designers.

Aim: Define principles for conducting experiments into usable security and privacy, to improve study robustness and usefulness.

Data: The authors’ experiences conducting several research projects complemented with a literature survey.

Method: We extract principles based on relevance to the advancement of the state of the art. We then justify our choices by providing published experiments as cases of where the principles are and are not followed in practice to demonstrate the impact. Each principle is a discipline-specific instantiation of desirable experiment-design elements as previously established in the domain of philosophy of science.

Results: Five high-priority principles – (i) give participants a primary task; (ii) incorporate realistic risk; (iii) avoid priming the participants; (iv) perform double-blind experiments whenever possible and (v) think carefully about how meaning is assigned to the terms threat model, security, privacy, and usability.

Conclusion: The principles do not replace researcher acumen or experience, however they can provide a valuable service for facilitating evaluation, guiding younger researchers and students, and marking a baseline common language for discussing further improvements.