Context & Challenge

IARPA's ReSCIND programme was built on a striking premise: that cognitive biases — the systematic errors in human reasoning documented across decades of behavioural science — could be deliberately exploited by cyber defenders to disrupt or deter adversary decision-making. The CASPAR team's task within this programme was to translate that premise into a scientifically credible empirical research programme, with the ultimate goal of producing Cognitive Cyber-attack Countermeasures and Mitigations (C3Ms): operationalised interventions a defender could deploy in a real network to trigger a measurable change in attacker behaviour.

The challenge was not a shortage of relevant science. Cognitive bias research is one of the most replicated bodies of work in psychology. The problem was one of construct validity and measurement design: it was unclear whether biases reliably observed in lab tasks and questionnaire studies would manifest in the same way when a skilled attacker was navigating a realistic network under operational pressure. Without a principled framework for defining what each cognitive vulnerability looks like in a cyber context, how to trigger it, and how to detect it, any experimental findings would be ambiguous — and the C3Ms built on them unreliable. Getting the measurement science right was a prerequisite for everything downstream.

Approach

• Developed a structured cognitive vulnerability taxonomy covering 18 biases across six categories — information quality and representativeness, loss aversion, information presentation, misleading feedback, information volume and difficulty, and illusion of control — with each bias given a precise construct definition grounded in the existing literature

• Built a trigger-and-sensor framework for each cognitive vulnerability: triggers are the specific environmental conditions that reliably induce a bias (e.g., presenting an attacker with a target framed as a loss rather than a gain); sensors are the observable behavioural signals that indicate the bias has occurred (e.g., persistence on a compromised machine, choice between attack strategies, estimation of success probability)

• Designed a three-tier measurement architecture — Bronze (established psychometric questionnaires, based on canonical items from Kahneman, Tversky, and related literature), Silver (purpose-built behavioural tasks in an accessible format), and Gold (live scenarios embedded in a simulated cyber testbed) — allowing construct validity to be established at each level of ecological realism before moving to the most resource-intensive testing conditions

• Specified a two-stage human subjects research programme: an open-world exploratory stage to identify which cognitive vulnerabilities show the strongest and most consistent effects across a dual cohort of non-expert and expert participants, followed by a focused confirmatory stage targeting the most promising candidates for C3M development

• Adopted Bayesian optional stopping as the core inferential methodology, replacing fixed sample sizes with adaptive data collection that terminates as soon as reliable evidence is obtained — whether for or against an effect — and explicitly avoiding the inflated Type I error rates that frequentist repeated-testing produces

• Authored the CogVuln Playbook — a formal IARPA deliverable designed for use by security personnel, combining relational diagrams mapping cognitive vulnerabilities to the Cyber Kill Chain, per-bias visualisations, and detailed scenario examples linking triggers, sensors, and hypothesised attacker responses

Outcomes

• A formal CogVuln Playbook delivered to IARPA as a programme milestone — covering the two mandatory constructs (representativeness bias and loss aversion) in full, with a structured framework extensible to the remaining candidate CogVulns

• A validated three-tier measurement design that gave the programme a credible path from laboratory-established effects through to testbed-based evidence, with clear criteria for when each measurement class is necessary and sufficient

• Construct-level trigger and sensor specifications for 18 cognitive vulnerabilities, giving the engineering team actionable, testable targets for scenario design in the cyber range rather than loose descriptions of psychological phenomena

• A Bayesian adaptive methodology that reduced expected sample size requirements while strengthening the inferential quality of the human subjects research, directly addressing the resource constraints of a government research programme

• A research roadmap — spanning open-world and focused experimental stages — that connected individual bias hypotheses to the programme's goal of producing deployable C3Ms, making the scientific and operational logic explicit for both researchers and stakeholders

Call to Action

If your organisation is designing research programmes, assessments, or AI tools that need to measure human cognition or decision-making in high-stakes, complex environments, I'd welcome a conversation.