The purpose of this study was to evaluate the ability of expert blinding and consensus feedback to improve the validity of expert testimony, specifically in the context of forensic science.
The study conducted an experiment that consisted of developing a hypothetical criminal case with forensic evidence, carrying out an exercise with a panel of relevant experts to generate a consensus interpretation of the case evidence. The consensus exercise used a questionnaire consisting of the case description and the forensic question. Two rounds of the consensus exercise was needed to reach the unanimous solution, and this solution comprised the feedback for the justice survey.
The justice survey was conducted with a new sample of scientists using the consensus interpretation. The survey was designed and staged on SurveyMonkey, a web-based commercial tool for survey design and administration. Survey participants were randomly assigned to one of three types of party representation (prosecution, defense, or blinded). Approximately half the participants in each condition received consensus feedback in advance of providing a response, and the other half did not. In addition, the group without advanced feedback received an opportunity to change their baseline response following receipt of consensus feedback (a pre/post within-subjects measurement). After the completion of the survey, respondents were invited to a independently hosted website, SelectSurvey.net, to request compensation. This procedure allowed the partition of identifying information from survey responses.
Consensus exercise: Twelve RAND employees who held doctorates in behavior decision theory, biostatistics, clinical psychology, criminal justice, economics, experimental forensic psychology, industrial and organizational psychology, social psychology, psychology, and statistics.
Justice survey: Fourteen professional scientific societies were targeted. Eligible societies were those with a focus on social/behavioral sciences or statistics and with a primarily United States membership. Nine societies agreed to distribute the survey invitation to their membership list via e-mail or newsletter. 685 accessed the survey and 580 submitted completed surveys.
People with an advanced degree in a relevant scientific field where training would normally include exposure to Bayes theorem or equivalent experience.
Consensus exercise data (JusticeConsensusExerciseDataCollectionInstrument_20140716163600.pdf, n=12) consists of expert's answers to a forensic question.
Justice survey data consists of two Excel files.
N700_RawData_May1_2014 - Archive3.xlsx Excel file contains three worksheets.
Values worksheet (n=700, 35 variables) and Labels worksheet (n=700, 33 variables) consists of nearly the same variables with the Labels worksheet's values coded using the coding scheme. These worksheets contain variables on respondent's answer to a forensic question and their confidence in their answers, if a consensus response influenced their final answer, opinion on the survey, how much they would ask for payment for consulting for the prosecution and defense, if their answer would favor the party that hired them, how many times they served as an expert consultant for prosecution and defense, age, gender, race, highest degree earned and field, and other comments on the survey. Only in the Values worksheet are variables on survey exclusion and reason.
Summary worksheet contains summary on number of respondents per between-subjects survey condition and a diagram of the conditions.
N580_ProcessedData Oct2014 - Archive.xlsx Excel file (n= 580, 57 variables) contains one worksheet Merged Data. This worksheet contains the same variables as the Labels worksheet in the N700_RawData_May1_2014 - Archive3.xlsx Excel file with the addition of variables on type of survey taken; response to question and confidence level; if respondent has a doctoral degree, has experience, has a forensic degree; and statistics on respondent's answer.
82.9% of people who accessed the survey landing page submitted response data and were thus defined as participants.
Likert-type scale was used.