What tomorrow's evaluation looks like today: Lessons from the evaluation of the RANI Project, a first of its kind social norms-based cluster randomized controlled trial (RCT) in India
Summary:
The SBCC field's ability to generalize, with scientific rigor, findings from interventions has been circumscribed because of two methodological shortcomings: inability to attribute outcomes to the intervention because of study design limitations and an excessive reliance on self-reports to assess study outcomes. We address both shortcomings in the Reduction of Anemia through Normative Innovations (RANI) Project. To enhance attribution ability, we run a cluster randomized controlled trial in which 30 clusters (across 81 villages) are assigned to either a social norms-enhanced or an usual-care arm. To address limitations due to self-reports, we assess variables through surveys and other means, including anthropometry, hemoglobin counts, heartbeats after undergoing a step test, physical activity assessment through ActivPal accelerometry, and cognitive functioning through latent response time measures. Because of supplemental measures, participant burden is a potential error source, which we address through (a) a random selection of a subset of participants who undergo all study assessments (whereas others only provide limited number of measures) and (b) a planned missingness design in which only a subset of variables are assessed among a subset of participants. We report on important lessons learned in addressing these methodological and study design limitations through such a complicated (and a first-of-its-kind) design that allows us to assess the added value of the proposed innovations. We conclude with recommendations for future SBCC interventions seeking to implement similar designs, addressing the enhanced costs and the associated added value in scientific rigor.
Background/Objectives:
Past SBCC conferences have emphasized importance of evidence-based social and behavior change communication strategies to promote recommended social and health behaviors. In the field, exclusive reliance on self-reported measures has often led to defensiveness and second-guessing of results. Novel approaches are needed to generate intervention impact evidence, which can be triangulated by measures from multiple modalities that complement self-reports. Yet doing so appears cost prohibitive and likely add to participant burden, thus reducing overall data quality and confidence in findings. Our presentation describes an innovative resource allocation design to balance these competing interests in field experiments.
Description of Intervention and/or Methods/Design:
We evaluate the Reduction in Anemia through Normative Innovations (RANI) Project using a cluster RCT and a longitudinal design. The evaluation is being carried-out in 15 treatment and 15 control clusters, stratified by proportion in a scheduled caste or tribe, covering 81 villages. It uses multiple innovative methods to triangulate self-reported data through questionnaires to assess Iron Folic Acid (IFA) consumption, knowledge and perceptions about anemia and IFA, self-efficacy, social norms, mental health status, diet, and quality of life. In addition, we assessed: hemoglobin levels with a HemoCue photometer; anthropometric measures for body mass index; work capacity through Queens College Step test; Simon (computerized) and Corsi Blocks (non-computerized) tasks for cognitive functioning; and three-day physical activity (reclining, sitting, standing, and walking) tests through ActivPAL accelerometers.
Results/Lessons Learned:
Using 51 data collectors in 8 teams, we collected data (N=4,110) over six-weeks. Participant burden, a primary challenge in our study, was minimized in two ways. First, we used a randomized design to delineate two groups in each arm. A smaller (n=375) non-pregnant sample provided all survey data; anthropometry; hemoglobin count for anemia assessment; step test for assessing work capacity; response time (in milliseconds) and other measures of cognitive functioning; and a three-day ActivPal accelerometry. Another larger group (n=3,735) provided only survey data, supplemented with HemoCue measures and anthropometry. We also executed a planned missingness design in which participants were divided into four groups, one of which received the full questionnaire battery, and three others received only the primary variables (approximately 75% of survey items) and a subsample of other variables. We will describe the planning and logistics required for executing this design in the field.
Discussion/Implications for the Field:
Rigorous SBCC evaluations must adopt state-of-the-art knowledge and technology to assess behaviors and their manifestations. Some lessons we learned are: laying out study components, developing study instruments & pilot testing; using technology-enabled data collection methods; proactively training staff; running intra- and intercoder reliability statistics and providing feedback in real time; and adapting as necessary to ensure data quality. Through this experience, we believe it is possible to generate SBCC evidence beyond self-reports, though this effort comes with challenges. We will describe these challenges and how we overcame some and were less able to address others.
Abstract submitted by:
Satyanarayan Mohanty - Satyanarayan Mohanty
Rajiv Rimal - Johns Hopkins University
Hagere Yilma - George Washington University
Ichhya Pant - George Washington University
Erica Sedlander - George Washington University
Bikash Kumar Panda - Dcor Development Corner
The SBCC field's ability to generalize, with scientific rigor, findings from interventions has been circumscribed because of two methodological shortcomings: inability to attribute outcomes to the intervention because of study design limitations and an excessive reliance on self-reports to assess study outcomes. We address both shortcomings in the Reduction of Anemia through Normative Innovations (RANI) Project. To enhance attribution ability, we run a cluster randomized controlled trial in which 30 clusters (across 81 villages) are assigned to either a social norms-enhanced or an usual-care arm. To address limitations due to self-reports, we assess variables through surveys and other means, including anthropometry, hemoglobin counts, heartbeats after undergoing a step test, physical activity assessment through ActivPal accelerometry, and cognitive functioning through latent response time measures. Because of supplemental measures, participant burden is a potential error source, which we address through (a) a random selection of a subset of participants who undergo all study assessments (whereas others only provide limited number of measures) and (b) a planned missingness design in which only a subset of variables are assessed among a subset of participants. We report on important lessons learned in addressing these methodological and study design limitations through such a complicated (and a first-of-its-kind) design that allows us to assess the added value of the proposed innovations. We conclude with recommendations for future SBCC interventions seeking to implement similar designs, addressing the enhanced costs and the associated added value in scientific rigor.
Background/Objectives:
Past SBCC conferences have emphasized importance of evidence-based social and behavior change communication strategies to promote recommended social and health behaviors. In the field, exclusive reliance on self-reported measures has often led to defensiveness and second-guessing of results. Novel approaches are needed to generate intervention impact evidence, which can be triangulated by measures from multiple modalities that complement self-reports. Yet doing so appears cost prohibitive and likely add to participant burden, thus reducing overall data quality and confidence in findings. Our presentation describes an innovative resource allocation design to balance these competing interests in field experiments.
Description of Intervention and/or Methods/Design:
We evaluate the Reduction in Anemia through Normative Innovations (RANI) Project using a cluster RCT and a longitudinal design. The evaluation is being carried-out in 15 treatment and 15 control clusters, stratified by proportion in a scheduled caste or tribe, covering 81 villages. It uses multiple innovative methods to triangulate self-reported data through questionnaires to assess Iron Folic Acid (IFA) consumption, knowledge and perceptions about anemia and IFA, self-efficacy, social norms, mental health status, diet, and quality of life. In addition, we assessed: hemoglobin levels with a HemoCue photometer; anthropometric measures for body mass index; work capacity through Queens College Step test; Simon (computerized) and Corsi Blocks (non-computerized) tasks for cognitive functioning; and three-day physical activity (reclining, sitting, standing, and walking) tests through ActivPAL accelerometers.
Results/Lessons Learned:
Using 51 data collectors in 8 teams, we collected data (N=4,110) over six-weeks. Participant burden, a primary challenge in our study, was minimized in two ways. First, we used a randomized design to delineate two groups in each arm. A smaller (n=375) non-pregnant sample provided all survey data; anthropometry; hemoglobin count for anemia assessment; step test for assessing work capacity; response time (in milliseconds) and other measures of cognitive functioning; and a three-day ActivPal accelerometry. Another larger group (n=3,735) provided only survey data, supplemented with HemoCue measures and anthropometry. We also executed a planned missingness design in which participants were divided into four groups, one of which received the full questionnaire battery, and three others received only the primary variables (approximately 75% of survey items) and a subsample of other variables. We will describe the planning and logistics required for executing this design in the field.
Discussion/Implications for the Field:
Rigorous SBCC evaluations must adopt state-of-the-art knowledge and technology to assess behaviors and their manifestations. Some lessons we learned are: laying out study components, developing study instruments & pilot testing; using technology-enabled data collection methods; proactively training staff; running intra- and intercoder reliability statistics and providing feedback in real time; and adapting as necessary to ensure data quality. Through this experience, we believe it is possible to generate SBCC evidence beyond self-reports, though this effort comes with challenges. We will describe these challenges and how we overcame some and were less able to address others.
Abstract submitted by:
Satyanarayan Mohanty - Satyanarayan Mohanty
Rajiv Rimal - Johns Hopkins University
Hagere Yilma - George Washington University
Ichhya Pant - George Washington University
Erica Sedlander - George Washington University
Bikash Kumar Panda - Dcor Development Corner
Source
Approved abstract for the postponed 2020 SBCC Summit in Marrakech, Morocco. Provided by the International Steering Committee for the Summit. Image credit: Dcor Development Corner