BALID Informal Literacy Discussion 13 and AGM
27th November 2013
Literacy and learning for development:
Reflections on policy-focused research
Anna Robinson-Pant, University of East Anglia
What does it mean to conduct commissioned ethnographic research for policy makers on literacy and learning?
In this session, I shared my recent experience of working with a research team in Cambodia investigating how learning can help enhance rural livelihoods. Led by a major international development agency, the research project is shaped by global policy priorities, and aims to improve agricultural and educational policy and programmes for young people in Cambodia. By adopting an ethnographic perspective on learning and literacy, the project team faced the challenge of mediating the aims and findings to stakeholders unfamiliar with this kind of research. Through sharing insights into this research study on young people’s everyday learning, I explored some questions that had arisen for me as an ethnographic researcher.
I have previously conducted commissioned research similar to this study in Cambodia and long reflected on the differing assumptions held by policy makers and ethnographers around research questions and the research process. I wrote about these tensions from my experiences in Nepal on a USAID-funded project which aimed to analyse the relationship between women’s literacy and health outcomes (see ‘Women’s literacy and health: can an ethnographer find the links?’ in Street, B. (2001) Literacy and Development: Ethnographic Perspectives). Unlike the earlier Nepal research study, the project in Cambodia had more exploratory aims, appropriate to an ethnographic approach. The study was intended to ‘deepen understanding of how teaching and learning for agriculture and rural livelihoods is communicated to rural youth’ [from project TOR] in order to ‘contribute to detecting innovative teaching and learning as well as new ways of promoting rural employment’. The study also had participatory aims in terms of taking account of young people’s views and set out to undertake ethnographic research to see how they were learning agricultural skills and knowledge in everyday life.
I was appointed to work with the Cambodian research team who were recruited from a Government agricultural research institute. My major role was to give methodological advice and training as they had no previous experience of qualitative research. The overall research design was developed by myself and colleagues from the international development agencies funding the study, with the aim of conducting and comparing similar research in other contrasting country contexts. In the BALID meeting, I focused on discussing the research process and the methodological issues that arose – rather than the actual findings around literacy and livelihoods in Cambodia, partly because the research project was still ongoing.
In order to ensure that the research aims and methods were appropriate for the Cambodian context, an initial workshop was held in Phnom Penh with government and NGO stakeholders to discuss specific aspects of the research design. At this workshop, several issues arose from the stakeholders’ perspective:
- They questioned taking the definition of youth as 15 – 24 years old (this UN definition had been taken for the project in order to ensure comparability across the other countries where research would be undertaken). In Cambodia, workshop participants considered this to be a ‘severe bias’ as under-25s were born in the late 1980s, after the Khmer Rouge regime, so had a very different outlook on life. In the end, the research team ended up interviewing respondents over the age of 24 in order to explore their contrasting experiences on agriculture and learning.
- The research design documents specified that two research sites should be selected. However, participants at the workshop felt that three sites were needed to represent the three major contrasting topographical zones in Cambodia (coastal, plateau and lowland). Due to time and resource constraints, it was not possible to extend to three field sites.
- The research team, being used to experimental design, wanted to sample respondents according to ‘receivers’ and ‘non receivers’ of educational interventions (as a control group). Learning tended to be seen in terms of ‘providers’ rather than informal and incidental learning, as was the focus of the original research design.
- Stakeholders shared a strong belief that basic literacy skills were required before young people could participate in vocational/agricultural skill development programmes (a ‘literacy first’ approach).
Following the workshop, two contrasting field sites were selected – Siem Reap (a prosperous tourist area due to the temples at Angkor Wat, and near to Thailand, where people migrated for work) and Kampot (a hilly area with different kinds of agricultural activity, such as collecting forest products). The team of three researchers and two assistants spent about a week in a village in these two areas to conduct what we termed ‘ethnographic style’ research. The main research tool was life history interviews focusing on learning and agricultural skills (with 15 respondents in each area), as well as ethnographic observation (to find out about agricultural and rural livelihoods), focus group discussions with young people and interviews with providers (e.g. private companies, NGO and government skills training programmes). Considering that this was the researchers’ first experience of qualitative interviewing and observation, they collected some interesting data about young people’s values and practices (such as the emphasis that young women put on work as a social space – preferring to work in factories to the fields so that they could find ‘their future spouse’).
The research team valued this new research approach – saying how much they had learned just through talking to people. In such a short period of fieldwork, they found it difficult to get young people together for focus group discussions, commenting that the youth were usually too ‘hung over’! The team also welcomed the opportunity to bring themselves into the research process rather than having to downplay their influence/role. However their biggest challenge as novice ethnographers was how to analyse the huge amount of data. Being more used to statistical evidence, they started by analysing the life history data quantitatively, developing tables of ‘facts’ from the various accounts and analysing livelihood strategies according to push/pull factors. They tended to emphasise findings from their data that were supported by the literature review – for instance the notion of a legacy of ‘secrecy’ from the Khmer Rouge period, which influenced how far people today shared agricultural skills and knowledge.
As research advisor, I suggested going back to the raw data to conduct thematic analysis – and we did this together (by email or Skype) with a sample of life history interviews that they had translated into English. The team then began to incorporate extracts from the data into their report and to move away from generalising statistically about the most common or least common responses. However, when they presented the draft report to the advisory group in Cambodia, they were told it was inappropriate to include quotations from respondents. As this report was to be shared with researchers in the other country teams, I also helped the team to see what needed to be included for an outsider audience – taken-for-granted knowledge about institutions or practices in the two villages. However, as an outsider I was not so aware as to what would be considered sensitive in the Cambodian context – such as references to people having learned agricultural skills under the Khmer Rouge regime, which might be interpreted as a positive endorsement.
A second workshop was held with stakeholders to share the research findings and finalise policy recommendations. Much of this discussion ended up around our differing assumptions about how or whether qualitative research should inform policy on adult learning. Ethnography in particular was seen as ‘problematic’ because of focusing on a small sample of the population and participants stressed the need for ‘the bigger picture’. There were repeated objections to the small sample size not being ‘representative’ and the research team were asked about the actual number of people that they had spoken to. This was not an easy question as they had talked to many people informally in addition to the formal interviews and research activities. Several people from the partner agricultural institution argued for ethnography on the grounds of budget, that it had not been possible to conduct research on a larger scale with the resources available. Much of the discussion around validity was in terms of what participants knew or did not believe to be ‘true’ in the findings reported (particularly regarding a lack of data from both field sites on government extension workers as the team said they had not found anyone who had met one). Some participants suggested that a quantitative survey was needed to support the qualitative study, before the findings could be used in policy recommendations. A major outcome from the workshop was the decision to amend the title of the report to include ‘A case study of two villages …’ to reflect the research approach and perceived limitations. An interesting dimension of the discussions for me, as international advisor, was the notion of ethnography as a ‘foreign’ and inappropriate methodology for Cambodia. Some participants also suggested that these very local small-scale findings would gain importance and validity by being included in the international level report (which would compare the Cambodian study with the other country studies).
The experience of conducting this study intended to inform educational and agricultural policy in Cambodia has made me reflect more generally on the relationship between ethnographic, or even qualitative, research and policy. I went back to an ODI (2002) briefing paper on ‘Bridging Research and Policy’ (by de Vibe, Hovland and Young, www.odi.org.uk/rapid/projects/r0040a/Biblio_Summary.html), which begins:
‘Traditionally, the link between research and policy has been viewed as a linear process, whereby a set of research findings is shifted from the ‘research sphere’ over to the ‘policy sphere’, and then has some impact on policy-makers’ decisions. At least three of the assumptions underpinning this traditional view are now being questioned. First, the assumption that research influences policy in a one-way process (the linear model); second, the assumption that there is a clear divide between researchers and policy makers (the two communities model); and third, the assumption that the production of knowledge is confined to a set of specific findings (the positivist model).’
Looking at the Cambodian research study in relation to these assumptions, I realised that the situation was particularly complex as the research had originally been influenced by international policy (being commissioned by two international development agencies), as well as aiming to inform national policy in Cambodia. So it was not so much a two-directional linear process but multiple layers of policy institutions (in Cambodia and internationally) all with different assumptions about research aims and outcomes. All the researchers (including myself) could also be seen to be policy actors too (the ‘two communities model’). Ethnographic research brought a new dimension of complexity as it was not clear what ‘the set of findings’ (from ODI extract) were: the quotations from the data were disputed as not being ‘representative’ of other communities in Cambodia. The team eventually presented four ‘key’ findings and three policy recommendations in bullet point form at the final workshop (in line with the ‘positivist model’ mentioned above).
However, from my perspective, the most useful contribution of the study had been less tangible: the research evidence broadened the discussion of ‘providers’ and ‘programmes’ that most participants had started with (i.e. educational institutions) – to include consideration of how private companies (for instance, contract pig rearing) and individuals (e.g. fertiliser sellers) supported learning in the community.
We ended this BALID session with a wider discussion of our experiences of linking research and policy in the literacy and adult learning field. What emerged in particular was the ‘political’ nature of such interventions, which is often not recognised as such.