7 Reviews. This work was partially funded by the UK National Institute for Health Research (NIHR) School for Public Health Research, the NIHR Collaboration for Leadership in Applied Health Research and Care of the South West Peninsula (PenCLAHRC) and by Public Health England. ‘Well-being evaluation tools: a research and development project’, Centers for Disease Control and Prevention (1999). Qualitative evaluation and research methods (pp. Evaluation methods Choosing methods for evaluation. Alternatively, you may have obtained findings from a questionnaire and wish to discuss these with a focus group to get a deeper understanding of what the results mean. Understanding the needs of the community is an important process, and with emic data we can construct projects and activities and set indicators that are culturally appropriate. ‘The program manager’s guide to evaluation’, WK Kellogg Foundation (2004). It is most useful to ask open-ended questions to encourage participants to speak, and it is usual to start with some generic questions at the beginning to relax the participants and build a rapport. Interviews can be conducted by telephone, through an online video resource or in person. Different methods are suitable for answering different types of evaluation questions. Open-ended questions in structured questionnaires Semi-structured and in- depth interviews Focus groups Document analysis Workshops - table-based/group exercises Go-along interviews - shadowing Photo-elicitation - photovoice Mental models- mind maps Participant diaries, logbooks, recordings Observation Qualitative research often aims to describe and explain rather than predict or identify cause and effect. We also need to consider Bamberger, Rugh, and Mabry's (2012) constraints of time, budget, data, and politics as we plan our qualitative research and evaluations. In semi-structured interviews, questions are usually relatively ‘open’; that is, they encourage a detailed answer from the interviewee rather than a ‘yes’ or ‘no’ response. qualitative evaluation methodscan be incorporated into the range of options’ available to evaluators and their clients, and will offer some com-parisons between the two approaches. As above, observation and participant observation allow us to observe and understand change that has or has not taken place in society as a result of our program. Qualitative research is usually ‘bottom up’ rather than ‘top down’. This site is intended to enhance your use of Qualitative Research & Evaluation Methods, Fourth Edition, by Michael Quinn Patton. The content of a guide would be decided based on the evaluation question(s) and objectives as well as after discussions with any oversight or steering group and/or with the type of people you will be interviewing. Analyses of qualitative data include examining, comparing and contrasting, and interpreting patterns. Qualitative Research & Evaluation Methods: Integrating Theory and Practice [Patton, Michael Quinn] on Amazon.com. Drawing on more than 40 years of experience conducting applied social science research and program evaluation, author Michael Quinn Patton has crafted the most comprehensive and systematic book on qualitative research and evaluation methods, inquiry frameworks, and analysis options available today. Without that baseline data, we might not be in a position to show our project’s impact, so we need to think about collecting baseline data during the planning or at the start of the implementation phase if we want to show this later on. You might use interviews, focus groups, case studies of selected cases, etc. In these cases, qualitative methods are usually more suitable. ‘WHO evaluation practice handbook’. -exclude any questions that are not necessary for the study, exclude any questions to which the respondent might not know the answer, include an ‘other’ option when necessary and provide space for respondents to write/add their own answers, keep the questionnaire as short as possible, leave any sensitive or confidential questions to the end to avoid putting people off at an early stage, check whether the questionnaire is reliable and valid for the sample to be studied, plan how the data generated by the questionnaire will be captured and stored (for example, entered into a spreadsheet), summarised (for example, to produce overall scores) and analysed. Ian Shaw shows how evaluation practice can utilize qualitative Lincoln and Guba (1985) defined a document as "any written or recorded m… If we want to show the impact of our project, or the changes in people’s attitudes, behaviors, or competencies, then we may need a baseline to compare to. ‘The Magenta Book: guidance for evaluation’, Department for International Development (2013). This kind of data can be important for our project, depending on its scope, objectives, and activities. They are also unsuitable for understanding aspects such as the meaning of participant experiences or for exploring issues in-depth. 2 issue may not be readily apparent. Written by Sarah Denford, Jane Smith, Sarah Morgan Trimmer, Charles Abraham, and Krystal Warmoth, Facilitators must ensure that one person does not dominate the discussion and that everyone gets the opportunity to have their views heard. The results of qualitative methods are more descriptive and the inferences can … His wife is elderly and 2 of his 4 children have moved away from the area. This publication is available at https://www.gov.uk/government/publications/evaluation-in-health-and-well-being-overview/evaluation-methods. There are computer packages that can help with analysis of qualitative data. date published. It is used to describe general trends or distributions, or to test a theory or hypothesis about relationships between things, including cause and effect. *FREE* shipping on qualifying offers. If a topic is sensitive and people might be uncomfortable talking about their own experiences, they could be asked to give advice to someone else in a similar situation or to imagine a person and what their responses would be. Don’t worry we won’t send you spam or share your email address with anyone. Qualitative methods are also effective in identifying intangible factors, such as social norms, socioeconomic status, gender roles, ethnicity, and religion, whose role in the research Overview 1 OVERVIEW. It can be conducted in different ways but usually proceeds in the following stages. combining the theoretical with the practical, this book examines both how and why to conduct evaluations, reviews evaluation research history, and presents a qualitative research model. Next, segments of data are labelled through ‘coding’, and then higher level, more interpretive themes are developed from these codes in one or more further stages. This has led to an increased interest in mixed methods evaluations. Quantitative and qualitative methods generate different types of data. ‘WK Kellogg Foundation evaluation handbook’, World Health Organisation (2013). Qualitative Research & Evaluation Methods. Support for statistical training or knowledge of the relevant software packages will need be to identified when planning an evaluation. We should pay special attention to ethical considerations, as qualitative researchers tend to spend a lot of time with informants, gathering sensitive data in the process. Checkliste Qualitative Evaluation – Seite 3 von 20 Seiten. See the section on outcome evaluation. ‘Learning and growing through evaluation: state asthma program evaluation guide’, Centers for Disease Control and Prevention (2011). You may wish to use qualitative research to find out how people feel and follow this with quantitative research to find out if these feelings are commonly experienced. Depending on our project, we might use a census table or a structured interview schedule to collect baseline data during the planning phase of a project. The opening questions also give insight into how forthcoming participants are likely to be. Andrew, S and Halcomb, E J (2009). 67081. author(s) m q patton. 3 likes. This question is for testing whether you are a human visitor and to prevent automated spam submissions. This can be more helpful to us than quantitative data, such as “number of people trained.” Indeed, one of the most common uses of qualitative data is to help explain or add perspective to quantitative data. Geschichten der Teilnehmenden aufgezeichnet und kommuniziert werden. ‘Planning evaluability assessments: a synthesis of the literature with recommendations’, Medical Research Council Guidance (2008). Looking for a practical introduction to the strategies and methods of qualitative evaluation? For example, if you want to explore pregnant mothers’ understanding and knowledge of antenatal care and their compliance with recommendations, you could start by conducting a focus group of mothers from different areas and backgrounds. Advantages of administering surveys in person: Limitations of administering surveys in person: Advantages of administering surveys online: Limitations of administering surveys online: Questionnaires are usually accompanied by some demographic questions about age, gender, employment, place of residence and so on. Using Qualitative Methods in your Evaluation – Webinar It is important to determine if the required data are already available and accessible, or whether the data will need to be collected as this will have time and resource implications. In line with research ethics, it is good practice to discuss confidentiality issues and ask interviewees to read and sign a consent form to confirm that they are happy with the arrangements. A common and quick way of collecting quantitative data from large samples of people is through surveys and questionnaires. Interviews allow topics to be explored in depth. ‘Developing and evaluating complex interventions: new guidance’, Medical Research Council Guidance (2015). ‘Process evaluation of complex interventions’, Medical Research Council Guidance (2010). It commonly investigates how people make sense of the world and experience events. Qualitative Methods in Evaluation. As we craft our qualitative or mixed method evaluation designs, we should consider what qualitative methods we would use, and what kind of data those methods would give us. Beverly Hills, CA: Sage. This will be influenced by the questions being addressed and the most suitable study design for answering these questions within the resources available. We use observation to see what is happening in our project, who is participating, and who is not participating. Chapters address qualitative interviewing, fieldwork strategies, and observational methods. When used along with quantitative methods, qualitative 169-186). It takes time to set up a focus group and to ensure that everyone attends. Impact evaluation methods: Qualitative Methods 1. When planning an evaluation, you should identify what data is needed, from whom and when. Framework is one of the qualitative data analysis methods that are gaining popularity across multiple areas, including health, social policy, and psychology. At this stage, key informant interviews might give us some possible project parameters, and this might be of particular importance if there are gatekeepers in the community who could help or hinder a project and its activities. They are usually based on a topic guide. I am providing my consent by leaving the opt-in checked. To view this licence, visit nationalarchives.gov.uk/doc/open-government-licence/version/3 or write to the Information Policy Team, The National Archives, Kew, London TW9 4DU, or email: psi@nationalarchives.gov.uk. ‘Handbook on planning, monitoring and evaluating for development results’, UN Women (2013). Qualitative Research & Evaluation Methods: Integrating Theory and Practice However, evaluation often relies on self-reported or pre-existing data, and various tools are available to support collection of this type of data. More complex analyses (for example, comparing groups, looking at difference between subgroups, or controlling for extraneous factors) will require knowledge of statistical analyses and associated software packages (for example, SPSS, Stata, SAS). This analysis can usually be managed in Excel. An example of a quantitative evaluation would be "This year our company had a total of 12 clients and completed 36 different projects for a total of three projects per client." What about those of his wife and children?’. Where would you suggest that they start?’, Alternatively, if you wanted to gain insight into the concerns of someone with terminal illness regarding hospices, you might say: ‘Mr McKenzie is moving to a hospice 20 miles from his home after a long illness. Sometimes, simple qualitative research methods are combined with quantitative research, for example, via the inclusion of a comments section within a questionnaire. Generally, the interview topic guide/schedule would contain prompts in case a topic doesn’t naturally arise. These programmes allow large amounts of qualitative data to be stored, analysed and summarised in a systematic way. Please note that all the materials on this site are especially geared toward maximizing your understanding of the material. qualitative methods has been made: the importance of engaging an evaluator with specific experience in using qualitative methods and a consideration of the time and budget implications of the decision to use these approaches. Depending on our evaluation statement of work or performance monitoring plan, we use different methods on particular occasions to elicit certain kinds of data. To learn more about American University’s online MS in Measurement & Evaluation or Graduate Certificate in Project Monitoring & Evaluation, request more information or call us toll free at 855-725-7614. After developing the method through a three-year DFID/ESRC funded research project, staff from CDS set up Bath Social Development Research (BSDR) to continue the development and dissemination of the QuIP. Qualitative research encompasses a variety of methods, but is often defined in terms of using words and text in data collection and analysis, rather than using numerical measurement and quantification (Bryman, 2016). 1980 annotation. Participant observation might give us more of an emic understanding of what is happening, especially if we are allowed into the backstage where the observer effect is no longer as evident. As we craft our qualitative or mixed method evaluation designs, we should consider what qualitative methods we would use, and what kind of data those methods would give us. They are called guides because they are meant to be adapted to the needs and experiences of the interviewee or focus group and used in a flexible way. However, evaluation studies do not always make claims for the population at large, and may only provide more specific information about the effects of an intervention in a specific group of people or context. Outcomes and ImpactShowing causation between the baseline and outcome data is something to consider in the design of an impact evaluation. Advantages of focus groups are that they: Topic guides include questions and prompts used when conducting a focus group or interview. About the Author:Dr. Beverly Peters has more than twenty years of experience teaching, conducting qualitative research, and managing community development, microcredit, infrastructure, and democratization projects in several countries in Africa. The most common of these are NVivo, Atlas and MAXQDA. Qualitative methods that monitor progress are particularly important during the implementation phase of a project. If the interviews are lengthy, it is generally a good idea not to arrange more than 2 in a day. There are different types of qualitative research methods like an in-depth interview, focus groups, ethnographic research, content analysis, case study research that are usually used. This will help with adapting later topics and deciding in which order to ask questions. Analysis will likely include the identification of themes, coding, clustering similar data, and reducing data to meaningful and important … Use of these packages requires training and a good understanding of qualitative methods. It is useful to record a focus group or at least have a second person taking notes of the discussion as it is difficult to do this alongside facilitating the group. A wide variety of research methods and data collection tools are available for use in evaluation: qualitative and quantitative. Qualitative methods are a good starting point when you begin your assessment. Evaluation, methods, qualitative Introduction Extension professionals may not feel they have the time, resources, or expertise for conducting advanced statistical analyses (Higginbotham, Henderson, and Adler-Baeder 2007). Guest, G. and Namely, E E (2014). Muchos ejemplos de oraciones traducidas contienen “qualitative evaluation methods” – Diccionario español-inglés y buscador de traducciones en español. Scenarios can also be used in interviews or focus groups to depersonalise an issue. For example, Tearfund h… Our designs are only going to take us so far, and that we as evaluators need training and expertise to use qualitative methods in culturally appropriate ways. Observation might give us a picture of what is happening in the project setting. Using qualitative data to monitor projects gives us insight into a project’s activities as they are being implemented. Quantitative methods are used to investigate things that can be measured or quantified to generate numerical data. Psychology Applied to Health, University of Exeter Medical School. For evaluations, specific questions are then based on what the intervention to be evaluated aims to achieve. Participatory tools like seasonal calendars might help us to understand the emic needs of the community, and the different local events or micropolitics issues that might impact project implementation and beneficiary access. Interviews can be tiring for interviewers, as they have to listen carefully to determine which topic to go to next and to ensure understanding. Message and data rates may apply. ‘Introduction to evaluation for local authorities: a brief guide to evaluating food enforcement projects and innovative approaches’, HM Treasury (2011). Collecting baseline data may be important if we want to be able to show outcomes or conduct an impact assessment after the conclusion of our project. Pro-grammgeschichte erzählen, indem die . Complete transcription of recordings is very time-consuming and partial transcriptions or summaries are sometimes used, depending on the type of evaluation. The QuIP was developed by researchers at the University of Bath’s Centre for Development Studies (CDS). We’ll send you a link to a feedback form. If possible, it is better to use or adapt a questionnaire that has been used for similar purposes or in a similar sample and, ideally, has been demonstrated to be reliable and valid – see outcome evaluation section. To begin with, have a strong rationale for why you have chosen a qualitative approach to Qualitative and quantitative methods can usefully be combined in ‘mixed methods’ research to provide more comprehensive information for an evaluation (Andrew and Halcomb, 2009). Michael Quinn Patton. Don’t include personal or financial information like your National Insurance number or credit card details. Evaluators have a large toolkit of qualitative methods, and we use each of these methods under different circumstances to gather different kinds of data. Following qualitative methods are generally used in job evaluation: Qualitative Research & Evaluation Methods: Integrating Theory and Practice Drawing on more than 40 years of experience conducting applied social science research and program evaluation, author Michael Quinn Patton has crafted the most comprehensive and systematic book on qualitative research and evaluation methods, inquiry frameworks, and analysis options available today. The advantages and limitations of some of the more common methods are outlined below. A qualitative needs assessment might give us more of a disaggregated perspective of literacy or health issues that takes into account emic perspectives. Keep up with the latest happenings, news and accomplishments at American University. ‘Using natural experiments to evaluate population health interventions’, National Science Foundation (2002). Key informant interviews and focus groups give us insight into the change, or lack thereof. ‘Guide to analyzing the cost-effectiveness of community public health prevention approaches’, US Department of Health and Human Services (2010). A wide variety of research methods and data collection tools are available for use in... Qualitative research. ‘Doing quantitative research in the social sciences: an integrated approach to research design, measurement and statistics’ Sage Publishing. There are many more qualitative research methods such as naturalistic or ethnographic observations, video recordings of behaviour, open-ended questions on questionnaires, and analysis of text (for example, in advertising or educational materials) or photographs. Qualitative data answer such questions as “What is the value added?”, “Who was responsible?”, and “When did something happen?’’ Qualitative data are collected through direct or participant observation, interviews, focus groups, and case studies and from written documents. In this way, qualitative analysis goes beyond merely summarising data, to drawing out underlying themes that provide in-depth insight into an issue. The last section of this chapter outlines less common but, nonetheless, potentially useful qualitative methods for project evaluation. For example, if data from a survey or questionnaire shows that one hospital has more infections than others, you may wish to follow this up by interviewing staff or visiting the hospital and undertaking observations to get a more in-depth understanding of why that might be and how it could be resolved. A good general reference on issues of qualitative methods, and strategies for analysis and interpretation of qualitative data. It will take only 2 minutes to fill in. The person who facilitates the focus group should be experienced, and there is training available to support development of relevant skills. ‘The 2002 user-friendly handbook for project evaluation’, Avon Primary Care Research Collaborative website, NHS Health Scotland (2003). These methods include document studies, key informants, alternative (authentic) assessment, and case studies. Advantages of telephone/online video interviews are: Limitations of telephone/online video interviews are: Advantages of face-to-face interviews are: Limitations of face-to-face video interviews are: In advance of the interview, researchers should decide if: the interview should be recorded, how it will be transcribed and analysed and how excerpts will be presented in any write up. It is a good idea to test questions with a pilot group to ensure they are understandable and appropriate for the type of participants in the sample. As with interviews, issues of consent and confidentiality should be discussed with participants. We can use qualitative data to tweak or change direction of our programming, especially if we are not hitting our intended objectives or making progress towards our indicators. School of Professional and Extended Studies. Each of the research approaches involve using one or more data collection methods.These are some of the most common qualitative methods: Observations: recording what you have seen, heard, or encountered in detailed field notes. This would allow participants to discuss a topic in a more general way without feeling that the questions are intrusive or require sharing too much personal information. The participants might be strangers or people who meet together already. The facilitator should not have a particular vested interest in the findings, so they do not lead the discussion or impose their own views. Michael Bamberger, Jim Rugh, and Linda Mabry, Master of Public Administration and Policy, MS in Counter-Terrorism and Homeland Security, MA in Economics, Applied Economics Specialization, MS in Human Resource Analytics and Management, Qualitative Methods in Monitoring and Evaluation: Thoughts Considering the Project Cycle, Sylvia Burwell Inaugurated as American University’s First Female President, School of Public Affairs Advances to #18 for US News & World Reports’ 2019 Top Public Affairs Schools. Projects evaluated range from rural livelihoods interventions to microfinance initiatives. ImplementationWe incorporate qualitative data into our monitoring efforts and formative evaluations so that we can improve project activities. She also conceptualized and developed the proposal for Darfur Peace and Development Organization’s women’s crisis center, a center that provides physical and economic assistance to women survivors of violence in the IDP camps in Darfur. ‘Monitoring and evaluation: some tools, methods and approaches’, Treasury Board of Canada, Secretariat (nd). qualitative evaluation methods. Quantitative observations are made using scientific tools and measurements. Questionnaires are less useful when you wish to conduct more exploratory research (for example, where you might not have enough understanding of something to know what questions to include). When selecting or designing a questionnaire, these issues may be considered: It is not always necessary to collect new data as data relevant to an evaluation may already exist; for example, data that’s been collected for administrative purposes or added to patient records. It can be used to enrich our understanding of an issue or to improve services (Willig, 2008). Qualitative methods include three kinds of data collection: (1) in-depth, open-ended We should consider what qualitative methods we would use, and what kind of data those methods would give us. Where we have identified any third party copyright information you will need to obtain permission from the copyright holders concerned. See the section on evaluation planning. Focus groups and participatory tools are also important for us so that we can get a wider perspective of project activities and outputs. Qualitative inquiry typically focuses in depth on Qualitative research is not concerned with representing a population, in the way that quantitative research is, but with investigating topics in depth (Willig, 2008). Qualitative evaluations use qualitative and naturalistic methods, sometimes alone, but often in combination with quantitative data. Quantitative research methods usually aim to tell you something about a population (for example, smokers or older adults) based on a representative sample selected for a study. Leading questions or otherwise indicating to an interviewee that a certain type of answer is the ‘right’ one should be avoided as this would bias the data. Qualitative Evaluation Methods . Some people speak readily and at length whereas others can be more reluctant to express their views and need prompts to explore issues in depth. For example, if you are interested in the views of men with an alcohol problem, you might try to get permission to conduct a focus group before or after an alcohol support group session. Concluding ThoughtsWhile our evaluation designs need to be solid, we also need to have knowledge to implement the designs within other particular historical, cultural, and linguistic settings. Quantitative research aims to be objective and usually collects data in a pre-defined, structured way (Black, 1999). For example, if you wanted to interview someone about addictions and their views on the services available, you could ask: ‘If a friend or neighbour wished to access addiction services in this area what advice would you give them? ‘LEAP for health: learning, evaluation and planning’, The World Bank (2004). PlanningAs we are planning our project, if we are lucky, a donor will give us money to carry out a needs assessment. The benefits to using qualitative methods are that they: The limitations of qualitative methods are that: Interviews are frequently used to collect data in qualitative research. ‘Norms for evaluation in the UN systems’, United Nations Development Programme (2009). Newbury Park, CA: Sage. Lini Wollenberg, CGIAR Research Program on Climate Change, Agriculture and Food Security and Gund Institute, University of Vermont LORTA Program Inception Workshop, 24-26 July 2018 Impact Evaluation Methods: Qualitative Methods Big Lottery Fund (2008). We use participant observation and key informant interviews to understand what is happening in our project as it is being implemented. Learn more about Dr. Peters. ‘Framework for program evaluation in public health’, Centers for Disease Control and Prevention (2011.) In general, it is best not to include people with more power than others in the same group, such as teachers and pupils or junior and senior staff, as this can stop participants being open with their views or opinions. ‘Implementing evaluations: learning and growing through evaluation module 2’, Food Standards Agency (2015). If so can you say why?’) Focus groups bring together a small number of selected participants to discuss a specific topic and are usually facilitated by someone who does not know the group.