Nominal group technique and focus groups

 The use of the nominal group technique as an evaluative tool in

medical undergraduate education

G Lloyd-Jones, S Fowell & J G Bligh

(Lloyd-Jones, Fowell and Bligh, 1999) The nominal group technique (NGT) has been used for various purposes, including course evaluation, and appears well suited to this application. It combines qualitative and quantitative components in a structured interaction, which minimizes the influences of the researcher, and of group dynamics. (p.8).

 

Standardized questionnaires allow comparison between different settings and over time but may lack relevance to novel ventures. On the other hand, more exploratory methods, such as interviews and focus groups, which tap the student perspective, are costly in terms of organization, time and analytical skills (p.8).

 

NGT is a structured group activity designed to elicit the views of group members on a given topic (1). Initially promoted as a procedure for facilitating group decision making (2), it has been used in curriculum development (3.4), Used in curriculum evaluation its purpose is to explore the student or consumer perspective in a way which combines qualitative and quantitative elements. This is achieved by allowing the students to create and prioritize items with minimal influence from the researcher.

 

·        1, Delbecq AL, Van de Ven AH, Gustafson DH. Group techniques for program planning: a guide to nominal and Delphi processes. Glenview, Illinois: Scott, Foresman & Company; 1975.

·        2, Delbecq AL, Van de Ven AH. A group process model for problem identi®cation and program planning. J Appl Behav Sci 1971; 7940:467±93.

·        3, Hegarty EH. The problem identi®cation phase of curriculum deliberation. J Curr Stud 1977; 91:31±41. 4 O'Neil MJ, Jackson L. Nominal group technique: a process for initiating curriculum development in higher education. 1983; 8:129±38.

 

Each subgroup of five students elected a scribe whose job was to write on a flip chart all the items created in

the silent phase. During the round robin phase students read out one item in turn around the group but without

discussion or comment being allowed. These rules enable each student to contribute equally to list construction without the more dominating members inhibiting the others. (p.9).

 

Each subgroup discussed the individual listed items to ensure meanings were clear and shared by all. Although groups were not allowed to grade or discard items, they did edit overlapping and duplicated items. (p.9).

 

The NGT has proved a valuable evaluation tool in course development due to its ability to reflect the student perspective. (p.11)

 

Another advantage claimed for NGT over group interviews is the number of ideas generated in the early phases of the sequence. (p.11).

 

Lomax & McLeman suggest narrowing the focus of the stimulus question in order to limit item generation. (p.11).

 

By making the students responsible for the round robin and clarification phases we sought to reduce any influences our roles as teachers might have upon the students' willingness to disclose critical comments. (p.11).

 

The results of NGT are claimed to represent the consensus view of the group5 but this is not universally accepted7. The debate revolves around the degree to which group dynamics influence the NGT process essentially, whether the outcomes of a group in which interaction and group dynamics are controlled can be equated with those of a freely interacting one. (p.12).

 

In common with previous researchers, we endorse the inclusion of the nominal group technique as a useful tool in course evaluation. (p.12)

 

The Pitfalls and Promise of Focus Groups as a Data Collection Method

Jennifer Cyr (2016).

 

(Cyr, 2016)

 

focus groups generate data at three units of analysis, namely, the individual, the group, and the interaction. (p.231).

 

Focus groups were introduced to the social sciences in the early 1940s and have since grown in popularity (Liamputtong 2011:9). (Cyr, 2016; p.232).

 

Theoretically, focus groups may simultaneously produce data at the individual, group, and interactive levels (Kidd and Parshall 2000). (p.232).

 

There are few norms regarding how researchers present their focus group data in publishable research. (p.232).

 

In focus groups, a group of individuals is convened to discuss a set of questions centred on a particular topic or set of topics. The primary objective of focus groups is to generate conversations that uncover individual opinions regarding a particular issue. They also help to reveal group consensus, where it exists, on the issue at hand. The potential for data collection emerges from the ‘‘range of experiences and perspectives’’ that these focused conversations uncover (Morgan 1996:134). (p.233).

 

Given the conversational nature of the method, focus groups excel in revealing what participants think and why they think as they do (Bratton and Liatto-Katundu 1994:537). (p.234).

 

Focus groups enable researchers to collect multiple individual reactions simultaneously (Carey and Smith 1994:125). (p.234).

 

Indeed, unlike most data collection methods, focus groups involve group conversation and debate. They are inherently ‘‘social events’’ that yield data through the interaction of individuals (Smithson 2000:105). The synergistic (Stewart and Shamdasani 1990:16) nature of focus groups means that the data collected via the group are greater than the sum of its parts. The ‘‘rich experiential information’’ generated gives focus groups a comparative advantage over other data collection methods (Carey and Smith, 1994:124).3 (p.234)

 

Participants can work together to tackle complicated ideas and concepts. Researchers can therefore ascertain the level of agreement on those phenomena (Morgan and Kreuger 1993: 16–17) (p.235).

 

Researchers may use the conclusions from focus group conversations to assess how people ultimately understand and speak of specific phenomena. (p.235).

 

Specifically, researchers can glean important insight from the specific interactions that take place between participants as a conversation unfolds. (p.235).

Specific interactions or moments in an extended conversation may uncover surprising and unexpected reactions to a question. They may, therefore, spark new ideas about the phenomenon under consideration. Focus group interactions demonstrate how ideas and perspectives are engendered (Kitzinger 1995). Because of this, they are useful for exploratory work and hypothesis building (Fern 1982). (p.235).

 

Although the group unit is useful for assessing the measurement validity of a particular question under consideration, the interactive unit can spark an entirely new research question to investigate. (p.235).

 

Table 2 reveals that the use of focus groups as a pretest for other methods is very common. Many also used them to help construct an argument, either through the integration or (most commonly) through the triangulation of data. (p.238).

 

Found that focus groups were an efficient way to survey and elicit multiple reactions to a question at once. (p.241).

 

Focus groups are a relatively inexpensive and efficient method to ‘‘rapidly appraise’’ (Bratton and Liatto-Katundu, 1994:537) or assess what people think about a question. The cumulative effect is not much more than undertaking several interviews at once. (p.241).

 

focus group data were used as part of a triangulation strategy, that is, the findings from each focus group helped to corroborate or substantiate evidence collected via alternative methods (p.241).

 

This finding is not surprising. Focus group scholars recognize that the group dynamic inevitably shapes how individual participants react to questions (Farnsworth and Boon 2010:609). (p.243).

 

The final outcome or consensus that emerges on a given question may not accurately reflect every participant’s individual opinion perfectly. But pressures to conform permeate our social interactions constantly. Personal opinions are a product of the environment and are influenced by the individuals with whom we interact (Krueger 1994:10–11).15 (p.243).

 

The group’s influence on the individual implies that researchers who tap into the individual unit of analysis must consider the impact that the group has on the personal opinions that are expressed, as Paluck and Green (2009) 9p.243).

 

Because of this, researchers who tap into this level typically use focus groups to pretest other methodological instruments, especially survey questions (Fuller et al. 1993; O’Brien 1993). As Table 2 demonstrates, all four articles that most clearly utilize the group unit of analysis do so as pretests. Two articles (Gibson 2004; Sue and Telles 2007) use focus groups to pretest a survey instrument, one article (Ghazal Read and Oselin 2008) uses them to pretest interview questions, and the fourth article (Krysan et al. 2009) undertakes focus groups as a pretest for an experiment. (p.244).

 

In order to avoid systematic biases and the data distortions they produce, researchers use focus groups to ensure that the questions they ask measure what they seek to measure. (p.244).

 

For example, Focus groups were used in Ghazal Read and Oselin (2008:303) to pretest interview questions dealing with gender-role attitudes and behaviors and family dynamics. (p.244).

 

Prior research has shown that participants work through multiple and potentially conflicting views on a topic before arriving at a final, constructed opinion (Chong 1993; see also Barabas 2004). (p.245).

 

Beliefs and ideas regarding intersubjective phenomena are less easily explored or elicited in an individual interview (Savigny 2007; see also Krueger, 1994). Intersubjectivity reflects relationships of agreement or disagreement and understanding or misunderstanding among individuals (Gillespie and Cornish 2009:24). It follows that phenomena that are created and understood intersubjectively (e.g., race or class stereotypes) are better captured through social data generation processes. Focus groups, therefore, should be used when investigating these kinds of phenomena (see, e.g., Cyr 2014). (p.246). 

 

By tapping into the group unit of analysis of focus groups, researchers assess the extent to which agreement exists. In other words, they tap into the intersubjective nature of the phenomenon at hand (p.246).

 

Researchers look for group consensus to assess the validity of other data collection instruments. As we will see subsequently, focus group interactions can yield new insights that can be useful for exploratory research. (p.247).

 

The focus group conversation served as a source of new ideas that the author could then explore via other methods (p.247).

 

The interactions that unfold in the focus group setting can be a source of data that is unique to the individual or group unit of analysis. The interactive unit of analysis pays close attention to the back and forth that occurs between participants. This interaction allows answers to build and evolve (Stewart et al. 2009:594), uncovering nuances and complexities that may not otherwise be anticipated. 9p.248).

 

Focus group interactions can engender collective responses on a particular issue, as participants dialogue and debate about different perspectives (Smithson, 2000:109). They reveal how social processes unfold and how opinions evolve (Kitzinger 1995:116). (p.248).

 

Instead, interactions may lead to the formulation of new hypotheses, ‘‘fresh insights’’ that can later be tested via other methods (Bratton and Liatto-Katundu, 1994:538). (p.248).

 

Kidd and Parshall (2000), for example, used focus group interaction to develop a new workplace injury prevention program that they hypothesized would better capture the ‘‘cognitive, schemata, folk models, and narrative patterns’’ that underpin workplace dynamics (Kidd and Parshall, 2000:297). (p.248).

 

The author recognizes that focus groups are particularly useful for exploratory work, because they ‘‘allow deliberation among participants’’ (Prabhakar 2012:81). (p.249).

 

Data generated at the individual level enabled researchers to quickly appraise multiple opinions or viewpoints that could then be triangulated or in some cases integrated with other evidence (p.250).

 

Group consensus served as a successful pretest for survey questions or other instruments. Finally, data generated through interactions produced unexpected findings that raised new research questions and hypotheses. (p.250).

 

The affinities discovered here corroborate the long-held convention that focus groups are best used in conjunction with other qualitative and quantitative methods (Morgan 1993). (p.250).

 

In conjunction with other methods, however, focus groups can reinforce alternative types of evidence and establish the measurement validity of indicators. (p.250).

 

Therefore, researchers who measure socially produced and reproduced phenomena should seriously consider undertaking focus groups as part of their research design (Cyr 2014). (p.250).

 

Focus Group meets Nominal Group Technique: an

effective combination for student evaluation?

Tünde Varga-Atkins, Jaye McIsaac & Ian Willis

 

(Varga-Atkins, McIsaac and Willis, 2017)

 

A wide range of methods exists to audit academic life in the UK. Questionnaire based surveys are a predominant way to gather student feedback about their learning experience for teaching enhancement and curriculum development (Richardson, 2005; Yorke, 2009).

Survey questions are pre-determined by the survey designer and can be administered to a large number of students (Watt, Simpson, McKillop, & Nunn, 2002; Wilson, 1997).

 

The authors report on an approach, that makes use of stages of the Nominal Group Technique in a Focus Group process. Our aim was to find out if this combined approach, the Nominal Focus Group,

has the potential to lead to enhanced student ownership and actionable outcomes for

curriculum development for members of programme teams. (p.289).

 

This approach involves working with, and drawing from the student experience, commonly referred to as the ‘student voice’ in higher education literature. (p.290).

 

In this study, the student voice is about using students for enhancing curriculum development process for staff (Campbell, 2007); therefore, we consider this to be a participatory approach. We would also argue that the consultative aspects of the method represent further steps towards empowerment as students are encouraged to make their own decisions, rather than just being used as feedback-givers. (p.290).

 

The Focus Group is a face-to-face small-group technique in order to explore participants’ perceptions of given topics. It utilises group interactions in a neutral environment (Wilson, 1997). Focus Groups can explore a range of questions. One advantage is that participants can contribute their own issues, not just those pre-determined by the facilitator. Focus Groups generate a large amount of qualitative data in relatively little time (Parker & Tritter, 2006). (p.290).

 

Focus Groups allow the facilitator to probe participants for more in-depth meanings (Wilson, 1997) and create more detailed accounts (Porter, 2013) (Figure 1). Critiques of Focus Groups have raised concerns such as participant bias (Breen, 2006), conformity (Lomax & McLeman, 1984) and the public/private voice or group effect (Wilson, 1997). (p.290).

 

The Nominal Group Technique is a structured face-to-face group method with the achieving group consensus (Varga-Atkins & McIsaac, 2011b). It has been used for a number of years for educational purposes, typically in a curriculum evaluation context (Abdullah & Islam, 2011; Crenshaw et al., 2011; Kristofco, Shewchuk, Casebeer, Bellande, & Bennett, 2005; Lloyds-Jones, Fowell, & Bligh, 1999; O’Neil & Jackson, 1983). (p.290).

 

Established originally by Delbecq and colleagues (1975), it comprises five stages which combine both individual (‘nominally a group’) and group work resulting in immediate action planning. Akin to Focus Groups, Nominal Group Technique sessions are conducted by a facilitator who poses one or two questions (p.290).

 

The first stage involves participants responding to the question in writing, typically on post-it notes, which are then put up for the whole group to see. Responses are read out and discussed. This stage is followed by participants working as a group to identify and collate items that are the same. The regrouped items are displayed so that the last, individual, ranking stage, can take place. Participants are asked to select their top five responses in order of importance, with associated points for each item. Once the facilitator calculates the points given for each response, a ranked list is drawn up with the group’s top five priorities (Figure 2). (p.290).

 

The structured process of the Nominal Group Technique aims for group consensus based on the sum of individual viewpoints, thus aiding shy or reticent individuals (a challenge in Focus Groups). (p.291).

 

One of the greatest values of Nominal Group Technique is the prioritised list produced during the session. As it is the participants who do the coding and categorising, the process promotes their ownership of the results (Wilson, 1997). It also reduces potential researcher bias in the data analysis. The ranked items also act as an actionable list for the commissioners of the research, programme teams in this case, making it conducive to action planning for curriculum development (Chapple & Murphy, 1996; Porter, 2013). (p.291).

 

One limitation is, however, that it can only really explore one or two pre-determined questions. (p.291).

 

The Nominal Group Technique was considered, as this structured process provides a ranked list that is effectively the key analytic stage that in turn is conducive to action planning. (p.292).

 

The programme teams had a range of topics that they needed exploring. These included perceptions or experiences with assessment and feedback; independent student study patterns; and identification with the chosen undergraduate programme. This range necessitated more in-depth discussions that would not have been possible with the Nominal Group Technique. The idea of combining the two methods as two stages in one session occurred in the hope that the benefits of each method would be usefully synergised. (p.292).

 

This combination, a Focus Group exploration followed by a second, Nominal Group Technique stage is what the authors are proposing as the ‘Nominal Focus Group’ approach (Figure 3). (p.292).

 

The first stage of the session consists of a Focus Group. The neutral facilitator guides participants through each question and explores their views. The aim of this stage was to allow for a more in-depth discussion of the topics outlined by the programme team. The second stage is a shortened Nominal Group Technique stage. Students are asked to answer one or two questions decided jointly by the commissioners of the research and the facilitators, with a view to producing a ranked prioritised list for action planning. (p.292).

 

Proposed areas of benefits were from data collection through to analysis, reporting and action planning. (p.292).

 

An important aspect of the work was to follow the full evaluation cycle from purpose, method, data collection, analysis and finally action (Brennan & Williams, 2004). (p.293)

 

Many students enjoyed the Nominal Focus Group evaluation sessions and found that they were better able to formulate their ideas with the help of others: (p.294).

 

Responses from the student surveys also showed that the neutral environment was a key: ‘it was amongst peers and there were no lecturers present so you felt you could be honest’. The group sessions also allowed participants to obtain a deeper appreciation of the issues explored (p.294).

 

The majority of student responses identified the open discussion as the element that most helped them in drawing out their experiences and opinions; a benefit attributed to Focus Groups (Eubanks & Abbott, 2003). (p.295).

 

The dynamic between the two stages, and so, the power of the combined technique was clearly evident to those students who said that both stages contributed to the quality of results: ‘The open discussion allowed many ideas to be put forward, whereas the second [Nominal Group Technique] part of the session allowed a summary of all of the views that were discussed’. In another case, the relationship between the two stages was that of (1) opinion formulation and (2) making opinions count within the group members: ‘The open discussion helped to get me thinking of my own experience of feedback, whilst writing my opinion on the post-it note [voting] helped get my opinion across’. Thus, the first stage enabled an open exploration, whilst the second ranking stage helped students to summarise, consolidate and prioritise what was important to them. (p.295).

 

The combined Nominal Focus Group brought the advantage of maintaining an equal voice of each participant through its final ranking stage. In a traditional Focus Group session, participant bias or group effect can occur. The combined approach helps reduce such potential effects with the presence of individual stages in the process. Traditional Focus Group participants rarely see the outcome of their session in the Focus Group. We observed another advantage, namely, that having this visible, tangible output of with the ranked items on a flipchart created a sense of student ownership. (p.295).

 

The combined Nominal Focus Group sessions were reported to have given staff a rich picture of student experiences, not usually possible with traditional module or programme evaluation surveys (p.296).

 

Reporting the analysis of the Focus Group transcript gave context and detail of the issues discussed. This paved the way for staff to accurately evaluate the ranked list of student recommendations. It was clear that having only either a report or transcript (the output of a Focus Group) or a ranked list (output of a Nominal Group Technique session) would have been of less value. The combined output helped staff see what they should focus on in making choices on curriculum enhancement. (p.296).

 

A major benefit was that in the combined approach, in the first Focus Group stage, it was possible to accommodate more than the usual 1–2 questions of the ‘original’ Nominal Group Technique. This was useful when staff had a range of questions to explore. In addition, the Focus Group stage allowed an in-depth exploration of all questions, a characteristic that is not usually possible with ‘just’ the Nominal Group Technique. (p.296-7).

 

However, there were time gains in the analysis stage since the voting stage in the Nominal Group Technique made it easier and quicker to analyse the data. The combined approach brought another benefit linked to the quality of data analysis. In the two-staged Nominal Focus Group, data analysis was carried out by participants during the session (the voting stage), which Massey (2011) calls the ‘first stage of analysis’. This process both enhanced student ownership and reduced any risk of interpretative bias of the researcher that can happen in a traditional Focus Group (Breen, 2006). (p.297).

 

Second, the literature suggests that during a Nominal Group Technique session, participants views are sought beforethese get discussed in a group setting (Lomax & McLeman, 1984; Williams, White, Klem, Wilson, & Bartholomew, 2006). Moving the ‘Nominal’ Group, stage after the open Focus Group discussion, one question that arises is whether the group discussion would bias individual suggestions; this is an area that would be worth further exploration. In our experience, this was not a problem as the discussion helped to formulate cohesive responses whilst also enabling individual views to emerge in the final stage. This is supported by other researchers who concluded that the wide range of disparate responses was difficult to pull together cohesively in a traditional Nominal Group Technique session (O’Neil & Jackson, 1983). (p.297).

 

One key purpose of this study was to examine what it was about the combined approach that could enhance the outputs of the process. It seemed clear from the student and programme team evaluations that the combined process produced data that was both ‘owned’ by students and was readily actionable by programme teams. (p.298).