International Journal for Quality in Health Care 14:139-147 (2002)
© 2002 International Society for Quality in Health Care
Developing standards of care: a tale of two panels
Vice President for Information Technology, Delmarva Foundation for Medical Care, Easton, MD, USA
Address reprint requests to David Boan, Vice President for Information Technology, Delmarva Foundation for Medical Care, Easton, MD 21601, USA. E-mail: dboan{at}dfmc.org
Objective. How will a group of experts convened to develop standards of care communicate in meaningfully different ways when interacting online in contrast to traditional face-to-face meetings. Furthermore, does online interaction facilitate discussion among participants from diverse cultural backgrounds and at what cost?
Design. A project to develop standards of care for deaf and hard of hearing adults used two panels of experts operating in two different formats: online and face-to-face. One panel interacted via an online system while the other met in a daylong face-to-face conference. As one component of the project, we tracked the interactions of these two groups in order to describe and contrast their group processes.
Study participants. The subjects were volunteer experts in hearing impairment and health care, from organizations across the United States. They were a geographically diverse group with widely varying communication needs. We applied two different systems for facilitating communication among culturally diverse participants, and assessed interaction and satisfaction.
Main outcome measures. Data were collected on the two groups on the pattern of interactions, satisfaction with the process, and satisfaction with outcomes.
Results. The results showed a high level of user satisfaction with both process and outcomes, and provide data for a description of the source of user satisfaction and the management of the groups.
Conclusion. Online interaction offers unique advantages but poses unique management requirements for success.
Keywords: collaboration, computer-mediated communication, expert panel
 |
Introduction |
|---|
 Top Introduction MethodsResultsDiscussionAcknowledgementsReferences |
|---|
The Internet enables collaboration among people who would normally be separated by distance, time, or communication barriers. The ease of use and proliferation of online systems makes it convenient for people to create such online collaborations, but so far, little is known about the special requirements for managing these. Are people more or less likely to express themselves online; do they interact with the same frequency; are online sessions subject to the same social pressures? Is managing an online panel essentially the same as a face-to-face panel or do they differ in important respects? A project to develop standards of care afforded an opportunity to examine these questions.
Background to the current project
In October of 1999 the Delmarva Foundation for Medical Care, a health care quality improvement organization in Maryland, and Gallaudet University (Washington, DC) received a special contract with the US Department of Health and Human Services, Health Care Financing Administration (HCFA, now Centers for Medicare and Medicaid Services) to propose standards for health care services to deaf and hard of hearing adults. The contract called for a panel of experts to propose a set of standards that would then be evaluated for possible inclusion in Federal regulations. The project team proposed to HCFA to use two expert panels. One would interact online via e-mail for two months, the second would meet face-to-face after the first group had generated a set of standards and would evaluate the proposed standards. The structure of the panels had special implications for this project. The number of people who are truly expert in health care to deaf and hard of hearing is limited. In order to get the greatest participation possible we needed to create a mechanism that would allow people to participate who have very busy schedules, who, in some cases, would not be able to take the time to travel to a traditional conference, and were widely separated geographically. In addition, we wanted representatives from a variety of consumer advocacy groups, which are also limited in number. In order to get the greatest participation we created two formats, one virtual and one face-to-face, and allowed people to select the group that best fitted their needs and interests.
Literature review
Several studies have compared face-to-face interactions with computer-based communications. Weisband and Atwater [1] compared face-to-face and electronic media and decision-making. They found that liking of other participants was a significant factor in face-to-face meetings, but not electronically. They concluded that in an electronic medium, people are more likely to base their judgements on actual contributions rather than on feelings of liking for another person. Given these differences between face-to-face versus electronic communication we were concerned that panel members interacting electronically would have more difficulty coming to consensus even if they were able to interact more freely. In other words, would an electronic forum produce results that were less supported by participants or that the participants felt were less representative of their own views compared with a face-to-face conference. If either were true it could be argued that the advantages of an electronic forum might be outweighed by the disadvantages. Using e-mail would create efficiency, but would the product be of the same quality? Michailidis and Rada [2] compared e-mail, fax, phone, post, and face-to-face interactions. They concluded that none of the methods was as robust as face-to-face contact, but that each had value. Alternatively, Weisband and Atwater wrote that an electronic forum would more closely duplicate a face-to-face conference if sufficient time were allowed to create trust [1]. We decided we would allow two months for the online group to interact. Straus [3] compared face-to-face and computer-mediated groups to see whether computer-mediated communication would have participation that is more equitable. She found that while there was more participation in computer-mediated communication, participation was still a function of extraversion and some individuals can still dominate, even in computer-mediated interaction. Hollingshead [4] studied face-to-face and computer-mediated interaction, to see how the media impacted on decision making. Like Straus, Hollingshead found that computer-mediated communication suppresses certain types of information exchange and the related perceived influence of some members, but was still subject to differential influence from some members. She suggests that the relationship between media and status is more complex than previously reported. Computer-mediated groups have also been reported to be less productive but to show greater equality of communication [5], take on the same social qualities of face-to-face groups [6], and experience the same status issues as face-to-face groups [7]. Kiesler and Sproull [8] point out that the complexity of computer-mediated interaction makes it essential to carry out field-based research.
Our approach, allowing more time for an online interaction, was seen as a better approximation of a real-world case. An advantage of online communication is that it is typically asynchronous, meaning it does not require the recipient to be available at the same moment (synchronous communication). This is convenient because recipients can respond when their schedules permit. It does mean a delay in the interaction, thus more time is required for an extensive interaction.
We were also interested in the relative cost of the two approaches. Previously, Bowers [9] reported that e-mail interaction among an international group of nurses was a very cost-effective and beneficial method for the participants. Likewise, Tse [10] compared standard mail and e-mail for response times and response quality for use with questionnaires and found that e-mail produced comparable reply quality at a much lower cost. These reports did not weigh the cost benefit of the approach, only the cost. In this project, the different processes do not allow for a cost-effectiveness analysis, but cost information is included for the benefit of the reader.
The panels
In addition to concerns about attracting the maximum participants from a small population, we were also concerned about the quality of interaction of individuals who would vary in age, education, professional stature, and communication skill. Many of the participants would themselves be deaf or hard of hearing. Such people are typically at a disadvantage in a group setting where the discussion moves quickly from person to person. Furthermore, deaf people exist in a unique culture that is different from the culture of the hearing world, and therefore have a unique context for communication [11, 12]. Deaf people do not see themselves as disabled, they see themselves as people with a unique language and culture, who do not hear. The language of this culture is American Sign Language (ASL), which, after years of struggle, is now recognized as a unique language with its own structure, syntax, and cultural context [13]. Thus, the participation of deaf individuals brings with it the dimension of culture and the related concerns over different social expectations and behaviors. We expected that an online forum would have advantages over face-to-face interaction when mixing people from diverse linguistic and cultural backgrounds. Furthermore, we expected the virtual panel to approximate the methods of the Delphi approach.
Delphi is an approach designed to overcome the limitations of group discussion. In Delphi, idea generation is isolated and anonymous, allowing participants to respond to and build upon the work of others without the intervening social-behavioral issues that contaminate face-to-face meetings [14]. Interaction between members is orchestrated by the panel director rather than directly between members. Clayton [15] concludes from his review that Delphi is the most effective technique for obtaining the most reliable consensus of a group of experts. This same technique has been used in quality improvement efforts, such as the Harvard Q-SPAN-CD [16]. A modified Delphi technique was used by Delmarva Foundation staff in conjunction with researching ambulatory-care quality-improvement indicators [17]. It also seemed the most desirable approach in this project, where we planned to recruit participants from a wide geographic area. Conducting the group by e-mail allowed the elimination of travel, which is a major barrier to participation. We believe that an e-mail-based system is effectively a specialized Delphi approach. It is semi-anonymous, managed, and interaction is asynchronous; thus it is similar to indirect communication. It is different in that the moderator does not edit and then distribute the e-mail. In our process the moderator initiated the discussion and then participated by offering clarification and direction to the group.
One of the project requirements was a process for validating the results of the panel. We elected to convene a second panel that would meet face-to-face and review the standards. Having a second panel, drawn from the same population, with a related task would allow us to collect data on the two processes. We could then contrast the virtual panel with the face-to-face panel as a means of clarifying the unique features of the online panel. However, it also posed a challenge in that we might have artificially created a dysfunctional process by having a face-to-face conference with people whose communication needs would dispose the panel toward negative results. This was addressed by holding the conference in a multimedia conference room at Gallaudet University that was designed for maximum efficient communication for people with diverse communication needs. Full multimedia services were arranged, including a team of sign-language interpreters, amplification systems, real-time transcription to a bank of monitors, and stenographic services. What we accomplished was a test of two systems using a population sensitized to problems in group interactions, and using the most sophisticated communication service available. In comments after the conference the participants repeatedly stated that the conference allowed for a level of interaction that far exceeded anything they had experienced before. Thus, we had two state-of-the-art systems to support interaction between two groups of people with diverse linguistic and cultural attributes. The multi-media and interpreting services would ameliorate the impact of hearing impairment but would not mitigate the factor of culture.
The first panel, dubbed the virtual panel, began in late January 2000. The group was asked a general question concerning what was needed to eliminate the health quality gap that exists between deaf and hard of hearing patients and the population in general. Twenty-six people from across the United States were registered to participate. The response from the panel was remarkable. The group generated 386 e-mail messages over eight weeks, plus another 57 generated by the moderator, making a total of 443 messages. As standards were proposed and discussed the moderator would clarify and solicit comments. The final week was devoted to listing and voting on the standards in preparation for presenting them to the face-to-face conference.
The virtual panel was also supported by a dedicated website so that it was not a completely e-mail-dependent system. The website served as an archive for materials, technical support, quizzes for panel members, and for public information regarding the project. In this respect the service was broadly online and not simply an e-mail system.
The face-to-face conference took place on March 31, 2000. Twenty participants were presented with the standards produced by the virtual panel and invited to add, delete, or edit as they chose. The panel was charged with validating whether the proposed standards were sufficient to address the disparity that is believed to exist in the health care quality received by deaf and hard of hearing adults compared with the general population. Like the online panel, this group was highly motivated and held strong opinions concerning the need for standards of care. This created a dilemma in that many people wished to address an issue, but time was limited. The morning of the conference was largely taken up by presenting and explaining the standards of care, the rationale for them, and the process used to create them. As the daylong conference progressed, we were faced with the need to structure the interaction and limit discussion. The members of the conference did not use any online features nor were any of the communications by electronic means.
Moderation
The author carried out the moderation of both panels. The same approach was used in both cases: introduce a task, followed by leaving the group relatively unstructured early in the discussion and free to take their own approach, then increasing use of clarification to focus the group during the second half of the process. Frequent statements were made regarding the purpose of the panel and reinforcement of the necessary outcomes. The moderator refrained from making evaluative statements on any of the messages or the standards. Most of the moderator comments were of a clarifying and supportive nature, aimed at sustaining the interaction and keeping the group on track. Messages that distracted from the focus were quickly directed to an off-line discussion. Many people were concerned about the potential for their suggestions to become regulations. This concern led to a tendency to want to carefully construct every sentence, in spite of repeated assurances from the moderator that this was an early phase in the process and that the goal was to define the domain of possible standards, not engineer specific regulations.
|
Methods |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
The panels
The project began with a review of the literature to identify existing standards of care and the people who were currently working in the field. The project team then generated a list of potential panel members, in part from those identified from the literature review. In addition, consumer and professional groups were contacted for recommendations. All members of the resulting list were contacted and invited to participate in the project. Interested individuals who knew of the project were able to apply to be on a panel, such as professional medical interpreters, directors of programs for the deaf and hard of hearing, and medical directors of schools for the deaf. Panel applicants were invited to state a preference for the online (virtual) panel or the face-to-face conference. More than half of the people contacted agreed to participate.
The panel composition emphasized a mix of medical and research professionals, allied health professionals, other professionals (interpreters, lawyers), and consumer representatives. We also sought a mix of deaf, hard of hearing, and hearing subjects. The project team reviewed the makeup of the two panels and made adjustments so that there would be a balance of professional service providers, educators, consumers, and others between the two panels. The two panels were quite similar with regards to age and years of education, as shown in Table 1.
|
The online system
The online system for this project was a project website integrated with a list server (a system for managing interaction among people on an e-mail mailing list). The project website (http://www.deafhoh-health.org) contained registration forms and materials, a quiz for assessing the basic knowledge of the subject, a system for collecting case stories, and support information for the panels. This proved useful in providing a way for people to access resources without resorting to e-mail, thus reducing the amount of administrative or support e-mail necessary. It also provided a way to verify that people had in fact viewed the introductory materials by requiring panel members to register on the site. The system then recorded when each person accessed the materials. The list server was a Lyris server, allowing for both e-mail and web access. All e-mails were moderated, but prior approval was not required for release. This improved the speed of interaction in that a message was distributed to all members within seconds of being received by the server. Archiving of all e-mail made it possible for panel members to visit the site and view prior messages. The performance of the system was fast and error free during the project, except for a 24-hour system problem during the last week of the interaction. Generally, the participants found the system to be responsive and complimented the performance, indicating that system problems, at least on the server end, did not impair interaction.
The conference
The conference was held in the boardroom of Gallaudet University. This allowed 20 participants to sit with a view of each other plus a monitor with a transcript of the session. Three interpreters rotated the interpreting duties and a stenographer sat to one side and recorded every interaction. Participants were not given the standards in advance. The session began with introductions, a description of the task and rules for the day, and then a review of the proposed standards. Following the review of the standards, the panel began discussing standards proposed by the virtual panel. Their charge was to determine what changes or additions to the proposed standards were needed to eliminate the gap in quality of care for deaf and hard of hearing people and the general population. Discussion went for seven hours after the introductions and review. Afterwards participants were encouraged to submit in writing any final comments or recommendations.
Data collection
Data were collected on the interaction within each panel, the satisfaction with the panel process, and the satisfaction with the panel outcomes. For interaction during the conference, a stenographer recorded all discussions. This recording was done on a stenographic recorder connected to a laptop computer. The transcript was verified by the stenographer and then provided to the team on computer disc. The resulting file could then be analyzed electronically. On the virtual panel, all e-mail was archived in its entirety to the discussion server and stamped by author, date, and time. The entire archive was then transferred to an analytic workstation where the messages could be analyzed both individually and as a batch.
Satisfaction was measured by a short survey developed by the team and given to both virtual panel members and conference attendees. Half of the questions addressed satisfaction with outcomes, and were the same for both groups. The other half of the items addressed process issues and were worded differently for each group. One set addressed online interaction and the other face-to-face discussion. In addition, there were several questions specific to each panel that assessed their perception of that specific format. The items in the surveys are shown in Tables 2 and 3.
|
|
Cost information was easily measured for the conference since it was a discrete event with a fixed budget. Cost information for the online panel required some estimation of the share of expense of services that shared systems.
|
Results |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
There are four sets of data for the project from the two panels: interaction, satisfaction with process, satisfaction with outcomes, and overall assessment. In addition, we asked for assessments of the specific format, such as satisfaction with interacting via e-mail, etc.
Panel interaction
Panel interaction was measured in two ways. For the virtual panel it was the frequency of e-mail messages contributed per subject. In the conference it was instances of verbal contributions, as recorded in the project transcript. A count of contributions was calculated for each subject, along with a percentage of total interaction. The percentage reflected the level of participation relative to other participants.
Virtual panel survey results
The mean item rating on the satisfaction survey from the virtual panel was calculated. In addition, we identified two subsets of items: a group of process-oriented items, and a group of outcome-oriented items. This distinction is seen as useful in distinguishing the satisfaction with the virtual panel as a means of communication, as opposed to the satisfaction with the outcomes of this particular panel. The means for these items were calculated as well and are shown in Table 4.
|
Cost
The operational expenses for the two approaches are very different. The face-to-face conference cost approximately $7500 in direct costs. This does not include the staff time, only meeting-room costs, interpreters, and other special services, recording, lunch, refreshments, etc. These costs may seem high and are due to the additional services that were required to address the communication needs of the participants.
The costs for the virtual panel were negligible, again excluding staff time. The Foundation used an existing web server and inexpensive software. The number of participants was small, thus no significant load was created for the server. If we allocated the cost of the server, technical support time, and software, for the two-month period we might use the following formulae: assuming a 36-month life for the server and software, and ten other projects running at the same time, we would have a cost of one-half of one percent of the lifetime cost of the server. Estimating lifetime cost at three times the purchase price gives us a server cost of $24,000, making the project costs $132. Alternatively, if the service had been outsourced to the list-server vendor the cost would have been approximately $100 per month.
|
Discussion |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Subject to the limitations that are discussed below, it is reasonable to conclude that an online interaction is an effective tool for group discussion and consensus building. The virtual panel shared many of the issues and features of the face-to-face group, but also differed in important respects. In this project, the same issues related to social influence, volume of contributions, and development of rapport among members were seen in both groups.
The conference was a very dynamic meeting with lively discussion. The subject, health care for deaf and hard of hearing adults, was one that every person at the meeting felt passionately about. Considerable time was spent allowing everyone to express their views, such that as the day went on the moderators increasingly structured interaction and worked to focus the group on the tasks. The group did not review all of the standards as originally proposed, but did identify issues that needed to be addressed. In the end many people thought the tendency to elaborate used up considerable time and that more time was needed to do a more thorough review, hence the low conference ratings for staying on task and for meeting goals. If the group had met for two days these ratings would most likely be higher, but this was cost prohibitive.
The online panel was not completely free from the social influence that is seen in face-to-face meetings. Having two months to interact allowed sufficient time for people to establish their identities and develop influence in the group. In fact, a few members who were more productive in generating messages tended to frustrate other members of the panel by generating messages that were seen as not entirely necessary. However, these frustrated members declined to make direct statements to these people, instead asking the moderator to control the more ‘expressive’ members. This is an interesting contrast to the face-to-face meetings where people would comment about themselves that they had taken enough time and would allow others to speak. One possibility is that in an online meeting not only do you lack the social cues available in personal meetings, but also it is more difficult to identify who is participating. A verbal statement is associated directly with a speaker, and frequent statements from the same person are easy to identify. In e-mail, it is more difficult to associate a message with a person because the only cue is the address. Determining who is or is not contributing becomes a more complex task.
It is interesting to note that the people on the virtual panel sought more social information from each other, and in many cases established relationships that persisted apart from the panel. The panel interactions began as anonymous exchanges (return addresses were masked), but from the start people identified themselves and offered personal biographies to the group. The masking of addresses was later dropped at the panels request and to increase ease of use (no need to constantly add identification). During the follow-up survey of participants, several people expressed the desire for a face-to-face meeting of the members so they could put a face to the name. While they did not think this would have made the panel more effective, they did think it would increase their enjoyment of participation.
The social aspects of the online panel point out that the online group was not as anonymous as might be expected, especially when allowing time for sufficient interactions so that personalities would begin to emerge. Running a group in a more anonymous fashion, where participants interact with the moderator and not each other might have provided different results.
Also of interest are the comments of many of the participants. Many panelists said they needed more time for the task, and would have been more satisfied with the results if more time had been available. Several people commented on their frustration with other conference members, who would talk off the subject and use up valuable time. There was only one comment from a virtual panel participant who felt that one virtual panel member generated a lot of unnecessary e-mail. While changing these factors would have influenced the results, both are realities of the modalities. People speak extemporaneously, get off the subject, and use up valuable time. When the subject is one people feel passionately about this is even more likely to happen. Having a time limit is a reality of a face-to-face conference just as much as having people generate inconsequential e-mail is a reality of an e-mail-based system. In this respect both panels represented the strengths and flaws of their respective systems well. An online panel that is operated over an extended time period has the option to increase the frequency and intensity of interactions if a time limit is looming. A face-to-face conference has far less flexibility in this regard.
Moderation
While the moderation of the two groups was carried out in a similar fashion, there were important differences. Both groups began with a general task, allowing time for the group to explore and respond to one another, followed by increasing structure and specific task deadlines. In the face-to-face group, it was apparent halfway through the project that the process used in the early part of the meeting would never review the guidelines even if substantially more time were afforded. Several conference members expressed concern over this and raised the issue to the group, seeking discussion on changing the approach to the task. In contrast, the virtual panel also had a rapidly approaching deadline, but never expressed concern over running short on time and failing to meet the target date and deliverable. The moderator had to initiate and enforce interim deadlines with the virtual group, while the face-to-face group raised this issue on their own. In other ways, the virtual panel deferred more to the moderator, and contacted the moderator over concerns rather than communicate directly with one another. As the moderator, I would describe my experience of the two groups as more facilitating and supporting the face-to-face group, but more directing and eliciting with the virtual panel. Although more study is needed before drawing any definitive conclusion, there was a more passive and dependent pattern with the virtual group in contrast to the face-to-face group. Given that the moderator was the same in both cases and the general approach the same, I ascribe this to the differences in modality. However, the face-to-face group was more aware of an imminent deadline and so may have felt more anxiety.
Another distinction in moderating the two groups was the need to provide cues to the virtual panel that exist in the environment of the face-to-face panel. Firstly, social/emotional cues exist in the context and vocal cues of speech. People not only have more data on the communication, but people also moderate their interactions more in face-to-face discussions. These cues are missing in text messages, which can lead to misinterpreting one another. The moderator needs to be aware of this and not only clarify content of a communication but also meaning and emotional tone as appropriate. Time is another factor people can get from the context. In asynchronous communication, the relationship between current time on task and an approaching deadline is not discernible. Time is fragmented into smaller units, making it difficult to judge whether you are allowing sufficient time to meet a deadline. In face-to-face meetings it is fairly easy to judge how much time is required in early tasks and project whether there is sufficient time for the overall project. The moderator of a virtual group needs to provide time cues and assist the virtual group in managing time.
A combined approach
Many of the participants in the virtual panel requested a face-to-face meeting, ostensibly to build upon the relationships created during the panel discussions. This also raised the question of the optimal method for integrating the virtual panel with a face-to-face meeting. In our own work, we frequently use online interactions as a way to maximize the use of direct meetings. Any routine discussion, background materials, planning, etc. that does not require decision-making is conducted in advance online. The face-to-face time, which is considered the most expensive and difficult to co-ordinate, is used for on-topic discussion and decision making. The question raised by this project is whether the virtual format can completely replace a face-to-face meeting or whether some hybrid approach is preferred. Time is certainly one deciding factor. The strength of the virtual panel lies in part in the asynchronous format that allows people to adapt the demands of the panel into their own schedule. This advantage also means that more time must elapse for panel interactions, although not more time on task, because people spread the tasks throughout their schedules. If time is limited or a deadline is close, online decision-making may not be a practical solution. Given sufficient lead-time, it appears that a combined approach makes for the greatest efficiency, but this remains to be examined.
Lessons learned
Based on this experience, we would make the following recommendations to people considering a virtual format for an expert panel:
- Make everything clearer than you might otherwise. People are used to scanning online information, and therefore are more likely to skip important information. Using a tool to verify that people have understood crucial information is important if that understanding is critical to the purpose of the group.
- A related clarification issue is the fact that most people write cryptic e-mail devoid of cues as to the authors emotional tone. This makes it easy for messages to be misinterpreted. An experienced moderator may assist with clarifying such communications. The moderator generally plays the role of facilitating accurate understanding, and in a virtual panel this may include accurate understanding of emotional tone. This occurred most often when subjects expressed disagreement. There appears to be a bias in the reader toward interpreting the tone in an e-mail in a more critical or personal way than intended. Emotions (symbols that convey emotion, such as a happy face $$$) can be useful in clarifying tone.
- Set a clear expectation for the volume of mail and limit people so that they cannot exceed that volume. Frequent e-mail messages with low content are burdensome for people with expectations of a low frequency of participation, while people seeking a higher rate of interaction see a low rate as a lack of support or interest. The result can be the division of the group into low-frequency and high-frequency factions. Likewise, set a clear expectation for the time a panelist may be expected to participate. Clear expectations for rate, volume, and duration of participation will reduce discrepancies in expectations. In a face-to-face group, these expectations are clearer. The meeting has a time-frame, social cues to speak up or speak less are more apparent, and the deadline for group interaction is clear. The normal events of a day, such as meals and breaks, cue the face-to-face group that a deadline is approaching. In a virtual group, these need to be made explicit and restated periodically.
- Insure that your online system is responsive to a variety of systems. We had people using relatively outdated systems and had to make sure that our site was responsive to the lowest common denominator. Alternatively, set minimum equipment expectations and then determine the impact on the panel composition of excluding people who cannot meet those minimums.
- Set clear expectations for responding to material presented online. An asynchronous system is much more dependent upon the participants to structure their time and hold themselves accountable to the task. Correspondingly, the moderator has far less control over the participants and is more vulnerable to their ability to maintain their motivation. In this project, when draft guidelines were distributed for discussion or vote, a time-frame was stated and reminders posted periodically. In addition, the moderator had the responsibility of tracking the participation of each panel member and directly contacting those who were not meeting established time-frames or participation standards.
- Augment the e-mail discussion group with another system that can provide support services. In this project the virtual panel was encouraged to use a website designed for the project. Background materials, technical support, documents, and other information for the project can be separated from the discussion so that the e-mail is focused on the task and not interspersed with administrative requests. Quizzes on concepts central to the project verified the general understanding of basic concepts. The e-mail from the discussion groups was also archived online to allow research and reference to prior messages.
- The role of the moderator, crucial to any panel, takes on some additional tasks with a virtual panel, in addition to the points mentioned above regarding time management and setting guidelines and expectations.
Limitations of the present study
There are several important limitations to the present study that will need to be addressed in the future. Firstly, the primary task of the project was the creation of standards rather than the experimental testing of two forms of collaboration. Subjects were not assigned at random and the tasks assigned to the two panels were not identical. The virtual panel started with nothing but a general assignment—what is needed to eliminate a gap in quality of care—while the conference started with the standards proposed by the virtual panel. These differences are important in considering the ratings of the panel process since the processes and expectations for outcomes were different. On the other hand, we believe that the approach used is much closer to how a panel would function in a real setting.
Another limitation is the determination of quality of product. In this project, quality was defined as the members’ perception of quality: the perception of the standards as a quality standard, the perception that the standards reflect their views, and the perception that the process was effective. A more compelling approach would be to assign a task that lends itself to objective measurement, and to assess the different methods for producing the outcomes from the task. This would help address the question this study could not answer: does one method produce better standards than the other?
Several questions for future study are raised by this project. These include: What is the optimal relationship between a face-to-face conference and a virtual panel? What distinct roles and skills are needed to successfully manage an online panel? What types of decision making are best suited for online panels? Are some individuals better suited for one type of panel or the other?
Lastly, the cost information is illustrative of the potential for low cost collaboration, but it is not within the scope of this project to determine cost-effectiveness. A follow-up project where comparable tasks are assigned to two groups working in different media is needed to assess cost-effectiveness.
|
Acknowledgements |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
Special thanks to Dr Dana Keller for his assistance with the statistical analysis, to Dr Carol Traxler and Ms Senda Benaissa for their assistance with the project, and to Mss Sally Dunn for managing the conference. This project was funded by a contract from the Centers on Medicare and Medicaid Services (formerly HCFA).
Accepted for publication November 16, 2001.
|
References |
|---|
|
TopIntroductionMethodsResultsDiscussionAcknowledgementsReferences |
|---|
- Weisband S, Atwater L. Evaluating self and others in electronic and face-to-face groups. J Appl Psychol 1999; 84: 632–639.
- Michailidis A, Rada R. Activities and communication modes. Int J Hum Comp Stud 1997; 46: 469–483.
- Straus S. Getting a clue: the effects of communication media and information distribution on participation and performance in computer-mediated and face-to-face groups. Small Group Res 1996; 27: 115–142.
- Hollingshead A. Information suppression and status persistence in group decision making: the effects of communication media. Hum Commun Res 1996; 23: 193–219.
- Straus S. Technology, group process, and group outcomes: testing the connections in computer-mediated and face-to-face groups. Hum Comp Interact 1997; 12: 227–266.
- Finholt T, Sproull L. Electronic groups at work. Organ Sci 1990; 1: 41–64.
- Weisband S, Schneider S, Connolly T. Computer-mediated communication and social information: Status salience and status differences. Acad Manage J 1995; 38: 1124–1151.
- Kiesler S, Sproull L. Group decision making and communication technology. Organ Behav Hum Decis Process 1992; 52: 96–123.
- Bowers L. Constructing international professional identity: what psychiatric nurses talk about on the Internet. Int J Nurs Stud 1997; 34: 208–212.
[Medline] - Tse A. Comparing the response rate, response speed and response quality of two methods of sending questionnaires: e-mail vs. mail. J Mark Res Soc 1998; 40: 353–361.
- Myers L, Thyer B. Social work practice with deaf clients: issues in culturally competent assessment. Soc Work Health Care 1997; 26: 61–76.
[Medline] - Phillips B. Bringing culture to the forefront: formulating impressions of deaf and hard of hearing people at time of medical crisis. Prof Psychol 1996; 27: 137–144.
- Hogan A. Issues impacting on the governance of deafened adults. Disabil Soc 1997; 12: 789–801.
- Speight S, Thomas A, Kennel R, Anderson M. Operationalizing multicultural training in doctoral programs and internships. Prof Psychol Res Pract 1995; 26: 401–406.
- Clayton M. Delphi: a technique to harness expert opinion for critical decision-making tasks in education. Educ Psychol 1997; 17: 373–386.
- Palmer RH, Banks N, Edwards J, Fowles J, Nerenz D, Zapka J. Final Report: External Review Performance Measurement of Medicare HMOs/CMPs. Easton, MD: Delmarva Foundation for Medical Care, 15 August 1994. Grant under a contract with the Health Care Financing Administration, contract 500-93-0021.
- Palmer RH. Using clinical performance measures to drive quality improvement. Total Qual Manage 1997; 8: 305–311.
- Michailidis A, Rada R. Activities and communication modes. Int J Hum Comp Stud 1997; 46: 469–483.
CiteULike 
Connotea 
Del.icio.us What's this?
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||