Robert G. Main, Ph.D.
Department of Communication Design
California State University, Chico

Students in a communication technology class at California State University, Chico critically tested an interactive multimedia distance learning system for use in teaching psychomotor and cognitive learning objectives. The test included evaluation of system software and hardware setup and operation procedures and a series of trials in lesson delivery.

The distance learning setup was a multipoint system that was partially simulated in a local area network. (See Figure 1.) The students were grouped into three clusters (classrooms) of three stations each. The instructor was located in another location with communication limited to the use of the system technology. The communication link included a video server as well as the nine student stations and the instructor station. Each station had a VisualClass software developed by the Mei Technology Corporation installed in Windows for Workgroups 3.1.1 environment. In addition, Starworks, Microsoft Office Suite, and Quizzard application programs were provided on each hard drive. Quizmaster software was installed only on the instructor's workstation.

Figure 1
LOCAL AREA NETWORK SETUP

Two-way audio and visual communication were provided by a PictureTel system. Small color cameras allowed the instructor to see each workstation as well as view the students demonstrating psychomotor skills they had learned. A classroom monitor allowed the students to see the instructor's face or the instructors workstation screen display which served as an overhead projector in presenting graphic presentations. The instructor could also transmit video taped material over the PictureTel system. (See Figure 2.)

Figure 2
AUDIO AND VIDEO SET-UP

Research Design and Data Collection Methods

A major contribution of this project is the design and development of a research methodology and the data collection techniques for evaluating interactive multimedia distance learning systems.

A two by two design was used to evaluate instructor led and student centered learning modes for classes involving psychomotor skills development in using measuring tools and for a class on interviewing skills. Each lesson had a traditional instructor delivered class mode and one in which the learners collaborated with each other in small groups. (See Figure 3.)

Test Design

Figure 3. Two by two test design

There were two components of the beta trial evaluation. The first was an assessment of the prototype interactive multimedia system's installation and operation and the second was the development and validation of a research design and data collection technique appropriate for any multimedia distance learning system.

The specific systems' performance evaluation was done using observation and journal records of students participating in the instructional design and lesson development process. A human factors evaluation team developed the methodology and collected and organized the data for the research design and data collection test.

Components of the system performance evaluation included:

Evaluation data was collected by direct observation of learners and instructor and by multiple video recording of classrooms and instructor and the lesson presentation as transmitted. The entire data stream as it appeared on the instructor's screen was captured for analysis with observation reports and the classroom video recordings.

Live Observation Methods

Two observers were present for all trials. One observer sat to the left front of the remote classroom and the other about l-5 feet to the left of the instructor in the simulated originating site. The observer in the classroom introduced himself briefly to the students before the class began, explaining that he was there to observe how the system functioned and that he had nothing to do with evaluating the students in any way. The observers did not interact with any of the students, instructors, or other members of the Chico research team during the class sessions. Members of the technical team did interact with students and instructors while trouble-shooting system difficulties such as workstation freezes or the technical failure of the delivery of instructional material. None of these interactions were related to lesson content, only lesson delivery.

The goal of live observation should be to aid the task of organizing and making sense of the large amount of data that can be collected through audio and video tape, not to just generate more data. Live observation can help index and catalog what recorded events to analyze and to provide some additional data about those events from a first hand perspective. The perception of the importance of events is often difficult to assess from just videotapes, and the recording of all media of communication can disclose events as meaningful when live observations would miss them.

A simple coding scheme for noting key events can be useful if used with a time indexing system that can efficiency link the notes to specific portions of the audio and video recordings.

The observers agreed upon a few basic key symbols to quickly note features of the interaction during the trial. The observers also synchronized stop watches at approximately the same time as video tapes began taping the trials. Whenever a symbol was written down, the time was noted. The purpose of the notation system was to document when key types of interactions took place rather than attempting to analyze the specific behaviors. This was to be used later to assist in narrowing the focus of the analysis of the video tapes. The following symbols were used:

P = content relevant pause. Unusually short or long pauses after a question, request, or other utterances after which a delayed response is expected.

C = cross talk. Two or more people attempting to talk at the same time.

R = repair mechanism. An utterance or exchange of utterances serving to correct some violation of usual conversational rules or norms. These include apologies, jokes, or metacommunication about interrupting, cross talk, speaking too loud, too soft, not providing enough information, providing too much information, providing irrelevant information, not providing relevant information.

Audio Recording

Three directional microphones were suspended from the ceiling pointing to each of three clusters of work stations. These microphones provided the audio feed to the instructor and they where mixed and fed to the VCR that was recording the split screen images. Audio tracks on three camcorders were recorded directly on the camcorder tapes. A miniature cassette tape recorder was temporarily used by one observer during part of the first trial to take notes but was abandoned as too intrusive. The instructor wore a lapel microphone that was used to feed audio to the remote site, and this was recorded along with the video of the feed to the remote classroom.

Video Recording

Twelve views of the classroom and system screens were recorded on video tape. The large number of views was tested to verify what angle of vision, depth of field, and screen size would prove useful for taping of the system operation in the next phase of the project. The camera views included:

A screen splitter allowed for four views to be recorded on one tape using an industrial quality VCR at VHS resolution. The other views were recorded on separate consumer quality VCRs or on cassette tapes in camcorders. Some views included a sound track and others did not. Taping on all machines began at approximately the same time but a common time code was not available.

Evaluation of System Performance

Although the data collected was used for evaluating the system performance and learning outcomes, the small number of subjects in each of the four trials (nine) did not permit valid statistical manipulation. The primary purpose of these trials was to validate methods and technology of data collection. other purposes were to make recommendations to the instructional design team based on observations of the use of the system, and to make recommendations to the technical team regarding efficient enhancements or corrections to be made in the system from a human factors perspective.

Conclusions from the four trials were that the system was effective for fostering the learning of content for each of the classes taught over this system. Once students had grown accustomed to the features of the system, they seemed to enjoy it. In addition, once the instructors reached a certain comfort level with the system, they were able to improvise part of the lesson on their own using other features in the system which were not called for at that particular time in the lesson plan script.

Hardware and Software.

Although some temporary hardware limitations and software bugs prevented the delivery of all of the material prepared by the instructional design teams, the system did prove flexible in its ability to offer a variety of methods for delivering content.

However, typical of any new software development, there were numerous bugs in the execution that frustrated both students and instructors. Some of the problems included the following.

• Erroneous error messages appearing at random on student screens.
• E-mail messages lost when student logs off, sender unaware of messages lost, receiver doesn't know if message is personal or to the group.
• Screens frozen when opening some applications or when instructor freezes screen to launch an instructional media, the screen cannot receive the media if an application running on the student station. Cannot be unfrozen without rebooting CPU.
• Passing control from instructor to student created many problems.

1. If student was given instructor control, then returns control to instructor; student's log off button continues to act as instructor's. If student pushes this button, it will log everyone off the system even though that student returned control to instructor.
2. When instructor has enabled students to see the student list: after instructor passes control to a student and student returns control, the student list is lost from that student's screen. Instructor must take student list away from all, then resend student list to all before that student can see it again.
3. If instructor passes control to a student, then that student passes control to yet another student, instructor loses ability to regain control. The student that passed it has the regain control button on their screen. if second student passes control to a third student, both the first and second students have regain control buttons.
4. If instructor passes control to a student, that student has the ability to freeze everyone's screen–including the instructor's. That student also has the ability to "snoop" into someone else's screen without their knowledge or permission. That student can also log everyone off the system, including the instructor.

• The electronic hand raising function did not permit queuing. If instructor has not acknowledge first student's virtual hand raised, subsequent students' hand raises do not activate instructor's hand up screen unless they reraise their hand after the first has been acknowledged.
• The quiz master function allowed multiple copies of the test on some student stations. If a student completed both copies of the test, the CMI function accepted them and included the results in scoring computations distorting class statistics.
• The video server cards installed on the station hard drives were not compatible for running with the application program.
• Unless the start-up sequence was followed EXACTLY, student stations tended to freeze up, had access problems to the network, or had access problems in Visual-Class.

Instructional Design and Lesson Development

A number of obstacles were logged in the instructional design and lesson development processes. These included:

• Time constraints of lesson instructors and student designers made for less input from instructors than would have been ideal. Instructors had to use lesson plans that they did not design themselves. They also did not get to do a practice class session.
• Needed more time for collaboration with other teams. Some student designers were not aware of the technical problems concerning launching video out to each station. Had they known, they would have designed their lesson plan differently. Both teams had to make some last minute changes in the lesson plan. Measurement team scrapped their video and PowerPoint presentations altogether. The interview team had to install files to individual hard drives minutes before the test class began.
• Lesson instructors needed to be familiarized with the capabilities of the hardware and the software to which they were restricted. They needed time to get used to the new technology to obtain a comfort level in which they could concentrate on teaching the lesson, not on figuring out the computer.

The inexperience of the student designers also caused problems during the design and development process.

• Skill levels of student designers were varied. It took some time for team leaders to discover the skill levels of their group members. Those student designers who knew what they were doing did most of the design work, but those who did not know how were frustrated by the complexity of the projects and their inability to contribute.
• Due to limited knowledge about distance learning, student designers found it difficult to judge how the students would react to common strategies of instruction if used with this new medium. This resulted in bouncing ideas back and forth several times that would normally be accepted in a traditional training situation without controversy.
• Again, due to limited knowledge about distance learning, actual lessons took longer than was allotted in the lesson plan to present. The biggest problems seemed to be the amount of time the instructor needed to obtain feedback from the students and vice versa.

Lesson Presentation

The software technical bugs and the flaws in the design and development process impacted the instructional presentations. Other problems with the system were also exposed in the presentations.

• The camera at the measurement lesson's workstation was not set to auto-focus. The instructor was unable to read the dial clearly enough to judge if the student had performed the operation correctly or not.
• The student at the workstation could not view the instructor via the classroom monitor, so could get only audio feedback from instructor. It was inadequate.
• For the interview lesson, it was difficult for the instructor to gauge distances between students by only watching them on the cameras.
• Instructor did not control the camera views. A technician in the classroom who attempted to anticipate the instructor's and student's needs and switched the cameras accordingly.
• There were problems with the instructor's microphone/headphone set-up. The headphone cable would brush against the lapel mic, often over riding the instructor's voice. Having the instructor's microphone cut off the classroom mics became too confusing. There was also an echo problem that was annoying to the instructor.
• There was no way to view the students in three classes at one time. There was also no view of individual students.

Interaction

Interaction is considered a particularly critical variable in achieving learning objectives in the psychomotor and affective domains. The interactive multimedia distance learning system was specifically designed to facilitate instructor-student, student-student and student-instructor exchanges. The trials indicated a number of corrections need to be made before the prototype meets the design expectations. Problems exposed included:

• Students were confused on the best medium for communicating with the instructor. Some clicked on their virtual hand on screen, some used e-mail, some raised their physical hand, some just called out audibly. There were problems with all methods.

1. Raising their virtual hand appeared futile because there was no feedback to tell the student that the instructor could see their virtual hand raised.
2. Communicating by e-mail also had no feedback that the instructor received their message. It was also very slow to communicate this way.
3. Since the instructor did not control the camera switching, and only one cluster of students could be viewed at a time, raising their physical hand did not gain the attention the students desired.
4. Calling out audibly had the quickest response, but also interrupted the class. The instructor did not have the option of "ignoring' the student's question for a moment while he finished his thought.
5. Instructors were not familiar enough with the capabilities of this set-up before they began teaching the class. They were not comfortable with the technology and how to operate it, and seemed more focused on the technology than they were on the students.
6. While one student was at the workstation, that student had the full attention of the instructor. The rest of the class could not view that student manipulating the calipers, so the rest of the class was put in an awkward limbo with nothing to do while waiting for their turn at the workstation.
7. While measurement instructor was answering e-mail, students to whom he was not communicating did not have anything to do and were not aware that instructor was communicating with the other students. There seemed to be some awkward silence from time to time.

Recommendations

The evaluation of the system performance produced many very specific recommendations for correcting the software design. Some of those with potential for broader application include the following.

• One instructor suggested having "hot buttons" for standard responses in the e-mail section. It could work like an automatic text or macro that could be preprogrammed with words such as "OK," "correct," and "check your answer again," etc.
• Log off button, freeze screen button, and view button should not be passed to students when instructor passes control to them. Instructor should maintain these controls at all times.
• Students would like to know who else in the class has their virtual hand raised.
• When instructor calls on student who has their virtual hand up, their virtual hand should come down automatically.
• Student should be able to withdraw their hand up with acknowledgment from the instructors.
• One suggestion was to have a queue on instructor's screen of all students who have their hands raised AND their questions.

1. Student writes a message and put their hand up. Hand shouldn't go up unless there is a message to send. Student's name goes to top of list on instructor's screen.
2. Instructor chooses, for example, the 3rd student down on the list. That student's hand goes down on their own screen. Instructor can broadcast that student's question to all students' screens and also answer it.
3. All other students can withdraw their questions by putting their hand down on their screen.

• Instructor should be able to control camera switching. It would be nice if computer could control it with the use of "F" keys.
• Students like to know who else is "in class" – suggest always having the student list available to every station. It should update every once in a while in case someone logged off.

Learning Assessment

As stated previously, the small number of subjects in each cell precluded any hypothesis tesing. Howeve, some assessment of learning outcome was possible through the student responses to quizzes delivered over the system and the appearance of learning during classroom interactions and demonstrations.

Students in the measurement class were able to follow examples of the physical manipulation of objects through the use of both a live camera placed on a display table, and through viewing digitized video clips. This was demonstrated by their ability to manipulate tools correctly on the display table and their responses to the instructor over the e-mail system. Students in the interview class were able to understand and critique the nonverbal behavior of simulated interviews. Instructors of both types of content were able to evaluate student performance of psychomotor skills by viewing the behaviors of others over the system. As expected, cognitive learning and its evaluation were effectively facilitated.

The affective dimensions of motivation and attitude regarding the learning situation and the content significantly improved as instructors and students became more comfortable with the operation of the system. The interview lesson instructor commented that for some reason she and the students were more at ease with one another using this system than they would have been if they were all in the same room performing the same activities. She said that the students were more personable and appeared to find it easier to talk to her and to each other than it otherwise would have been in a face-to-face classroom situation teaching this subject. She summed it up by saying that the students "were having fun." This seemed to promote learning through a classroom atmosphere that encouraged questions and student participation in exercises.

Investigating the use of peer mail may be more important than initially thought. At one point during the trials it appeared as if the students were busy sending responses to the instructor or figuring out responses t the problems, and the instructor's communication with the students seemed minimal. Upon analyzing some records of peer mail it was discovered that during this time, some students were sending the instructor personal messages while other students worked on the assignment, and the instructor was giving some short responses. Sending these messages appeared to encourage the students to explore the different functions of the peer mail system and overcome some student and instructor nervousness about using the system. The instructor eventually sent messages to discontinue using the system for messages that were not directly related to the lesson.

Several important events were taking place. Students and instructors were gaining experience with the system, students and the instructor were overcoming nervousness about the use of the technology, negotiation was taking place regarding the acceptable use of the system for student-instructor interaction, the instructor was expressing his authority, and the distinction between task related and non-task related virtual classroom communication was partially defined. All of this took place on the peer mail system entirely out of the awareness of the live observers and was not documented by the audio or video recorders. In fact, the interactions were not known to all of the students. Thus some of the students were engaging in these adaptive and norm-generating interactions while others were not. This could prove significant for explaining subsequent differences in classroom performance, levels of interaction through other portions of the system as well as through peer mail, reported satisfaction levels, and motivation.

The collaborative learning approaches used for these trials were primarily limited to exercises among students that were partially structured by the instructor. Nonetheless, there were significant differences between the instructor led instructional design trials and the collaborative design trials regarding effectiveness and efficiency. Instructors and students agreed that the collaborative instructional designs had clear benefits for quick adaptation to the system and more efficient learning of the lessons. Often collaborative learning is thought to increase the time it takes for covering the same amount of materials. In these trials the collaborative instructional designs decreased the time to cover the same amount of material.

The system was judged effective for fostering learning by the students despite hardware limitations and software bugs. The system did prove flexible in supporting a variety of instructional strategies and learning activities. In addition, once the instructors reached a certain comfort level with the system, they were able to improvise part of the lesson on their own using other features in the system which were not preplanned in the lesson.

Assessment of Data Collection Methods

Classroom Observers

Live observations of the IMDL classroom can be accomplished without interfering with the instruction or contaminating the data. The impact of an observer can be minimized by (1) explaining to the students that the observer's presence is part of the technical support of the system that has nothing to do with student evaluation, (2) avoiding interaction between the students and observer during the class period, (3) minimizing the data collection technology by using feeds from audio and video equipment being used as part of the IMDL system, (4) placing observers to the side or back of the classroom, and (5) the observer avoiding any unnecessary movement or noise, including facial expressions, posture or gestures that might be construed by students as responses to events taking place during the class.

Audio Recording

The directional microphones generally provided quality sound recordings of students in clusters of three workstations. However, when several people were talking, for instance when students at different workstation clusters were working on a collaborative project, it was difficult to distinguish separate conversations. The feed from different microphones should be recorded on separate tracks so that volume levels can be adjusted to better separate conversations.

The constantly open microphone configuration introduced some feedback and echo problems to the system. The instructors experienced the echo of their own voice to be one of the most distracting features of the system. The feedback problems at the remote site was tolerable. However, due to adjustments intended to minimize the feedback the volume level was lower than might be useful in a learning environment with higher volume ambient noise. The classroom was probably more acoustically isolated than actual classrooms will be.

Variations of room acoustics may impact feedback problems. During the trial there were large areas of smooth concrete floor and painted walls that were acoustically reflective. A curved curtain was pulled around the student area that provided some relief from room echo. Room acoustics that minimize sound bounce would help reduce echo.

During one of the classroom demonstrations, students moved to a different area to conduct a simulated interview. The microphone placement was difficult to adjust since they were suspended from two points to angIe them toward the workstation clusters and away from the speaker on the video monitor at the front of the room. Suspending the microphones directly down from a single point would probably cover a more useful area and be less intrusive. However, this would require quality directional microphones with good rejection characteristics or this configuration could introduce more feedback problems.

The volume level on the audio recording was so low that it significantly decreased the S/N ratio. Volume levels need to be carefully set and adjusted to different acoustic situations. The configuration during the trial did not allow for unobtrusive monitoring or adjustment of the audio levels. The audio mixer was placed at a distance from the recorder, requiring the technician to walk across part of the front of the room to check levels and make adjustments. He remained at the video controls throughout the class period.

It is recommended that a four track audio cassette deck with a built in equalizer is used for recording sound during future IMDL trials. A stereo feed from the recorder/mixer should be sent to the video recorder. Since such units integrate a mixer and recorder in a small footprint, the researcher could easily and quickly monitor and adjust sound levels. During analysis, if some interactions are difficult to hear, the four track tape could be used to better isolate conversations or side remarks, or to compensate for periodic ambient noise.

Video Recording

The common time notation between observers served to provide an index for comparing observational notes. The time and symbol notation did not serve well to identify video taped interactions to be analyzed later. There needs to be a common time frame among recording devices to which observers can refer while taking notes, and so video tapes from different views can be compared. This was requested for the trials but a time code generator was not available and since most of the recording equipment was consumer quality they did not contain time code generators themselves.

Since there will be video recording devices at both the originating and remote sites, it is recommended that a real time clock is used rather than attempting to synchronize frame counters or a time code beginning at an arbitrary zero. This would be preferable even at the loss of some measurement resolution or minor synchronization problems between the originating and remote sites. Observers should also refer to clock time during observations to assist indexing the tapes.

The four split screen images recorded at VHS quality are sufficient for analyzing most interactions, if a good sound track is also placed on the tape. Few facial expressions are clearly distinguishable at VHS resolution on split-screen images, but posture, gestures, and head movements along with a quality sound track provide good information. The ability to see four views simultaneously off-sets most of the disadvantages of poor image detail. The use of S-VHS quality cameras and recorder would solve some image quality problems stemming from the use of a split screen.

A recording of a workstation screen as one of the four split screen images proved useless, even with post-production image processing. Even a S-VHS definition split screen image is unlikely to yield useful data. Screen images need to be recorded full screen at S-VHS quality to provide anything but the most general information.

A Hi-8mm camcorder with hi-fl sound and a zoom microphone was used. Even though it was a relatively inexpensive consumer Hi-8mm camera (single CCD chip) the image quality was noticeably superior for observing facial expressions. Although it did employ a zoom microphone and hi-fl stereo sound, the sound quality was not sufficient due to ambient noise levels. The camcorder was placed nearly thirty feet from the subjects at a high angle.

BIBLIOGRAPHY

Baker, E.L., & O'Neil Jr., H.F. (Ed.). (1994). Technology assessment in education and training. Hillsdale, N.J.: Lawrence Erlbaum Associates.

Berge, Z.L., & Collins, M.P. (Ed.). (1994). Computer-mediated communication and the on-line classroom: Overview and perspectives. Cresskill, N.J.: Hampton Press.

Blaye, A., Light, P., Joiner, R., & Sheldon, S. (1991). Collaboration as a facilitator of planning and problem solving on a computer-based task. British Journal of Developmental Psychology, 9, 471-483.

Cathcart, R., & Gumpert, G. (1983). Mediated interpersonal communication: Toward a new typology. Quarterly Journal of Speech, 69(3), 267-277.

Chilcoat, Y., & DeWine, S. (1985). Teleconferencing and interpersonal communication perception. Journal of Applied Communication Research, 13(1), 14-32.

Clark, R.E. (1994). Assessment of distance learning technology. In E.L. Baker & H.F. O'Neil Jr. (Eds.), Technology Assessment in Education and Training (pp. 63-78). Hillsdale, N.J.: Lawrence Erlbaum Associates.

Clark, R.E., & Sugrue, B.M. (1989). Research on instructional media: 1978-1988. In D. Ely (Eds.), Educational media and technology yearbook (pp. 19-36). Denver, CO.: Libraries Unlimited.

Dweck, C.S. (1985). Intrinsic motivation, perceived control, and self-evaluation maintenance: An achievement goal analysis. In C. Ames & R. Ames (Eds.), Research on motivation in education: Vol. 2: The classroom milieu. Orlando, FL: Academic Press.

Farr, C.W., & Shaeffer, J.M. (1993). Matching media, methods, and objectives in distance education. Educational Technology, 33(7),

52-55.

Galbreath, J. (1995). Compressed digital videoconferencing: An overview. Educational Technology, 35(1), 31-38.

Giardina, M. (Ed.). (1992). Interactive multimedia learning environments: Human factors and technical considerations on design issues. New York: Springer-Verlag.

Haalen, T., & Miller, G. (1994). Interactivity as a predictor of student success in satellite learning programs. DEOSNEWS, 4(6) LISTSERV@ PSUVM.PSU.EDU.

Haynes, K.J.M., & Dillon, C. (1992). Distance education: Learning outcomes, interaction, and attitudes. Journal of Education for Library and Information Science, 33(1), 35-45.

Hodgson, V.E. Case Study: The interrelationship between support and learning materials., 56-61.

Holloway, R.E., & Ohler, J. (1991). Distance education in the next decade. In G.L. Anglin (Eds.), Instructional technology: Past, present, and future (pp. 259-266). Englewood, CO.: Libraries Unlimited, Inc.

Jaspers, F. (1991). Interactivity or instruction? A reaction to Merrill. Educational Technology, 31(3), 21-24.

Johnstone, S.M. (1991). Research on telecommunicated learning: Past, present, and future. In S.M. Johnstone (Eds.), Research on telecommunicated learning (pp. 49-57). The Annals.

Longworth, N. (1990). From computing to information technology to distance education: What do we do when teacher isn't there? In A. McDougall & C. Dowling (Eds.), Computers in Education (pp. 931-936). North-Holland: Elsevier Science Publishers.

Maloy, W.L., & Perry, N.N. (1991). A navy video teletraining project: Lessons learned. The American Journal of Distance Education, 5(3), 40-50.

Phelps, R.H., Wells, R.A., Ashworth Jr., R.L., & Hahn, H.A. (1991). Effectiveness and costs of distance education using computer-mediated communication. The American Journal of Distance Education, 5(3), 7-19.

Ritchie, H., & Newby, T.J. (1989). Classroom lecture/discussion vs. live televised instruction: A comparison of effects on student performance, attitude, and interaction. The American Journal of Distance Education, 3(3) 36-45.

Russell, T.L. (1992). Television's indelible impact on distance education: What we should have learned from comparative research. Research in Distance Education, 4(4), 2-4.

Shaw, B., & Taylor, J.C. (1984). Instructional design: Distance education and the academic tradition. Distance Education, 5(2), 277-285.

Simpson, H., (1990). The evolution of communication technology: Implications for remote-site training in the navy. (No. TN-90-22). Navy Personnel Research and Development Center.

Simpson, H. (1993). Conversion of live instruction for videoteletraining: Training and classroom design considerations. (No. TN-93-4). Navy Personnel Research and Development Center.

Simpson, H., Pugh, H.L., & Parchman, S.W. (1990). A two-point videoteletraining system: Design, development, and evaluation. (No. TR-90-5). Navy Personnel Research and Development Center.

Simpson, H., Pugh, H.L., & Parchman, S.W. (1992). The use of videoteletraining to deliver hands-on training: Concept test and evaluation. (No. TN-92-14). Navy Personnel Research and Development Center.

Wagner, E.D. (1992). Separating myth and reality in distance education. Educational technology, 32(10), 42-45.

Wagner, E.D. (1993). Variables affecting distance educational program success. Educational technology, 33(4), 28-32.

Walther, J.B. (1992). Interpersonal effects in computer-mediated interaction: A relational perspective. Communication Research, 19(1), 52-91.

Whittington, N. (1987). Is instructional television educationally effective? The American Journal of Distance Education, 1(1), 47-57.

Zhang, S., & Fulford, C.P. (1994). Are interaction time and psychological interactivity the same thing in distance learning television classroom? Educational Technology, 34(6), 58-60.