OVERVIEW OF THE 1 YEAR PROJECT
*Due to research ethics guidelines, I can only share some of my findings and design-research process.
*Due to research ethics guidelines, I can only share some of my findings and design-research process.
My Role: Main UX researcher in collaboration with a project team of 10 (a teaching faculty - principal investigator, teaching assistants, and university technicians) to deploy and evaluate a teleconference system between two classrooms at different campuses.
I conducted end-to-end user research for this project.
What I Led: Participant recruitment, dissemination of web surveys, stakeholder interviews, and study analysis.
Additional research contributions: literature reviews & framework analysis, mixed methods study design involving web-surveys and conversational interviews, usability + functional testing of the teleconference system, producing project proposals, writing progress reports, and findings presentation.
I conducted end-to-end user research for this project.
What I Led: Participant recruitment, dissemination of web surveys, stakeholder interviews, and study analysis.
Additional research contributions: literature reviews & framework analysis, mixed methods study design involving web-surveys and conversational interviews, usability + functional testing of the teleconference system, producing project proposals, writing progress reports, and findings presentation.
CONTEXT
PROCESS
KEY CONSIDERATIONS FOR RESEARCH INQUIRY
To further understand the value of a telepresence room for teaching and learning, I assisted with conducting literature reviews and identified that (1) adding a layer of interactivity and (2) supporting sense of physical presence can enrich the learning experience in a multi-campus classroom setting.
Based on the elements of physical presence and interactivity from the learning environment,
I derived three key considerations to address with reference to the framework for Universal Design for Learning:
- The display and sharing of multimedia course content between the two remote classrooms (Representation)
- The way students and instructors converse and collaborate during discussions and presentations (Engagement)
- The way students and instructors can articulate their intentions and actions verbally and on-screen simultaneously for both
situated and remote audiences (Expression and Action)
I derived three key considerations to address with reference to the framework for Universal Design for Learning:
- The display and sharing of multimedia course content between the two remote classrooms (Representation)
- The way students and instructors converse and collaborate during discussions and presentations (Engagement)
- The way students and instructors can articulate their intentions and actions verbally and on-screen simultaneously for both
situated and remote audiences (Expression and Action)
To address the key considerations, I formulated three main research questions to guide my user study:
- Will students accept the use of teleconferencing as a viable alternative learning experience to the traditional face-to-face
instruction model?
- What are the key parameters of telepresence to incorporate in the system and interface design?
- How can the parameters be used to enhance the experience for students?
- Will students accept the use of teleconferencing as a viable alternative learning experience to the traditional face-to-face
instruction model?
- What are the key parameters of telepresence to incorporate in the system and interface design?
- How can the parameters be used to enhance the experience for students?
SUMMATIVE & FORMATIVE RESEARCH HIGHLIGHTS
Evaluating Student Experiences in Web Surveys
To serve as an indicator for success, the mixed method survey design is informed by the Technology Acceptance Model (TAM) to elicit for students’ feelings towards the system. The TAM framework takes its premise on the usablity criterion of "acceptance" from the domain of information systems.
Indicators include:
- initial feelings (level of easiness, level of satisfying) when using the video-streaming system for their learning experience
- attitudes and willingness to adopt and engage the integration of the system for their learning experience
- finding insights on their degree of confidence and skills needed for their interactions with the system
- reveal social influence factors that arise which may impact their learning experience with the system
- identify difficulties and potential issues of key system components when accessing and using the system
The web-based survey was used as the primary data collection instrument since it provided greater accessibility and timeliness for participant’s response. The survey consisted of 16 questions, organized into 8 parts. We administered the survey using SFU Websurvey system to a total of 158 students across two semesters, and 115 responses were received. Students who participated in the survey were enrolled in the design course. The questions were measured using the Likert Scale of Agreement and inquired on how the system design has impacted their classroom experience and learning outcomes. Responses were gathered anonymously. The survey took approximately 15 mins to complete.
Indicators include:
- initial feelings (level of easiness, level of satisfying) when using the video-streaming system for their learning experience
- attitudes and willingness to adopt and engage the integration of the system for their learning experience
- finding insights on their degree of confidence and skills needed for their interactions with the system
- reveal social influence factors that arise which may impact their learning experience with the system
- identify difficulties and potential issues of key system components when accessing and using the system
The web-based survey was used as the primary data collection instrument since it provided greater accessibility and timeliness for participant’s response. The survey consisted of 16 questions, organized into 8 parts. We administered the survey using SFU Websurvey system to a total of 158 students across two semesters, and 115 responses were received. Students who participated in the survey were enrolled in the design course. The questions were measured using the Likert Scale of Agreement and inquired on how the system design has impacted their classroom experience and learning outcomes. Responses were gathered anonymously. The survey took approximately 15 mins to complete.
Demographic Summary for Survey Participation
Conversational Interviews with Students and Teaching Assistants after Lectures
During break times and after class, I conducted informal conversations with students/TAs on their teaching and learning experiences through the system. The questions focused on what students liked, moments of confusion, friction points, and overall impressions with the system. The questions similarly took basis from the TAM framework. These informal conversations were particularly useful for contextualizing the recommendations on teaching strategies.
User Observation (video analysis)
Video recordings of the lectures were taken to allow behavioral insights on the interactions between the teaching staff, students, and the teleconferencing system. Observing user behaviours were critical as we needed to identify tangible teaching strategies which the teaching staff can adapt into their existing classroom practices. There was also no prior data that provided insight on how to implement and facilitate the multi-classroom experience from the user's point of view, and how changes in the physical setup can impact the teaching and learning experiences.
Usability Testing & Weekly Reports
The system design for the teleconferencing system took an iterative approach. Adjustments and tweaks were made to the physical set-up incrementally to address technical issues and inconsistencies between the two campuses. Weekly reports were sent to the university technician and research team highlighting usability issues with proposals for updates to ameliorate the interactive experience between the two classrooms. When new changes are implemented, we will invite the teaching team to test out the system (outside of regular class time) using the talk-aloud method. This incremental and iterative approach helped reduced making recommendations and changes to the overall system based on assumption-based decisions as we were taking into account of the university's technician's advice, and the data collected from the teaching and learning experiences. Collaborating with the university technician was invaluable as it helped me understand the limitations and creating an optimal system set-up that balances the user experience with physical and budgetary constraints.
User Observation (video analysis)
Video recordings of the lectures were taken to allow behavioral insights on the interactions between the teaching staff, students, and the teleconferencing system. Observing user behaviours were critical as we needed to identify tangible teaching strategies which the teaching staff can adapt into their existing classroom practices. There was also no prior data that provided insight on how to implement and facilitate the multi-classroom experience from the user's point of view, and how changes in the physical setup can impact the teaching and learning experiences.
Usability Testing & Weekly Reports
The system design for the teleconferencing system took an iterative approach. Adjustments and tweaks were made to the physical set-up incrementally to address technical issues and inconsistencies between the two campuses. Weekly reports were sent to the university technician and research team highlighting usability issues with proposals for updates to ameliorate the interactive experience between the two classrooms. When new changes are implemented, we will invite the teaching team to test out the system (outside of regular class time) using the talk-aloud method. This incremental and iterative approach helped reduced making recommendations and changes to the overall system based on assumption-based decisions as we were taking into account of the university's technician's advice, and the data collected from the teaching and learning experiences. Collaborating with the university technician was invaluable as it helped me understand the limitations and creating an optimal system set-up that balances the user experience with physical and budgetary constraints.
A KEY CHALLENGE
What the research team did not know in advance was that we eventually did need to differentiate usability improves from overall system performance. There are technical limits to what usability design can accomplish, and other system performance issues that are unrelated to usability improvements such as signal integrity, IP protocols (which were out of the research team's control). (For example, there were spontaneous spam calls which took over our system midway through class. This dropped the connection between the two classrooms, and caused a disruption and abrupt stop of class).
OUTCOME
RESEARCH SUMMARY
Using a modified TAM framework, we were able to correlate the usability of the system design with the teaching and learning user experience based on the notion of acceptance and rejection by students.
RESEARCH ACCOMPLISHMENTS
Internal research report highlighting the teaching and learning experience of multi-classroom courses and system design
An internal report was provided to university management proposing the potentials of this instructional method with details on implementation, teaching and learning experiences.
An internal report was provided to university management proposing the potentials of this instructional method with details on implementation, teaching and learning experiences.
Fast Implementation Process (within a week) of new physical hardware setup (Monitors, Mics, Cameras)
Established a matured & regular communication flow between the teaching team and the IT services staff from Harbour Centre and Surrey campuses for the maintenance and updates to the video-streaming system.
Established a matured & regular communication flow between the teaching team and the IT services staff from Harbour Centre and Surrey campuses for the maintenance and updates to the video-streaming system.
Refinement in the system interface design for better audio-visual control
A notable feature which we updated was including a visual map of the classroom seating behind the control buttons. This helps to better orient the presenter on which controls (ex. camera capture) will impact which audience members.
A notable feature which we updated was including a visual map of the classroom seating behind the control buttons. This helps to better orient the presenter on which controls (ex. camera capture) will impact which audience members.
Developed system interface audio-visual guidelines and created user overview guidebook
This resource was created for teaching faculty, guest presenters, and general technicians when using the teleconference system at both campus locations.
This resource was created for teaching faculty, guest presenters, and general technicians when using the teleconference system at both campus locations.
Snapshot of User Manual for Updated Control Interface Design.
SIGNIFICANCE & CONTRIBUTIONS TO TEACHING AND LEARNING
Identified key motivator and elements that support in-class active participation through the teleconferencing system.
Students have highlighted the importance in the sense of physical presence between the instructor and students as a key motivator for in-class active participation. To help augment the sense of presence, more intuitive and natural exchange in conversations between the instructor and students during in-class discussions is needed.
Students have highlighted the importance in the sense of physical presence between the instructor and students as a key motivator for in-class active participation. To help augment the sense of presence, more intuitive and natural exchange in conversations between the instructor and students during in-class discussions is needed.
3 key elements the sense of physical presence across the classrooms, which include:
- mediation of eye contact,
- clear voices, and
- perceivable references to displayed audio-visual content during discussions
Showcased 4 main scenarios with instructor and teaching assistant roles to facilitate for a multi-campus course experience.
We presented where the teaching staff are positioned in the classroom, how the controls should be used, how to guide students to use the system, and how to present and refer to on-screen lecture materials during class.
- mediation of eye contact,
- clear voices, and
- perceivable references to displayed audio-visual content during discussions
Showcased 4 main scenarios with instructor and teaching assistant roles to facilitate for a multi-campus course experience.
We presented where the teaching staff are positioned in the classroom, how the controls should be used, how to guide students to use the system, and how to present and refer to on-screen lecture materials during class.
Research findings were presented as a case study focusing on learning technology and telepresence at the Annual Symposium on Teaching and Learning: Universal Design for All Learners in Spring 2016.
REFLECTIONS
What did I learn?
While it is exciting to try new things, It is important to account and balance the priorities of the different stakeholders involved during the research and development process - aka the "give and take". For example, while we want to implement a new feature to the system, it was important to represent the students' interest to postpone the update by a week to ensure the system does not break during the week of midterm reviews.
Things don't always go as planned, be flexible but firm when need be. Study plans should be used as a guide rather than a script to account for technical errors or time constraints. Be confident and vocal on what you need to accomplish during the usability testing/interview session.
While it is exciting to try new things, It is important to account and balance the priorities of the different stakeholders involved during the research and development process - aka the "give and take". For example, while we want to implement a new feature to the system, it was important to represent the students' interest to postpone the update by a week to ensure the system does not break during the week of midterm reviews.
Things don't always go as planned, be flexible but firm when need be. Study plans should be used as a guide rather than a script to account for technical errors or time constraints. Be confident and vocal on what you need to accomplish during the usability testing/interview session.
What would I have done differently?
Propose a co-creation session with various departments to gather broader insights towards the system design so it can be more accessible and inclusive for other disciplines. At the moment, the context of this research only focused on design, studio-lab settings and on-screen lecture materials.
Propose a co-creation session with various departments to gather broader insights towards the system design so it can be more accessible and inclusive for other disciplines. At the moment, the context of this research only focused on design, studio-lab settings and on-screen lecture materials.