Concept

Requirement Gathering

We conducted a semi-structured interview with 2 teaching assistants to get an idea of the tasks that instructors perform with Blackboard and Piazza. We identified the most common tasks to be answering unresolved questions on Piazza, uploading assignments, homeworks and posting grades.

An online survey was done with users of blackboard and piazza to elicit our requirements. Checking grades, posting/answering a question and checking for content availability stood out to be the most common tasks among student users. We also recorded users’ general concerns with voice assistants in order to address them in our design.

Informal observations to see how people interact with the skill during presentation of our hackathon project (open-room setting) and also by placing Amazon Echo in a closed-room setting. We learned that users preferred simple and short commands over longer sentences.

User Requirements

• Students must be able to check grades posted by instructors for an exam/assignment/homework/project.
• Students must be able to check for content availability (course materials/homework/assignments).
• Students must be able to check due dates for assignments/homework/projects/exams.
• Students must be able to set reminders for deliverable due dates and exam dates.
• Students must be able to ask/answer questions in a class discussion forum. There should be options for posting public, anonymous questions and answers to the entire class.
• Instructors must be able to check and answer unresolved questions posted by students in the discussion forum.

Sketches

Formative Evaluation

Wizard of Oz setup.

We conducted a formative evaluation study with 5 users - 3 students, one student who is also a TA, and one instructor. We used Wizard of Oz prototypes, a technique in which users interact with a system that users believe to be autonomous, but is actually being operated or partially operated by an unseen human being. We gave the users an overall idea of the tasks that they needed to perform and what was the end goal of each task, letting them perform the tasks naturally. We got feedback from how they interacted with the system and what follow up questions they had to ask after the initial command, and based upon this, modified our existing design sketches.

Lessons learned:

• Users sometimes wanted Alexa to repeat some of the options in case they missed/misheard them.
• Users wanted to know grades for specific homeworks and assignments in specific courses and also had follow-up questions regarding the class average.
• Before checking unresolved questions or answers, users preferred to know answers to “how many?” and “how long?”. Based on the answer they received, users then decided whether to send the content to their mobile phones or have them read aloud.

Summative Evaluation

Oz setup.

We used Oz studies, where users interact with the actual working system. Users performed four different tasks using the Alexa whiteboard skill, these studies were not the field studies, although field studies could have given us more information in natural settings due to time constraints we conducted the summative evaluations in a controlled manner. We measured time taken by the user to complete the task, success on task and the errors that occurred during the execution of the task. We also measured user satisfaction using the NASA TLX surveys.

Lessons learned:

• The user satisfaction score for whiteboard was 96%,the average satisfaction scores for all the individual tasks were all above 90%.
• When asked how likely they were to use our product on a scale of 1 to 5, all of our users gave a 5/5.
• The average success rate for all the tasks was 95%.
• The average time on each task was less than a minute.
• Most of the errors that we observed are recognition errors, Alexa had trouble recognizing user’s commands accurately we 65% of the errors are recognition errors and only 30% are interaction slippages and the rest 5% are out of vocabulary errors.
• All of the interaction slippages were in the first two tasks, and most of them are when users forgot the skill name whiteboard. Our users are new to using a skill on Alexa, we believe this was the prime reason behind the interaction slippages. With time they will eventually disappear.
• We believe speech recognition can be even more accurate in the coming years.

Setup

Video