-21-Course Quality Improvement in Design Education

Teaching design in technology and engineering education is one of the key Standards for Technological and Engineering Literacy (International Technology and Engineering Educators Association, 2021). Still, it can be challenging to provide a high-quality educational experience aligned with the standards required in introductory technology and engineering courses. Even after thorough preparation and delivering valuable content to students, students may feel the course was of low quality because it was required or not interesting. Our study investigated how improving the design thinking in technology courses based on the Expectancy Value Theory of motivation can impact students’ perceived course quality, as evidenced by course and instructor ratings. This study included nearly 1000 students in 25 sections. Quantitative analysis was completed using a t-test to measure differences in end-of-course evaluation scores for each section, and qualitative analysis of student evaluation responses was done through thematic analysis method. Our study showed that after making changes to a course based on Expectancy Value Theory, students’ ratings of both the course and instructor increased significantly. Expectancy Value Theory focuses the instructors’ course improvement efforts on three aspects which were manageable and resulted in significant improvement.


Introduction
Introductory engineering and technology courses are challenging to teach because the population is so critical for programmatic retention, and the main focus of these courses is to help with higher-order learning to make students capable of addressing today's complex industrial challenges (Morosan et al., 2017). According to some students in our required Introductory Design in Technology course, "This course did not teach me anything and was just a bunch of busy work" and "Personally, I did not find this class very useful, nor did I learn a great deal." As instructors in a prestigious research institution, we recognized the "...need to understand technology's impacts on our lives, society, and the environment, as well as how to use and develop technological products, systems, and processes to extend human capabilities" (International Technology and Engineering Educators Association [ITEEA], 2021, p. 3), but we were frustrated every semester to receive these kinds of comments on the end-of-semester student evaluations of course and instructor.
Many secondary school technology and engineering programs comprise elective courses where students can choose to take the classes or not, and student choice may be directly related to the quality of the course. If secondary school students do not perceive courses in an elective program to be high quality, that program may close due to declining demand. At the college level, some instructors may not be rehired if students perceive their courses to be less than high quality.
We used the Expectancy Value Theory (Barron & Hulleman, 2015) as a theoretical framework to redesign an existing Design Thinking in Technology course. This theory aligned with the concerns we found in previous course evaluations that related to expectations of success, the extent to which students valued the course, and the cost of engaging in course activities. This case study measured the impact of these changes on course quality using students' end-ofcourse evaluation data. Eccles et al. (1983) proposed the Expectancy Value Theory of motivation which posited that students would be motivated to perform a task if they have an expectation of success (whether they will succeed in the task), and they can see how much value is tied with the task (is it enjoyable now or will it be good for me later?). According to Eccles and Wigfield (2002), another aspect that plays a major role in the Expectancy Value Theory is cost. Both expectancies of success and achievement value are significant predictors of motivation and are considered positive, whereas cost is defined as a negative aspect related to student motivation. One aspect of cost, for example, relates to the duration of time required to complete a task. Cost is often measured by asking questions like, "Am I free of barriers preventing me from investing time, energy, and resources into the activity?" Cost can also be linked to stress, embarrassment, and other student emotions while doing the task and may be difficult to measure. Barron and Hulleman (2015) proposed the Expectancy Value Cost model of motivation, which describes the different dimensions associated with these three variables and a formula to predict student motivation. According to Barron and Hulleman, expectancy and value contribute to student motivation while cost detracts. Ball et al. (2016) and Wigfield and Eccles (2000) showed that a student's choice of achievement tasks was most directly predicted by their expectation for succeeding at the task. Following Wigfield and Eccles (2000), many studies have used interventions based on the Expectancy Value Theory, such as increasing student motivation through value-added assignments to understand its effect on students' learning outcomes (Abraham & Barker, 2015;Hulleman et al., 2017).

Expectancy Value Theory
Our study attempts to contribute to this literature by investigating the effects of improving a Design Thinking in Technology course based on the Expectancy Value Theory framework. The impacts of course improvements were measured using standard end-of-course evaluations. Our hypothesis is that emphasizing opportunities for success and the value of assignments while reducing the costs will motivate students and enhance their perceptions of the quality of the course experience.

Research Context
The study was conducted in a first-year course on Design Thinking in Technology at a Midwestern, post-secondary, research institution. Findings from this study may be relevant to technology and engineering teachers in high schools because the quality of introductory experiences impacts student retention (Aljohani, 2016). The course was offered in 12 sections in Fall of 2017 (comparison semester) and 14 sections in Fall 2018 (treatment semester). Instructors included graduate students, lecturers, clinical faculty, or tenure-track faculty. Each instructor taught 1-4 sections of 40 students. The focus of the course was based initially on the Standards for Technological Literacy (standards 8, 9 and 11;International Technology Educators Association;ITEA, 2000) and now on Standards for Technological and Engineering Literacy (STEL) standard on Design with practices which include: (a) Systems Thinking, (b) Creativity, (c) Making and Doing, (d) Critical Thinking, (e) Optimism, (f) Collaboration, (g) Communication and (h) Attention to Ethics (ITEEA, 2021, p. 14).
This active learning course had three projects for students, where the first two projects were small in scope and aimed to help students learn the design thinking process. The final project was eight weeks long. Here, students were expected to identify a global grand engineering challenge in small groups and narrow down the problem to identify the key stakeholders and users at a local level. The students were then challenged to develop and create solutions to mitigate or reduce the effect of the identified problem by designing a prototype. The prototype was further tested with users, and a final prototype was refined based on user feedback. The course was offered as a flipped class (to foster active learning) in that all the course content was shared with the students prior to the actual class meeting where it was applied. Class time was used for active discussions of topics and design activities. To monitor and encourage student preparedness for the class, assignments and quizzes related to the reading materials were given. Most of the before-class assignments were to be completed individually, and most of the in-class discussions and activities were group-based.
In response to student complaints and negative student evaluations, we made significant changes to the course in the summer of 2018 based on the Expectancy Value Theory. Pragmatically, this study investigated how the treatment group students perceived the revised version of the course in terms of course quality compared to a previous group of students.

Course Improvement Based on Expectancy Value Theory
Two theoretical foundations drove the course improvement approach. First, we followed the backward design approach (Wiggins & McTighe, 2005) to understand curriculum planning. According to Wiggins and McTighe (2005), the design of courses should be based on learning and not on teaching. The three stages of the backward design approach are: (a) identify desired results, (b) determine acceptable evidence, and (c) plan learning experiences and instruction. Second, course improvements were based on the Expectancy Value Theory.
To improve the course, multiple instructors collaboratively laid out all the course materials provided to students in the comparison semester and assessed an Expectancy Value Cost relationship for each assigned course material or activity. Based on what we know of students through past experiences, we asked ourselves how prepared were students to be successful in each learning experience. We also asked how the learning experiences scaffolded students from where they were to the successful completion of an assignment. Then we considered what value the students would perceive from engaging in the assignment or learning experience and how we could optimize the actual value and the perception of value. Lastly, we considered how much time (cost) was required by the student to complete the assignment. Other driving questions included:  How well did the assignment or experience align with the course learning outcomes?
 To what extent were the outside class readings aligned with the in-class tasks?
 How much work can students complete reasonably within the class period?
 Were the guiding questions and instructions clear for students?
Changes made in each of the three main focus areas related to Expectancy Value Theory are illustrated in Figures 1-3.

Figure 1
Expectancy of Success and Related Changes

Figure 2
Value of Educational Experience and Related Changes

Research Design
The primary purpose of this case study was to evaluate the impact of course revisions on three outcomes: (a) an overall increase in the course rating, (b) an overall increase in the instructor ratings, and (c) an improvement in the way students felt about the course quality.

Participants, Measures, and Data Collection
A total of 515 students were enrolled in the comparison semester (Fall of 2017), and 590 students were enrolled in the treatment semester (Fall of 2018). The majority of students in both groups were freshmen (Table 1) and male (Table 2). To assess pre-existing differences between groups, we acquired students' SAT and ACT scores from the university. SAT scores prior to 2016 and ACT scores were converted to equivalent current SAT scores for comparisons. Results of a t-test revealed statistically significant differences between groups (t(28) = 4.7, df =1046, p = < .001) with the comparison group having statistically higher mean scores (M = 1234.3, SD = 120.9) than the treatment group (M = 1269.5, SD =121.0). Though this difference existed, the practical differences between groups were not expected to impact the results drawn from this study, and, therefore, we continued with the analysis.
Both the treatment and comparison groups had similar instructors (Table  3). Some instructors from both semesters had previous teaching experience, some had previous teacher preparation program experience, while others were new to the course or new to teaching. Instructors in both semesters were provided standard materials through a copy of the course learning management system (Blackboard); weekly support meetings were guided by an experienced course coordinator

Student Evaluations of Course and Instructor
An end-of-semester course evaluation was sent to students through software called 'CoursEval.' CoursEval is considered a valid and reliable instrument as it is being used by many universities across the world to measure the quality of the course and teaching effectiveness (Hakimzadeh & Williams, 2006;Piascik & Bird, 2008;Socha, 2013). There were two quantitative items in this evaluation where students were asked to rate separately the course and the instructor using a scale of 1 to 5, where 1 = Very Poor, 2 = Poor, 3 = Fair, 4 = Good, and 5 = Excellent ("Overall, I would rate this instructor as:" and "Overall, I would rate this course as:"). One free-response, qualitative item directed students to "Make a suggestion(s) for improving the course. (A criticism alone is not helpful; tell your instructor how you would fix any problem.)"

Data Analysis
Out of 1105 (515 from comparison and 590 from treatment) students to whom course evaluations were sent, 83% (n = 428) from the comparison group and 84% (n = 495) from the treatment group responded to the evaluations.

Quantitative Data Analysis
For the quantitative analysis, the course ratings and the instructor ratings from the comparison semester and the treatment semester were compared using a two-tailed t-test for the difference in means assuming unequal variance based on Levene's test.

Qualitative Data Analysis
The study drew on the traditions of the case study approach (Creswell, 1998). A combined technique of 'inductive' and 'deductive' thematic analyses were used to analyze the qualitative data. This approach allowed themes to emerge from the raw data using inductive analysis (Fereday & Muir-Cochrane, 2006).

Inductive Thematic Analysis
According to Braun and Clarke (2006), inductive thematic analysis is "a process of coding the data without trying to fit into a pre-existing coding frame, or the researcher's analytic preconceptions" (p.77). NVivo 12 software was used for coding, and the process of coding followed the following six steps: 1. Familiarizing yourself with your data 2. Generating the initial codes 3. Searching for themes 4. Reviewing themes 5. Defining and naming themes 6. Producing the report (Braun & Clarke, 2006) Interrater Reliability To increase the validity and reliability of the thematic analysis, we checked interrater reliability. Interrater reliability is defined as the degree to which ratings of two or more raters or observations of two or more observers are consistent with each other (Tashakkori & Teddlie, 1998;Marques & McCall, 2005).
After coding in NVivo software, 20% of the un-coded data from both comparison semester and treatment semester were shared with a separate rater. This second rater was also given a codebook that consisted of descriptions of each code, an example, and rules for each code. Once the rater coded the shared data, a coding comparison was made to assess interrater reliability. According to Cohen (1960), the statistic used to measure interrater reliability is a value from 0 to 1. The analysis yielded a kappa value of 0.82. According to Marques and McCall (2005), this value of kappa is considered as a strong level of agreement.

Deductive Thematic Analysis
Crabtree and Miller outlined deductive thematic analysis, also known as the a priori template approach, in 1999. In this approach, a template in the form of codes/keywords is used to arrange the data for subsequent interpretation. In deductive analysis, the researcher gives a detailed definition of the template before conducting the deductive analysis. The template is sometimes based on the preliminary scanning of the data or based on the theoretical framework of the research study (Fereday & Muir-Cochrane, 2006). In this study, the deductive analysis was based on the theoretical framework for the Expectancy Value Theory of motivation.

Quantitative Results
There was a significant difference in the scores for course rating between the comparison semester (M = 2.95, SD = 0.46) and treatment semester (M = 3.41, SD = 0.42) (t(28) = -3.14, p = .008). This was also the case for the difference in scores for instructor rating between the comparison semester (M = 3.5, SD = 0.48) and treatment semester (M = 4.23, SD = 0.33) (t(28) = -5.18, p < .001). The test results suggest that students perceived the course and instructor significantly better after the course improvements were made.

Qualitative Results
An inductive coding in the NVivo software yielded 13 codes, such as 'busy work,' 'instructor complaints,' 'unorganized,' 'course should not be required,' and 'need more time on assignment.' Similar codes were grouped to form categories or themes. Three overarching themes emerged: (a) time, (b) instructor expectation, and (c) course relevance. Following inductive coding, a deductive analysis was performed on each of the three themes through the lens of Expectancy Value Theory to understand student perceptions of course quality.

Theme 1: Time Required to Complete Assignments
The emergent theme of time (required to complete assignments) was defined as the number of hours students spend to complete an assignment. This time theme includes perceptions of busy work or tedious work that did not contribute to significant learning. Typical student responses that were categorized in the time theme are seen in Table 4.

Comparison Semester Treatment Semester
"Tech 120 isn't a challenging course but contains a bunch of busy work not needed. I would suggest to only have one project in the semester instead of three [projects]." "A lot of the work felt like busywork and wasn't always easy to complete in a timely manner -not because it was difficult, but because it was tedious." "The class took the most time out of class on homework than any of my other courses did, and the work I was doing out of class felt very overwhelming and unnecessary." "Overall, I enjoyed taking this course and feel like I have learned quite a bit, sometimes I felt like the work required for us to do outside of class (at the beginning of the semester) was very time-consuming." "Get rid of all the busy work, students can tell when an assignment is for nothing especially when it is thrown to the wind the next day in class. Try to make whatever they're doing important." "At first the course seems like busy work until you realize at the end that the information taught actually helps you for the future. I found out that I didn't know how to brainstorm...who would've thought that? I'm not sure how to make the beginning of the course seem less tedious or how to make students realize the importance early on. Other than that, the course was structured pretty well." The deductive coding of the time theme revealed student perceptions of their ability to be successful and perceptions of the course's value and costs related to engaging in course activities. Students in both conditions indicated that the preparation was 'busy work' and took a lot of time to complete. Overall, the comparison semester students were frustrated by their perception that the coursework lacked value. Treatment semester students indicated that assignments were not difficult to complete, and the strength of students' complaints was less. Some students in the treatment semester proposed strategies to reduce the number of assignments. For example, one student in the treatment semester mentioned: "It would have been nice to only been assigned readings or videos or a worksheet and not all three at once. I think a combination like two of the three would be alright as well, but all three once was really quite a bit of work." The analysis revealed that treatment semester students enjoyed the course and realized the value of the course content and the assignments. One of the students from the treatment semester mentioned that: The class has not been nearly as bad as some make it out to be. I feel that it cannot be more suited to students without taking out needed lessons. The course is fine just the way it is in my opinion. It just pushes people out of their high school days and into college by making them work outside of class of their own free will and making them take initiatives or risk failing. Students wrote about the costs associated with completing assignments in terms of some assignments being too long. In the comparison semester, a student had mentioned: "I would suggest to possibly break the assignments down a bit more so that there are not any that take over 3 or 4 hours to complete." This was considered while improving the course; in the treatment semester, there were no student-revealed barriers to completing assignments.

Theme 2: Student Perception of Instructor's Expectations
The instructor expectation theme was defined as students' perceptions of how well the instructor communicated the objectives and expectations of the course with students. The codes included in this theme were 'unorganized course,' 'no clear assignments or goals,' and 'instructor complaints.' Typical responses to this theme are presented in Table 5.
The deductive coding of instructor expectation was based on students' perceptions of their ability to be successful, course value, and costs related to engaging in course activities. In the comparison semester, students frequently stated that they felt unable to meet the instructor's expectations because they did not understand the expectations due to unclear faculty instructions or unorganized course contents. Many students in the comparison semester were confused about the instructions given by the instructors; one student commented, "we were confused on what we were even supposed to be learning." Table 5 'Instructor Expectation' Theme: Examples of Student Responses to "Make a Suggestion(s) for Improving the Course."

Comparison Semester Treatment Semester
"There were times where the instructor said one thing about the grading of the assignments, but the grader used the rubric, which said something entirely different. This led to a lot of confusion." "I felt like the course went very smoothly and I have no suggestions on improvement." "The course at time seemed unorganized. It would be helpful if the due dates of assignments were clear, and your words matched blackboard [the course LMS]." "Perfect," "I have no suggestions," "I wouldn't really change the class." "Make the directions on the assignment clearer, give more than 5-10 minutes to do in-class assignments that take like 20-25 minutes to do." "[I] understand the materials better. The [instructor] shows up better prepared." In the treatment semester, students less frequently reported the course as being confused or unorganized but more frequently suggested having the option to switch groups if (or when) their group members were not contributing. Some of the comments in the treatment semester included, "I did not like working in groups within this class. I found it very frustrating when my classmates failed to participate and complete assignments on time," and "Group projects need to be reworked. If a group member is not cooperating, and thus work cannot be submitted in reasonable time, the entire group receives a zero for the assignment because no late work is accepted." Most of the students felt that the instructors did not do a good job in differentiating between the project objectives, which may have made many students feel that the course projects were repetitive within the semester. One of the students' comments related to this concern was, "Working on the design process also felt repetitive and felt like this course should not be an entire semester long. Project 2 felt like a smaller version of Project 3, it felt like we did everything twice." Another student suggested to "make it relevant and less repetitive." This general concern was voiced frequently by students in the comparison group spanning multiple instructors. On the other hand, the treatment semester students had many positive comments about instructor expectations, such as, "The course was very well set up and I would not change it." Another student mentioned that "I found the class to be very helpful in starting my scholarly project." In the comparison semester, some of the students complained about the costs associated with deadlines for assignments not being set correctly by the instructor. Where the deadlines were set incorrectly at the beginning of the semester and changed suddenly as the assignment approached, many students spent extra time and effort to complete the assignment. One of the students felt that the instructor "was not very good at assigning assignments and making it known they are due -stick to the schedule." Students articulated that it took too much effort to understand how to do the assignment, "I think he needs to do a better job explaining how to do some of our outside assignments. There were times when I felt that I couldn't do the assignment because I wasn't sure how to do it." Students in the treatment semester had no negative comments related to unexpected or undue costs associated with the course.

Theme 3: Course Relevance
Course relevance, as a theme, was defined as how the students felt about the course in terms of its contribution to their knowledge and skills (Table 6).

Comparison Semester Treatment Semester
"This course shouldn't exist." "I could see it being a benefit for someone never introduced to an engineering environment before…" "The course does not feel necessary for a lot of students." "Maybe don't limit the options for creative projects so much. For example, the 'Campus Safety project' was really hard for certain majors to come up with practical idea, whereas for other majors it's very straightforward and easy." "I don't think I'd have this class be a requirement for the college, as it doesn't help all majors." "This is a good class and can be very beneficial to people only in certain majors." Deductive analysis by the course relevance theme addressed students' perceptions of being successful, course value, and costs of engaging in course activities. The course was an introductory, common course for first-year polytechnic students. Regardless of the condition (treatment or comparison), some students struggled to see how the course related to their majors. Aviation majors, for example, who wanted to become pilots were not sure how the design thinking course was relevant to the aviation field. Though they had this general struggle, these students did not indicate any concerns with their expectations of being successful in the course. A student from the comparison semester stated: "All that was assigned were projects and homework that either don't really relate to what my major entails or that students are already doing in the classes within their specific major courses." Students in the treatment semester also found it difficult to make a direct connection between the course and their major, though many found the course content relevant and useful. One commented, "I have already found myself using and accepting the terminology taught in the class about user-based design." Another student from the treatment semester mentioned, "I believe it would be beneficial to the class as [a] whole if the different majors and areas of study were taken into greater consideration throughout the course." Though students in the comparison and treatment groups struggled with value in the context of their major, they did not mention any significant barriers (costs) to completing the assignments related to their perceived relevance to their major.

Discussion and Conclusion
It is challenging to teach introductory engineering and technology courses as students may not see the relevance of their study until later years or sometimes when faced with real-world problems (Yelamarthi & Drake, 2015). In Summer 2018, the instructors and course coordinators of a required design thinking course at a midwestern university evaluated and modified the course through the lens of Expectancy Value Theory. The revised course was implemented in Fall 2018. This study analyzed the impact of these curricular changes by comparing the end-of-course evaluation data from Fall 2017 (before revision) to Fall of 2018 (after revision, treatment).
There were significant differences in student ratings of both course and instructor between the comparison and treatment semesters, with the treatment group having higher ratings. The written comments requesting suggestions for improvement had some similar responses related to perceptions of time required to complete assignments and course relevance. Some notable differences in the written comments included:  In the treatment semester, many students responded with comments such as, no suggestions for improvement, the course was perfect, no comments, good course etc. compared to comparison semester where there were no such comments.
 The intensity of negative comments was noticeably less in the treatment semester compared to the comparison semester.
 Unlike the comparison semester, the students in the treatment semester provided valuable feedback on how to improve the course.
As a result of the quantitative improvement in the course and instructor scores, as well as the decrease in quantity and magnitude of negative comments, these findings suggest the use of Expectancy Value Theory as a successful lens by which instructors may review and improve their course materials. Collegiate and secondary school technology and engineering instructors may find that focusing upon students' expectations of success and their perceptions of value while reducing perceived costs associated with completing course assignments may also improve student evaluations of course and instructor. Using the Expectancy Value Theory of motivation as a lens to review our course materials led to a measurable improvement in students' course evaluations quantitatively and qualitatively. This may improve student retention and increase demand for both secondary and post-secondary design educators.

Limitations and Future Studies
A single survey of quality administered at the end of a course may not generate sufficient insights into the strengths and weaknesses of a course. Many factors could influence students' perceptions of the course value beyond those we control as teachers, such as the political climate, peer attitudes, and institutional value structure. Our study focused narrowly on aspects of the course that we could influence. Future research might use more diverse data gathering techniques spread across the semester. For example, conducting interviews at strategic times in the semester could provide richer insights.
Given that the comparison groups significantly differed by SAT scores, course and instructor ratings likely vary by SAT scores and other student demographics. Future research could examine these differences as they may suggest opportunities to address motivational differences among a diverse student population.