-4-Scaffolding Middle and High School Students’ Engineering Design Experiences: Quality Problem-SCOPEing Promoting Successful Solutions

Highly proficient expert engineers begin the iterative process of design by thoroughly investigating the design problem. Engineering students are often distracted by surface details, leading to a faulty conception of the problem and inappropriate solution strategies. Adequate problem-scoping is arguably the most important step in the design process. To address this issue, the researchers developed an instructional framework to help teachers scaffold students’ cognitive and metacognitive processes during the problem-scoping phase of a design challenge. The purpose of this quasi-experimental study was to investigate the impact that scaffolded instruction related to the SCOPE process had on students’ solution success during a design challenge. The SCOPE process is used to help teachers scaffold students’ design experiences during a tower design challenge and increase the overall effectiveness of their design efforts. Students in this study ( N = 802) were separated into treatment and control groups. Using hierarc hical multiple regression, the SCOPE process accounted for 40.4% (ΔR 2 = .404) of the variability of the design score, which was statistically significant ( p < .001). The results indicate that students who received scaffolded instruction from their teachers related to the SCOPE process during the design experience performed better on the design challenge.


Introduction
The purpose of this quasi-experimental study was to investigate the impact that scaffolded instruction related to the SCOPE process had on students' solution success during a design challenge. The independent predictor variable was the SCOPE process. SCOPE is an acronym for Study, Criteria, Organize, implement their knowledge and experience while utilizing various strategies, such as designing from first principles, to approach design tasks with a systematic view of the design situation (Cross & Cross, 1998). When designing, expert engineers apply both cognitive and metacognitive skillsets associated with problem-identification, task clarification, information management, project management, negotiation, concept generation, reflection, evaluation, and refinement that are crucial for solution success during complex design situations. Additionally, expert engineers have developed the social skills needed for effectively communicating designs as well as working with clients and team members (Dym et al., 2005).
Engineering students (i.e., novice designers) are singularly focused on a solution and not an iterative design process; students spend less time at nearly all stages of the design process (Atman et al., 2007;Becker et al., 2012). There is a pronounced difference between experts and students in time spent at the problem-scoping stage (Atman et al., 2007;Becker et al., 2012). "Problem definition is a critical step in design thinking. It is the first stage of engineering design [,] and it sets the foundation for developing solutions" (Becker et al., 2012, p. 18). Failing to thoroughly identify the problem, gather and manage information, and consider the systematic nature of the design situation results in novice designers misunderstanding the problem. Misled by a faulty conception of the problem and failing to realize that it is faulty, a combination of errorsboth cognitive and metacognitive-inevitably leads novices toward flawed solutions.

Dispositions, Cognition, and Metacognition
The design literature clearly identifies key differences between expert and novice designers related to specific skills underlying successful design (Atman et al., 2007;Atman et al., 2008;Becker et al., 2012). These underlying skills constitute broader skillsets, including dispositions, cognition, and metacognition, that should be a well-integrated explicit focus during K-12 design experiences. To help students manage the complexity of design, educators should explicitly focus on developing underlying skillsets during scaffolded design experiences. Dispositions, as well as cognitive and metacognitive skillsets, are presented as important in the design literature. These encompassing skillsets serve as umbrella terms within the design literature to represent numerous underlying skills (see Table 1). Design thinking and engineering habits of mind (i.e., dispositions) are examples of phrases used in the design literature to encompass other design skills such as systems thinking, communication, collaboration, ethics, and empathy. Sheppard et al. (2009) stated, engineer tries to discover the heart of the problem and explore beyond the solutions at easy reach. (p. 100) problem-framing) Cognitive skillsets are also implicitly identified as important in the design literature, including problem-scoping, generating alternative solutions, estimating (i.e., predicting), modeling, experimenting, and continuous evaluation (i.e., iterating). Additionally, terms such as reflection, planning, information gathering (i.e., information management), and knowledgeimplying the cognitive processing of declarative, procedural, or conditional knowledge-are used in the design literature to implicitly describe important metacognitive skills. For K-12 educators to foster students' abilities with these umbrella skillsets, educators will need to understand the underlying skills, the interconnectedness of those skills, and the recommended approaches for skill development within the learning environment.

Engineering Design Process
For expert designers, the engineering design process is not a step-by-step approach but rather a systematic and purposeful approach used for solving complex, often ill-structured, open-ended problems (Cross & Cross, 1998). To help novice designers develop their design ability, the engineering design experiences need to be more systematically structured and scaffolded (Denson & Lammi, 2014). All the engineering design processes seem to have common activities, including problem or need identification, information gathering, idea generation, modeling, analyzing, evaluation, decision making, communication, and implementation. Atman et al. (2007) used a relatively common design process that includes the activities listed above and added design stages. The systematic nature of the design process suggests that designers need to work more efficiently toward optimum solutions based on initial and thorough problem-scoping.
During the implementation of an engineering design process, there is intent to integrate the application of science and mathematics concepts; develop students' dispositions, cognition, and metacognition through the explicit application of skills; and thorough problem-scoping for outcome success.
The National Center for Technological Literacy suggested that "The key to educating students to thrive in a competitive global economy is introducing them early to the engineering design skills and concepts that will engage them in applying their math and science knowledge to solve real problems." (Becker et al., 2012, p. 2) Scaffolding students' experiences during design activities involves guiding them through the implementation of the engineering design process (Denson & Lammi, 2014). Teachers need to scaffold students' implementation of the design process in order to prompt students to allocate enough time during problemscoping or any other stage in the design process (Becker et al., 2012;Atman et al., 2007). Additionally, helping students make explicit connections between the engineering design process (i.e., the doing) and cognitive and metacognitive skillsets (i.e., the thinking) used during design is an important aspect throughout scaffolded design experiences.

Importance of Problem-Scoping
Problem-scoping, the first stage of the engineering design process, is directly related to the success of the design solution. Students engaged in design do not comprehensively scope the problem, and this negatively influences many aspects related to the solution's success (Atman et al., 2007;Becker et al., 2012). Problem-scoping has three interrelated yet distinct activities: (a) identifying the need, (b) defining the problem, and (c) gathering information. Problem-scoping activities further involve designers' dispositions and utilization of cognitive and metacognition skillsets. Novice designers do implement their cognitive and metacognitive skillsets, but they do not have the experience of experts nor the explicit, scaffolded design experiences to implement their skillsets in a way that leads toward success (Atman et al., 2007;Becker et al., 2012;Denson & Lammi, 2014). Becker et al. (2012) discussed The National Academy of Engineering Committee on K-12 Engineering Education's 2008 review of 15 high school engineering curricula that found design as a main theme with problem-identification most often listed as the first design activity. However, Katehi et al. (2009) noted that the curriculum did not engage students in robust problem-identification. The lack of focus on the importance of problem-scoping in middle and high school classroom design experiences expressly relates to the rationale for investigating the teaching and learning of design, specifically focusing on improving students' problem-scoping and, in turn, design outcome success.
Data presented by Becker et al. (2012) and Atman et al. (2007) indicated that high school students and novice designers (i.e., college engineering students) spent less time in the problem-scoping stage compared to expert engineers. Students spent about 40-50% less time defining the problem and gathering information than their expert counterparts. Overall, experts devoted about 24% of their time to problem-scoping, compared to about 18% for students (Atman et al., 2007;Becker et al., 2012). Despite the difference in time spent on problem-scoping, Atman et al. (2007) compared the design quality scores of senior college engineering students and expert engineers using the Mann-Whitney test and found no statistical difference. Although Atman (2007) did not compare design quality scores for freshmen engineering students and experts, the freshmen's design quality scores appeared to be significantly lower. Becker et al. (2012) did not compare the quality of students' designs with experts' designs either. However, Becker et al. (2012) noted that "without exception, a contractor would not be able to build" what the students designed due to "disorganized, messy, and incomplete" design documentation (p. 15). Both Becker et al. (2012) and Atman et al. (2007) made recommendations that teachers should focus on scaffolding design experiences for students, specifically during the problem-scoping stage. Dym et al. (2003) suggested that "we need to spend more time thinking about how we define the problem, rather than on the solution to a problem" (p. 106).

Background
The study was conducted alongside a 15-week, in-service teacher professional development (PD) program called Engineering for Educators (EfE) in Southern California in the United States. The EfE PD was designed around the Standards for Preparation and Professional Development for Teachers of Engineering (Farmer et al., 2014). The EfE PD addressed the following: Engineering Content and Practices; Pedagogical Content Knowledge for Teaching Engineering; Engineering as a Context for Teaching and Learning; Engineering Curriculum and Assessment; and Aligning Research, Standards, and Educational Practices (Farmer et al., 2014). The participants in the EfE PD consisted of middle and high school science and mathematics teachers.
During the first meeting of the EfE PD, all the teacher participants were asked if they would like to participate in this study. After the second EfE PD meeting, one of the researchers in this study and seven teacher volunteers met to discuss the study and the SCOPE process. The seven teachers had an average of 5 years of teaching experience and were all in the same cohort of a STEM master's program that one of the researchers in this study is acquainted with. Two of the teachers taught math, and five taught science. At this meeting, the goals were to give teachers the design materials, assign course sections to treatment and control groups, and train teachers to deliver the SCOPE process and the tower design challenge. The teachers were asked to implement the SCOPE process with only the treatment groups and implement the design challenge with both treatment and control groups.

SCOPE process
Participating teachers introduced the treatment group to the SCOPE process in a researcher-developed presentation before giving students the tower design challenge. The students in the treatment group were prompted to ask clarifying questions about the usage of the SCOPE process. While preparing the teachers to promote students' utilization of the SCOPE process, the researchers specifically addressed anticipated student questions. Students in the treatment group were asked to use the SCOPE process during the design challenge. The teachers encouraged students to think through the problem using suggested techniques and questions to foster thinking about the design challenge (see Table 2, Column 2). The SCOPE process inherently involves a focus on many of the underlying skills involved in design. Additionally, the SCOPE process prompts students to use various tools to record and analyze information (see Which approach seems like it would yield the best result(s)? What assumptions have you made? Select the approach that best seems to meet the criteria AND addresses the problem you identified.
Decision matrix T-chart Table 2, Column 3). The usage of tools to help manage information was not new to the teachers or students.

Tower Design Challenge
All participants were challenged to individually design and construct the tallest note card tower that would hold the most weight on top of it before failure. The design challenge further specified the following: (1) participants had a time limit of 20 minutes to design and build, (2) the tower must be selfsupporting during testing, (3) the materials used were assigned a cost, and (4) the individual with the lowest score using the equation provided would win. Small note cards (4 inches by 6 inches) cost 3 points each, and large note cards (5 inches by 8 inches) cost 5 points each. Each inch of tape costs 10 points. The score equation was: score = ((amount of tape in inches x 10) + (# of small note cards x 3) + (# of large note cards x 5) -(height of tower in inches) -(amount of weight held in pounds)).
The tower design challenge is an ill-structured, open-ended design problem; therefore, understanding the tower design challenge is more difficult than it might initially seem. The design challenge seemed to suggest that both the height of the tower and the weight supported were equally important, but they are not. Thoroughly scoping the tower design challenge involves interpreting the score equation to consider the tradeoffs between material usage, height, and weight supported.

Research Questions
1. To what extent does grade level, sex, ethnicity, academic ability, period within the school day, and teacher explain design solution success?
2. To what extent does SCOPE process instruction predict students' design solution success-as measured using a score equation-while controlling for grade level, sex, ethnicity, academic ability, school period, and teacher?

Methodology
The research design used in this study was a quasi-experimental design with treatment and control groups. Entire class sections were randomly assigned to a treatment or control condition. Each teacher had at least one treatment and control group. The average class size of the treatment group was 27 students, and there were 15 groups in the treatment group (n = 404). The average class size of the control group was 28 students, and there were 14 groups in the control group (n = 398).
The score generated from testing the towers served as the continuous dependent variable (i.e., criterion) in the study. The independent variable (i.e., predictor) was instruction in the SCOPE process and the scaffolding the teachers gave during the design challenge. Other variables controlled for during this study include participant's grade level, sex, ethnicity, academic ability, school period, and teacher. The academic ability variable stratified students based on academic ability by determining if they were enrolled in a math and science class below, at, or above the norm for students at their grade level. The teacher variable also represents schools and school districts because all seven teachers were in unique schools and districts in Southern California. The participants included middle and high school students.

Participants
Participants included 802 students: 45.4% self-identified as female, and 54.6% self-identified as male. The racial and ethnic breakdown was 7.6% Black, 5.9% Asian, 11.7% White, and 74.8% Latinx (see Table 3). The participants in the study all attended public schools in southern California.

Data Collection & Analysis
After the 20-minute time limit of the design challenge, the teachers measured and recorded the height and weight held by each student's tower design. The teachers asked the students to count the number of small and large note cards and measure the amount of tape used. The students were then asked to determine their score using the score equation. The teachers verified and recorded the number and size of note cards and the amount of tape used. All student designs were placed into their own individually sealed plastic bag and were given to the researchers. Initially, the researchers only deconstructed a few designs, but they noticed discrepancies between the student-and teacherreported data concerning the actual number and type of note cards used and the amount of tape used. The researchers deconstructed all 802 student designs, counting the number and type of note cards and measuring the amount of tape.
Other participant characteristic data, including participants' gender, period, academic ability, teacher, ethnicity, and grade level, were reported to the researchers by the teachers. The gender, teacher, and ethnicity characteristics are straightforward. However, period, academic ability, and grade level may require more explanation. Period was defined as the order of a class time slot in the school day for a regularly scheduled course session that participants had the teacher participating in this study. Academic ability was identified by the participants enrolled in math and science courses below, at, or above the norm for their grade level. Grade level was defined as participants being in Grades 6,7,9,10,11,or 12. All data was compiled into SPSS by treatment and control groups and later transferred to R for further analysis.

Results
Analysis of the participants' scores on the tower challenge indicated that students receiving the SCOPE treatment outperformed the control group on the design challenge (see Table 4). The participants in the treatment group built shorter towers that held more weight and ultimately scored lower as a result (on this design challenge, the goal was to achieve the lowest score possible). Note. A lower score is preferred in the design challenge. Height is measured in inches. Weight is measured in pounds.
The gender, period, academic ability, teacher, ethnicity, and grade level variables were included in the first model of the hierarchical multiple regression (see Table 5). Based on the R 2 value in Model 1, 2.1% of the variability in the design score is being accounted for by gender, period, level, teacher, ethnicity, and grade level. Model 1 was statistically significant, p = .009. However, based on the standardized beta coefficients (β) in Model 1, these independent control variables have a relatively weak effect on the design performance. The gender variable in Models 1 and 2 were statistically significant, p = .006 and p < .001, respectively. Young men performed better than young women in the control group, and young women performed better than young men in the treatment group. Note. Score, based on the scoring equation, is the criterion.
In Model 2, the SCOPE process treatment is included with the variables from Model 1. Including the SCOPE process treatment in Model 2 resulted in an increase in the predictive ability of the model. Based on the R 2 change value from Model 2, there was a 40.4% increase in the predictive capacity by adding the SCOPE process treatment. The R 2 value from Model 2 was statistically significant, p < .001. The standardized beta coefficient (β) for the SCOPE process treatment in Model 2 indicates a relatively stronger effect on the dependent variable (β = -.636). Additionally, the standardized beta coefficient (β) being negative for the treatment in Model 2 indicates that participants in the treatment group scored lower on the design challenge using the scoring equation; thus, they performed better. Using the R 2 value from Model 2 to calculate Cohen's f 2 , the effect size is .739. Using the pwr package in R to calculate power, with seven independent variables, 794 degrees of freedom, Cohen's f 2 is .739, and alpha level .005, power is rounded up in R to equal 1 (R Core Team, 2019).
In Model 2, academic ability was statistically significant, p = .019. In the control group, students enrolled in math and science courses below the norm for their grade level performed basically the same as students enrolled in normal math and science. However, in the control group, students enrolled in math and science courses above the norm for their grade level performed better on the design challenge than students enrolled in math and science courses normal for and below the norm for their grade level. In the treatment group, students enrolled in math and science courses above the norm for their grade level performed better than students in normal math and science courses, and students enrolled in normal math and science courses performed better than students in math and science courses below the norm for their grade level.

Discussion
The results of this study demonstrate that the SCOPE treatment was the most important predictor of a successful design solution for the tower problem. The results suggest that the SCOPE process improved teachers' instructional scaffolding of the design experience to promote students' problem-identification and their likelihood of having a successful solution. It is believed that cognitive and metacognitive scaffolds in the SCOPE process design instruction helped the students slow down, think, and more comprehensively engage in the design process. This finding is consistent with Roll et al.'s (2012) finding that metacognitive scaffolding increased the number and quality of methods that undergraduate physics students invented to describe the uncertainties in slopes. Aligning with the invention and productive failure literature, Roll et al. (2012) argued that the invention activity paired with metacognitive scaffolding exposed students to the challenges of the knowledge domains prior to giving students direct instruction and activated students' qualitative reasoning.
The 20-minute time limit for the design challenge may not have been sufficient time to enable students to adequately engage in the SCOPE process. Atman et al. (2005) suggested that the amount of time students spent working on the problem was an important factor for shorter design challenges such as the tower design challenge. The study design did not include a measure of the time students spent in scoping activities. However, teacher participants reported that students in the control group spent less than 1 minute scoping the problem, almost immediately starting to build a tower, and seemingly ignored the score equation in the tower design challenge. According to the teachers, the participants in the treatment group spent about 10 to 15 minutes scoping the problem transitioning through the SCOPE process. Atman et al. (2005) indicated that transition behavior was the more important factor in longer design challenges. Although not specifically measured, teachers reported that the SCOPE process appeared naturally iterative, seemingly promoting an increase in students' transition behavior. Teachers reported that students transitioned from recording ideas at one part of the SCOPE process to utilizing cognitive and metacognitive skillsets at another part and back to recording ideas based on continuous realization from prompting and tool usage. Römer et al. (2000) suggested that as a design problem becomes more cognitively taxing, the use of external prompts and tools, especially during the initial stages of problem-solving, will help support solution success. Future research should systematically compare the type and frequency of teacher prompts against the types and timing of SCOPEing activities in which students engage during the design challenge. In future work, the tools used by students to record ideas can be added to the evaluation to help improve and evaluate students' conceptual understanding of design. Potentially, these patterns could yield insights into best practices for teacher scaffolding.
In the current study, students enrolled in math and science courses above the norm for students in their grade level performed better than other students. Given that engineering design challenges require students to apply mathematical thinking and scientific reasoning during problem-scoping, students may benefit from prompts that focus upon mathematics and science crosscutting concepts and practices. Utilizing the score equation while scoping the problem requires mathematical modeling. Students enrolled in higher-level science and mathematics courses may have applied their domain-specific knowledge to make better design decisions related to the scoring equations (Shergadwala et al., 2018). Future research could examine the efficacy of SCOPE prompts and teacher metacognitive scaffolding on students' design quality and breadth of potential design considerations and constraints. Given the newness of the SCOPE process, the limitations of this study, and the sparsity of previous research on metacognitive scaffolding as it relates to enhancing engineering design outcome success in K-12 education, these findings are promising but deserve more extensive study.

Conclusion
Examination, formulation, and understanding of a design problemproblem-scoping-has been associated with the quality of design solutions, especially among novice designers (Atman et al., 2007). This tendency presents a pedagogical challenge for K-12 teachers who strive to employ engineering design challenges to meet state and national standards. Denson and Lammi (2014) emphasize the need for effective teacher professional development to address these challenges, particularly among high school teachers.
Hypothesizing that teacher's metacognitive scaffolding may improve the quality of design solutions, an instructional framework was developed to promote key cognitive and metacognitive processes during the problem-scoping phase of design. This SCOPE process employed a set of probing, domainindependent questions during the problem-scoping stage of design, including questions that promote thinking about studying (S) the problem, identifying constraints (C), organizing (O) information, predicting (P) potential outcomes, and evaluating (E) assumptions.
In this study, mathematics and science teachers received instruction in using the SCOPE process as part of a 15-week professional development program. Then, teachers taught their students the SCOPE process and diligently scaffolded the use of the SCOPE process during the tower design challenge. The results of hierarchical regression analysis demonstrated that the SCOPE treatment was the most important predictor of a successful design solution for the tower problem.
This study contributes insights into the professional development of Grade 6-12 teachers who strive to implement engineering design learning experiences in their classrooms. Results indicate that the SCOPE process is a worthy framework to guide teacher practice that, in turn, enhances student design process and quality of design solutions. Results are at the preliminary stage, but they are encouraging and align with published frameworks on engineering design experiences at the secondary level.