JTE v3n1 - The Development and Validation of a Test of Industrial Technological Literacy

The Development and Validation of a Test of Industrial Technological Literacy

Michael A. Hayden

It is often stated that technology affects every person on this planet, is growing exponentially, and is becoming increasingly complicated. With technology being such a force in our lives, it seems logical that we should be knowledgeable about it. This knowledge of technology has been labeled--technological literacy (TL). Several reports on education have addressed the TL issue and have all called for an increase in the level of TL exhibited by our students. Maley (1985) noted that, "in 1984 alone, ten major studies of education were reported...each one calling for changes in school to prepare our students to live in a technological society" (p. 16). Since 1984 an increasing amount of research concerning TL has been conducted.

A large number of educators and specialists in several disciplines are concerned about the level of TL displayed by the American public. They voice an opinion similar to Ley (1987), namely, "the level of technological literacy to which educators and others are able to bring the general population will determine the future world in which humankind will exist" (p. 7).

A set of very pertinent questions has recently been on the minds of many concerned leaders. These questions include:

1. As a public, how technologically literate are we?

2. How do we become technologically literate, or become more so?

To answer these questions we must first answer such questions as:

1. What is technological literacy?

2. Does it really exist, i.e., is it worthy of study by itself or is it part of something else?

3. Assuming TL exits, can we measure it with utility?

If TL is a viable concept and can be measured reliably, then studies can be conducted to determine the most advantageous educational environment for the attainment of such a characteristic. A test of TL with sufficient validity could be used to identify and diagnose those students in need of greater technological knowledge, as well as evaluate the effectiveness of programs teaching the construct.

Purpose of the Study

The purpose of the study was to investigate the validity of the construct of TL. The definition of TL will not be expounded upon here since it has been exhaustively covered by this and many other authors (see Hayden, 1989b). In this study the following definition of technological literacy was used: technological literacy is having the knowledge and ability to select, properly apply, then monitor and evaluate appropriate technology given the context.

Following are general questions the study investigated.

1. Does the empirical model (an instrument measuring TL) exhibit satisfactory psychometric properties?

2. Does the empirical model exhibit multiple dimensions?

3. How do the possible correlates (independent variables) of general achievement, grade level, gender, parental contact with technology, and prior industrial course exposure affect TL?

Theory of Technological Literacy

For the purpose of this study, technology was defined as: the process by which humans utilize available resources to extend their potential or alter their environment (Hayden, 1989b). Based on this definition it was theorized that technology is a learned phenomenon. Hence, being literate about technology should also be a learned phenomenon. Technological literacy is not skill in directly applying specific technology, i.e., technical literacy. Technological literacy for the purpose of this research was viewed as general knowledge, abilities and behaviors concerning technology. Furthermore, technological literacy was not viewed as being restricted to general skills in applying technology, i.e. technological operacy. Therefore, TL was assumed to be more closely related to achievement than aptitude.

Being related to general achievement, a TL instrument should perform psychometrically similar to other measures of general achievement. If TL is in part related to achievement, a person's standing on the attribute should increase with age due to developmental factors. Also, a person's score on a TL instrument should increase with additional learning and exposure related to the attribute. Additional exposure could come from several sources, e.g., parental/student interaction, school curriculum and hobbies.

Research Questions

To investigate the questions of the study, 29 hypotheses were tested, the majority of which related to eight dependent variables (total test and three subtests for male and female). A synopsis of the research questions follow.

1. Is the instrument unidimensional?

2. To what extent do the independent variables account for unique variance in instrument score.

3. Can the independent variables be used to predict scores on the instrument?

4. Does the instrument behave psychometrically in a similar fashion to other measures of general achievement?

5. Does the score on the instrument have a linear relationship with grade level?

6. Do males and females score differently?

7. Is there any interaction between gender and grade level on the instrument score?

Methodology

Instrumentation

The instrument was developed from a rigorous test plan which included a detailed rationale and definition of the domain including limitations, delimitations, the target population, uses and constraints of the test. A blueprint was created that detailed the format and scoring procedures. The test plan also included generation of items and revising algorithms and administration instructions.

Clearly, technological literacy is a broad term. Many authorities maintain that it encompasses operacy, i.e, what we can do with the technology, in addition to typical literacy concepts. Many (see Fleming, 1989) see technological literacy as a type of empowerment enabling us to do a very broad range of things. Additionally, general literacy concerning technology would cover all technology, e. g., agricultural, medical military, etc. Lastly, literacy about technology would encompass affective, cognitive and psychomotor domains.

The time and monetary limitations placed on this project necessitated a tightly focused test blueprint. The following delimitations and their accompanying rationale were used to focus the instrument.

1. Focus on the industrial strata because it was assumed that if industrial technological literacy was a valid concept then the more encompassing concept of TL was also valid; this strata was the researcher's area of expertise; prior research had most clearly explained this strata; and it was perceived that an instrument focused on industrial technology would have the most future utility for use in industrial programs.

2. Maintain a primary focus on the cognitive domain because it readily lends itself to item writing, test administration, scoring and data analysis techniques which are relatively unambiguous and can be used with a large number of items and subjects in a timely fashion.

3. Delimit the content of the items to general principles of technology or recent technological innovations/impacts because it was assumed that if an individual understood and was literate about these then they would know or could easily learn about other time periods.

The instrument was titled "Industrial Technological Knowledge" (ITK). Trained item writers developed and refined the items in the instrument. The writers were all faculty or doctoral students of the Industrial Education and Technology or Research and Evaluation Departments at Iowa State University. The mean number of years of industrial/consulting and teaching experience of the writers was nine and four years respectively. The items underwent several iterations of qualitative and statistical selection and revision. There were several item tryouts in addition to a pilot study.

Parental contact with technology was measured dichotomously for father and mother. Students responded to the question, "Does your parent through work, major home responsibilities or hobbies work with tools and/or machines?"

Industrial course exposure was measured by recording the number of semesters of industrial/technical/vocational (I/T/V) courses taken. This included the current semester.

General achievement was measured by the students' Iowa Test of Educational Development (ITED) subscores of Natural Science (NS), Quantitative (Q), Reading Total (RT) and Social Science (SS).

Data Analysis

Exhaustive analyses were conducted. Only the results will be summarized in this paper. For most analyses, cases with missing data were deleted case-wise, i.e., only subjects for whom complete information on the independent variables was obtained were used in the analyses. All correlational and t-tests were two-tailed and performed at the .05 level. Several post hoc analyses were conducted. Many of these were outside of the realm of the study and were of small sample size. However, they play a significant role in interpretation of the results and recommendations for future inquiry.

Primary analysis procedures included factor analysis to determine the dimensionality of the instrument and correlational, analysis of variance and regression procedures to investigate variance and predictability questions. The instrument's characteristics were compared with other instruments measuring general achievement to assess its psychometric properties. Visual inspection of plotted cell means and variable correlations were also used to assess statistical procedure assumptions such as linear relationships.

Descriptive statistics of the sample, the ITK scores, and the individual item characteristics were generated by a variety of techniques including one parameter logistic scaling, analysis of variance, and correlational methods.

Population and Sample

The population of this study was high school students in the state of Iowa. The sample was comprised of volunteering school buildings. Within the schools, intact classes were targeted for participation in the study. The classes selected were as heterogeneous as possible. Most classes selected were required courses. Approximately 25% of the subjects were from each of the grade levels 9 - 12. Within grade levels, males and females were represented approximately equally. Virtually all students had some I/T/V course exposure. Industrial Technological Knowledge test scores were collected from 826 subjects.

Interpretation of Results

Only those results most significant to interpretation and recommendations will be discussed. For detailed hypotheses and results of all statistical tests see the original study (Hayden, 1989a).

Psychometric Properties

Instrument Statistics.

The internal consistency reliability of the total test was .83. The internal consistency of the ITED subtests range from .86 to .96 (The University of Iowa, 1987). Application of the Spearman-Brown formula indicated that if the ITK instrument were the same length as the ITED composite instrument, the ITK's estimated reliability would be .975.

The average interitem correlation of the ITK was rather low, being only .10. Average item/total correlations were also small, being .28. It is possible that these correlations could be increased by revising the items and increasing sample size. If the average interitem correlation could be increased to .15 (the average interitem correlation of subtest #3), the reliability estimate for the 45 item instrument would be .89, uncorrected.

There was no floor or ceiling effect on the ITK instrument. The distribution of scores was nearly normal with an average item mean of .5. This allowed for maximum variance. However, due to the potential for guessing, a slightly easier test would have given more information about test takers.

The 2.94 standard error of measurement of the ITK instrument is proportionately large compared to the ITED composite. The standard error of measurement for the 361 item ITED composite is approximately 1.2. It is possible that further item revision could decrease the ITK's standard error of measurement. Table 1 summarizes the instrument's psychometric properties.

Table 1 Instrument Statistics - Male and Female Combined


Total test

H.S.

Number of items 45

Cronbach's alpha a .83

Minimum score 2

Maximum score 44

Mean 22.23

Median 23

Mode 24

Standard deviation 7.12

Standard error of measurement 2.94

Average item/total score correlation .28

Skewness of distribution -0.13

Kurtosis of distribution -0.55

Average inter-item correlation .10

Average item mean .50


a Calculated using uncorrected point-biserial correlation.

The data collected from the main phase of this study is equivalent to the main field testing phase of a research and development project. It is the view of this researcher that the ITK instrument exhibits satisfactory psychometric properties to be worthy of revision and advancement to the operational field testing stage of development.

Item Statistics.

Among items and between gender there was much fluctuation of item means. Females do better on some items, while males do better on others. These fluctuations are much more pronounced for items which are more difficult. Females tended to perform better on items requiring reading comprehension. There was no consistent pattern of item means between males and females except that males scored higher on most items.

Dimensionality.

Three factors or subtests were extracted from the ITK instrument. Factor #1 primarily contained items pertaining to the human adaptive systems. The two highest loading items on this factor and their loadings were:

#18 Why can computers lead to greater productivity in industry? Reduction of human Error, reduction of cost, greater speed, and increased accuracy. Loading of .81.

#29 The average worker has to be retrained because new methods of doing things Have been developed. Loading of .80.

Factor #2 primarily contained items pertaining to the application of other disciplines (most notably math and science) to the solution of technological problems. The two highest loading items on this factor and their loadings were:

#15 One of the most promising uses of microbes is for cleaning the environment. Loading of .69.

#31 Cooking in a microwave oven is done by exciting the atoms in the food. Loading of .64.

Factor #3 primarily contained items dealing with the interpretation of written and graphical material having a technological theme. The two highest loading items on this factor and their loadings were:

#37 Technology can best be defined as people using tools, resources, and processes to solve problems or extend their capabilities. .76.

#45 Consider this situation. The design, manufacturing, marketing, and management of product XX was performed by hundreds of individuals. After using product XX, many consumers were injured. In a latter court case, injured person John Doe testified that he thought product XX was unsafe even before he used it. Several employees that helped make and sell the product also testified that they had always thought XX was dangerous. The Judge asked the injured person why he went ahead and used the product. He also asked the employees why they went ahead and made and sold the product even though they thought it was unsafe. The situation presented above could be accounted for because individuals, when given directions by people in authority, often do things without thinking about the possible outcomes. Loading of .66.

Predicting Scores

It was found that the score on the ITK instrument could be predicted. Table 2 shows the regression equations for males and females. Table 3 further summarizes the prediction findings.

Table 2 Regression Equation and Summary Statistics for Prediction of ITK Total Score

***(Available as a graphic only)***

Table 3 Significantly Predicting Variables


Total #1 #2 #3

Grade M,F F F

Gender x a x x x

Father M M M

Mother

Father/mother combined M M

Indust./tech./vocat. course F F

Q F F

SS F

NS M,F M,F M M,F

RT M M

Q, SS, NS and RT combined M,F M,F M,F M,F


a Gender does not make a significant contribution to ITK prediction when used in conjunction with the other independent variables. However, gender affects the statistical result of almost every hypothesis tested.

Notes: M - indicates significant predictor for males.

F - indicates significant predictor for females.

Construct Validity Evidence

Grade level effects.

This author theorized that grade level, being closely related to cognitive development, should correlate positively with the construct. Grade level accounted for unique variance in ITK total score for males but not females. However, for either gender there is no evidence that score on the ITK instrument has a positive relationship with grade level for high school students.

A post hoc analysis was conducted to investigate grade level effects. By interview and observation, the amount of elapsed time since taking a course that fit the definition of Technology Education (TE) was ascertained. The following was discovered. It is required of virtually every student (male and female) in Iowa to take 1 or 2 semesters of technology type education in grades 7 or 8. These courses most closely fit the definition of TE. They are general in nature, do not focus on skill development and stress the impacts of technology. It was found that the more recent the student had taken a TE course the higher their score. This was after factoring out the effect of the other variables.

Gender effects.

Only for subscore #2 did gender account for unique variance in score. It was hypothesized that males would receive higher ITK total scores because of stereotypical male roles in society. While mean scores for males and females were not significantly different on the ITK total test, gender did play a significant role in testing hypotheses. Males and females performed very differently on most items. The total variance accounted for in ITK total score by the other independent variables was roughly two-thirds for females of what it was for males. For males the independent variables explain 52% of the variance in ITK total score; for females the figure is 35%.

For both sexes, but especially for females, there is much unexplained variance attributable to sources other than the variables investigated. The source of the unexplained variance could be attributable to various factors. However, from talking to students and observing instruction, it seems tenable that the student's level of literacy about industrial technology is related to specific course content learned and what the student reads, watches on television, etc. Other possible sources of variance include hobbies, part-time jobs, and interest/aptitude areas.

Parental effects.

It was hypothesized that parental contact with tools or machines should correlate positively with the construct. For males, father contact with tools or machines as perceived by the subject correlates positively with ITK total score. The unique variance accounted for in ITK score by this variable was attributable to subtest #1. The two highest loading items in subtest #1 concern working in industry. For females, father contact with tools or machines played no part in ITK score. Mother contact with tools or machines played no part in ITK score for males or females.

Course exposure effects.

Central to this writer's theory of TL and especially the industrial strata is that exposure to I/T/V courses should increase a person's level of the attribute. For males this was not found to be the case. However, for females such a positive relationship was exhibited. Two findings clarify this gender difference. First, the mean number of I/T/V courses taken by males was over three times that of females. Second, after one or two semesters of more general I/T/V courses (which are closest in content to the intended curriculum of TE) the additional courses taken are often more of a craft or vocational nature (which often deviate significantly from the intended content of TE). An examination of the plot between the semesters of I/T/V courses taken by males and their ITK score reveals a curvilinear relationship. When the ITK score is recoded into the categories of 0, 1, 2, and 3 or more semesters, the correlation is significantly positive. Based on course exposure effects, it is concluded that there is substantial support for the existence of industrial TL.

ITED subscore effects.

The proposed theory of TL states that the construct is a partial subset of general achievement. It was hypothesized that TL or more narrowly the industrial strata should intercorrelate and behave psychometrically in a similar manner to other subsets of general achievement. In this research project the other measures of general achievement were ITED subscores. The ITK instrument was found to intercorrelate in a similar fashion with ITED subtest intercorrelations. However, the ITK should not correlate too highly with other subsets of achievement if it is to be worthy of being partitioned from the encompassing set of attributes. Correcting for attenuation, the shared variance between ITK score and ITED subscores ranges from 36% to 56%. These percentages are large enough to infer a meaningful relationship between the latent trait being measured by the ITK and that being measured by the ITED. However, the shared variance is not so large as to negate the ITK instrument as an additional means of measuring general achievement. It is concluded that the ITK measures a part of general achievement, only part of which is also measrured by ITED subscores.

Summary

The results of the research give support for the existence of the construct of industrial technological literacy; and by extrapolation, technological literacy. Results also indicate that TL can be measured reliably. It is concluded that TL is a subset of the general achievement domain but is worthy of separate consideration. Table 4 summarizes the author's interpretation of the construct validity evidence.

Table 4 Strength of Construct Validity Evidence


Total #1 #2 #3

Grade 0,2a 0 0 0

Gender 1b 0 0 0

Father 2 2 0 0

Mother 0 0 0 0

Indus./tech./vocat. courses 2 1 2 1

Q 1 0 0 0

SS 1 0 0 0

NS 2 2 0 2

RT 1 0 0 0


a Substantial evidence when courses are recoded as to their TE content. b Gender has less of an affect when courses are recoded as to their TE content.

Notes: 0 = little or no construct validity support.

1 = partial construct validity support.

2 = substantial construct validity support.

Based on responses to items in the instrument it is this researcher's conclusion that there is widespread misunderstanding of technology on the part of many high school students. Based on observation, interviews, and the post hoc analyses it is this researcher's conclusion that the most appropriate content and pedagogy to increase TL would be those of TE.

Recommendations

Based on the results and the insights gained from conducting this research the following additional research projects are recommended.

1. Revision and operational field testing of the ITK instrument. The benefits of this would be improvement of psychometric properties and gathering additional validity evidence.

2. Replication of this research for cross validation purposes.

3. Investigation of strata other than industrial, e.g., agricultural, biomedical, etc.

4. Development of alternate forms of the ITK instrument so that alternate forms reliability can be calculated.

5. Development of other means of measuring TL and subsequent multi-method multi-trait studies.

6. Research aimed at investigating what type of content and/or methodology contributes to and/or increases TL most efficiently. There is much corroborating research (Baker, 1989; Croft, 1990; Hameed, 1988; Hatch, 1985) which defines and gives evidence for the validity and utility of measuring TL. This author believes this recommendation to be most important. It seems logical that before we try to increase a student's TL we first know the best way or ways to do so.


Michael A. Hayden is Assistant Professor, Department of Technology and Education, Mississippi State University, Mississippi State, MS.

References

Baker, D. (1989). Technological literacy: The essential criteria for a definition. Dissertation Abstracts International,

(University Microfilms No. 89-00392)

Croft, V. E. (1990). Technological Literacy: Characteristics of a High School Graduate. Paper presented at the International Technology Education Conference, Indianapolis, Indiana.

Fleming, R. (1989). Literacy for a technological age. Science Education, 73(4), 391-404.

Hameed, A. (1988). Development of a test of technological literacy. Dissertation Abstracts International, (University Microfilms No. 88-24509)

Hatch, L. O. (1985). Technological Literacy: a Secondary Analysis of National Assessment of Education Progress Science Data. Unpublished doctoral dissertation, University of Maryland, College Park.

Hayden, M. A. (1989a). The development and validation of a test of technological literacy. Dissertation Abstracts International, (University Microfilms No. 89-20136)

Hayden, M. A. (1989b). What is technological literacy? Bulletin Of Science, Technology And Society, 9(4), pp. 228-233.

Ley, C. (1987). Technological Literacy and Investigative Imperatives for vocational education researchers. Journal of Vocational Education Research, 12(1), 1-10.

Maley, D. (1985, October). Issues and Trends in Technology Education. Paper presented at the Technology Education Symposium VII, California, Pennsylvania.

The University of Iowa (1987). Manual for Teachers, Administrators and Counselors: Iowa Tests of Educational Development: Forms X-8 and Y-8. Iowa City: Author.

Permission is given to copy any article or graphic provided credit is given and the copies are not intended for sale.

Journal of Technology Education Volume 3, Number 1 Fall 1991