|
Executive Summary
I.INTRODUCTION
A. Purpose of the
Year Two Evaluation Report
B. Growth of the
Exemplary Faculty Project: Role of the Evaluation
II. YEAR TWO REVISIONS
TO THE PROJECT EVALUATION PLAN
A. Purpose of the
Evaluation: Responding to Project Priorities
B. Exemplary Faculty
Goals and Objectives and Outcomes
C. Hypotheses
D. Evaluation Design
E. Methodology
F. Enhanced Formative
Assessment of Project Activities
III. YEAR TWO EVALUATION ACTIVITIES
A. Revisions to the Year One Survey 12
B. Administering the Year Two Survey 12
C. Year Two Data Analysis: Comparisons Among Groups and Years 12
D. Formative Evaluation: Feedback to ATE Management Team 13
IV. PLANS FOR YEAR THREE EVALUATION
A. Response to GPRA 13
B. Revisions to the Year Three Survey 15
C. Year Three Analysis and Dissemination of Exemplary Faculty Data 16
D. Incorporating Exemplary Faculty Evaluation into SC ATE Center Evaluation
16
V. YEAR TWO RESULTS
A. Survey Results: Exemplary Faculty and Comparison Faculty 16
1. Sample and Methodology 16
2. Faculty Characteristics 17
3. Attitudes and Beliefs about ATE and Program Components 19
4. Use of Technology Outside the Classroom 27
5. Use of Technology Inside the Classroom 31
6. Business and Industry 34
7. Professional Activities and Pedagogy 36
8. Personal and Professional Goals, Objectives, and Planned Activities 44
VI. CONCLUSIONS AND DISCUSSION
A. Overall Findings 49
B. Assessment of Hypotheses 49
C. Recommendations 56
APPENDIX
A. Data Graphics A - 1
B. Year Two Exemplary Faculty and Control Faculty Surveys B - 1
I. INTRODUCTION
The South Carolina Advanced Technology
Education (SC ATE) Exemplary Faculty Project completed its second year in
September 1997. This report documents the project's evaluation, which is
being conducted by the Academy for Educational Development (AED). The report
addresses evaluation design, methods, findings, and conclusions from the
first two years. It also relates them to the larger SC ATE Center of Excellence,
within which the Exemplary Faculty Project has operated since September
1996.
A. Purpose of
the Year Two Evaluation Report
The study's overall purpose is to assess
changes in faculty attitudes and behaviors as a result of participation
in the Exemplary Faculty Project. The main hypothesis is that project participation
leads to several substantive and beneficial changes among faculty: More
positive attitudes and beliefs toward SC ATE objectives, greater use of
educational technology and other instructional tools, increased business
and industry involvement in teaching and learning, enhanced faculty development
and professional activities, and increased use of innovative pedagogical
techniques. Together, these changes help to transform the technical college
learning environment, promoting improved academic and occupational outcomes
among engineering technology (ET) students.
This report documents interim findings
based on two full years of project activity, from September 1995 through
August 1997. All faculty who have participated from inception ("cadre
faculty"), and those who have joined in year two, have responded to
an extensive questionnaire addressing project-related attitudes, beliefs,
professional practices, and personal and professional plans. At this stage,
analysis of exemplary faculty data is based on: a) comparison of responses
to the first year faculty survey with responses to the second year follow-up
surveys, and b) comparison of exemplary faculty responses to those of comparison
group faculty. In the study's final report, similar analyses will compare
change in exemplary faculty responses across all three project years and
exemplary faculty responses to those of comparison group faculty.
B. Growth of
the Exemplary Faculty Project: Role of the Evaluation
The Exemplary Faculty Project has provided professional development opportunities
for participating faculty at all 16 South Carolina technical colleges.
AED's evaluation has grown along with the
project. During the project's second year, evaluation activities expanded
to include several new formative and summative components, including:
· Surveys of participants after
project chautauquas to assess what they learned;
· Addition of survey items to measure faculty involvement with, and
participation by, business and industry in project activities;
· AED technical assistance to the SC ATE management team in using
evaluation findings; and
· Plans for Exemplary Faculty Project evaluation elements to respond
to NSF's Government Performance and Results Act (GPRA) requirements.
These components are further described
in relation to the overall evaluation design in Section III.
II. YEAR TWO
REVISIONS TO THE PROJECT EVALUATION PLAN
A. Purpose of the Evaluation: Responding to Project Priorities
The evaluation is designed to assess the
effectiveness of the Exemplary Faculty Project's professional development
activities in promoting the instructional, organizational, discipline, and
career development of the South Carolina technical college faculty. The
evaluation's three specific objectives are to:
· Design and implement a formative
evaluation for continuous project improvement;
· Conduct a summative evaluation of program outcomes at the Technical
College system (e.g., programs in advanced engineering technology) and faculty
acquisition and use of relevant skills (e.g., application of selected SCANS
competencies); and
· Design an evaluation system for long-term use in South Carolina
faculty development programming and by replicators nationwide.
Additional evaluation objectives, which
will be achieved through accomplishment of the objectives listed above,
are to:
· Identify organizational features
of the exemplary faculty project that are most effective and transferable
to NSF projects nationwide;
· Evaluate the extent to which successful techniques and practices
from other workforce development programs (e.g., Tech Prep) are effective
in the exemplary faculty project;
· Determine which professional development approaches, such as the
use of peer mentors, best promote faculty development and success;
· Assess the extent to which enhanced computer skills and use in
professional activities increases productive faculty interaction with the
wider academic and business communities;
· Identify linkages established between technical college faculty
and secondary schools, senior institutions, and business and industry;
· Evaluate effectiveness of innovative career guidance activities;
and
· Track changes in the format and flexibility of engineering technology
curriculum and accommodation of students with non-traditional backgrounds,
different learning styles, and diverse academic goals.
B. Exemplary
Faculty Goals and Objectives and Outcomes
As one of three main components of the SC ATE Center of Excellence, the
Exemplary Faculty Project is designed to encourage inter-disciplinary teaching
strategies, the use of relevant technological applications in the development
and the delivery of instruction, and ongoing dialogue and collaboration
with industry. Thus the project seeks to improve technical college faculty
members' ability to identify, define, develop and evaluate the skills that
engineering technology program graduates will need.
Exemplary Faculty Project leaders asked
each college in the State technical college system to form interdisciplinary
teams of exemplary faculty from the colleges' departments of mathematics,
science, communications, and technology. First, the project introduces exemplary
faculty to and instructs them in new classroom methodological and technological
applications. Second, the project assists the cadre of trained faculty leaders
at each college to serve as mentors, training and supporting their faculty
colleagues in the innovations they learned.
By creating a new team-based instructional
design approach capable of implementing the interdisciplinary, problem-centered
core curriculum (scheduled for pilot testing in the Fall of 1998), the Exemplary
Faculty project seeks to enable faculty to model the instructional and professional
practices needed to promote a transformed learning environment. By achieving
this goal, faculty will be able to help students enter the workforce with
upgraded skills in science, math, physics, engineering, technology, and
communications, and the ability to work with others to integrate and apply
knowledge of these disciplines in thinking, problem-solving, and decision-making
on the job.
In its first year, the SC ATE management
team concentrated on encouraging and empowering faculty to communicate and
collaborate with their peers across traditional departmental and institutional
boundaries. Along with their business and industry counterparts, faculty
members were encouraged to rethink their teaching methodologies and to design
new, active learning approaches and environments. These activities are intended
to lead to new work processes, such as team-based development (and possibly
delivery) of instructional materials, and ultimately new work products,
including a more relevant and effective engineering technology curriculum.
In its second year, the project expanded
to include greater faculty interaction with industry (e.g., through faculty
team return-to-industry projects), instructional technology, and student
assessment activities. The project also became integrated with the new SC
ATE Center of Excellence as one of the Center's three program components,
faculty development joining the Center's curriculum development and program
improvement foci.
Through the project, faculty are learning
to use a new statewide electronic communications system, computer networking,
and classroom and distance learning technologies as well as innovative teaching
and learning strategies. By equipping faculty with modern infrastructure
and new tools for instructional development and delivery, and providing
training and technical support (with State and institutional support), the
project seeks to produce two outcomes. First, to facilitate communication
and cross-disciplinary interaction among faculty of different departments
and different institutions, interdisciplinary faculty teams are experimenting
with new forms of organization and interaction in researching, planning,
developing, delivering and evaluating instruction and student support services.
Second, faculty members are pooling expertise, resources, and products (e.g.,
course materials) from across the State technical college system and from
their industry partners. In turn, these can be shared by all colleges in
the system through the new communications infrastructure.
This new collaborative dynamic promises
to help faculty to reflect critically on the role of their discipline, its
relationship to the curriculum as a whole, and the relevance of instruction
to the workplace. In learning to work together with new colleagues to develop
new teaching techniques and materials, it is hoped that faculty will be
stimulated to transform the learning environment to one that simulates the
work environment and promises to foster student-centered learning.
C. Hypotheses
The evaluation started from a set of broad
expectations of faculty change in attitudes and practices, and professional
development not only in instruction but also in professional collaboration,
communication, and teamwork. A set of broad categorical questions were associated
with each hypothesis, which we used as the basis for specific evaluation
questions used to design the faculty surveys.
l. Faculty will adopt outcomes-oriented
pedagogy.
Did faculty collectively define desired
student learning outcomes, and design curricula with the academic integrity
to produce those learning outcomes? Did these curricula include an accountability
system for measuring progress toward the desired competencies and standards?
Did the integrated Engineering Technology
core curriculum incorporate upgraded academic and occupational competencies
and standards? Were interdisciplinary, transferable skills (viz.: writing,
critical thinking, problem solving, information literacy, computer literacy,
multi-cultural perspective, quantitative analysis, oral communication, etc.),
the National Research Council's national science education standards, SCANS
workplace competencies, ABET Perkins performance standards, and national
voluntary occupational skills standards incorporated?
2. Reformed pedagogy will be based on best
available theories of learning and on sensitivity to variations in cultural
and academic backgrounds, and student learning styles.
Did faculty adopt active, hands-on, experiential,
applied, contextual, and collaborative learning approaches (stressing learning
experiences that relate theory to real world applications and encourage
students to construct their own knowledge through independent investigation
and use of tools, simulations and projects)? Did these approaches increase
students' mastery of academic concepts and motivation to learn?
3. Student-centered learning environments
will become the norm.
Were classrooms, use of class time, course
assignments and examination questions, and student-instructor and student-student
interactions rethought to orient teaching and learning to student's active
investigation, development of critical thinking habits, and career development
needs? For example, did students undertake individual and team projects
that develop critical thinking habits, but which also build a student portfolio
of products which might have more meaning for employers than grades?
4. Advanced assessment and instructional
technologies will be used to implement the new pedagogy.
Did faculty experiment any of with the
following:
· Interactive, exploratory teaching
and learning (using multimedia simulations, modeling and iterative drafting)?
· Outside-the-classroom academic support and reinforcement (tutorials,
etc.) using computer laboratories and instructional support systems?
· Access to up-to-date information and direct contact with experts
using communication and data search and retrieval technologies?
· New faculty roles such as student learning coach and project/courseware
developer?
Did faculty use of the statewide or college
telecommunications network and productivity tools (e.g., word processing,
spread sheet, data base, and graphics/presentation programs, electronic
grading and test generation/correction/analysis packages, etc.) improve
productivity and quality with respect to instructional management and delivery,
academic advisement, and department administration?
Did faculty adopt computer-adaptive testing,
computer laboratories and instructional support systems, interactive multimedia,
two-way audio and video teleconferencing, applications and productivity
software, or data search and retrieval technologies?
Did faculty use of new assessment and instructional
technologies help to inspire students to become independent learners? For
example, did students search for, locate, evaluate, synthesize and apply
information to forge new meaning, solve problems, or analyze issues?
5. The learning environment will extend beyond the classroom to the workplace
and the community.
Did faculty adopt distance education delivery
methods and seek to integrate classroom-based academic and work-based occupational
learning so that students' academic and work lives can connect to larger
social and economic concerns? For example, through academic year internships
and apprenticeships for students, and summer industry internships for faculty,
did students and faculty develop understanding of careers and the functional,
occupational and intercultural competencies necessary for today's diverse
and global workplace?
Did faculty use the Internet and a State
systemwide electronic communications system to communicate across traditional
institutional and disciplinary lines to share and gain ideas, pool expertise
and resources, and collaborate on course/module/materials development projects?
D. Evaluation
Design
The overall SC ATE Center evaluation design was revised in April 1997 in
response to input from the SC ATE National Visiting Committee (NVC) and
NSF's GPRA requirements. These included several modifications relevant to
the Exemplary Faculty Project, such as:
· Identifying and conducting focus
groups with key technical college stakeholders in order to understand the
conditions for adoption and institutionalization of reforms;
· Developing program and research plans in support of an organizational
change initiative, led by technical college chief instructional officers,
to promote adoption of reforms;
· Assisting in designing a curriculum development methodology;
· Offering technical assistance to college staff in conducting evaluation
research; and
· Developing GPRA measures of SC ATE impact on numbers of beneficiaries
based on a taxonomy of (1) types of beneficiaries, and (2) level of beneficiary
involvement with SC ATE.
With these modifications incorporated into
the design, The Exemplary Faculty Project evaluation now has the following
main components:
1. Formative Evaluation including:
· Observation and documentation of project activities;
· Evaluation technical assistance to the SC ATE management team;
and
· Faculty and system case studies.
2. Summative Evaluation including:
· Collection of longitudinal data
on faculty attitudes, beliefs, practices, and future plans;
· Tracking of State outcome data for all faculty in each of the major
areas of project activity; and
· Analysis and reporting of outcome data.
3. Faculty Development Evaluation System
including:
· Development of measures and instruments
for use in ongoing program assessment beyond the NSF funding period;
· Evaluation database available for use in continuous faculty development
program improvement; and
· Tested faculty development evaluation methodology for use by onsite
staff and for dissemination and replication in South Carolina and nationwide.
The evaluation is based on a quasi-experimental
design with both qualitative and quantitative elements. Evaluation activities
are designed to address each of the major areas of exemplary faculty project
activities, and to be readily adaptable to the evaluation of SC ATE National
Center activities when those become active in the Fall of 1996.
The formative evaluation monitors proposed
program services as they are delivered to determine whether the project
has been implemented as planned and enable institutional project staff to
improve the project plan, specific activities, communication, or project
management as needed. It includes a profile of the participants, a summary
of the "intervention" (e.g., new curricula, changed instructional
methods, inclusion of business and the community in institutional planning,
etc.) provided, characteristics and workloads of the faculty, staff, and
student participants, and opinions of their experiences in the project.
The results of formative evaluation serve two purposes: (a) regular feedback
of project-related data to the project management for continuous improvement,
and (b) potential replication or modeling of the project at other institutions
or states across the nation.
AED believes that formative evaluation
should include technical assistance, or "technology transfer."
Thus AED evaluators provide Center and institutional faculty and staff with
project design and evaluation expertise to enable internal institutional
evaluation and continuous improvement after ATE funding ends.
AED provides technical college faculty
and staff with materials, techniques, and resource information useful in
conducting evaluation research on campus. AED also advises staff on issues
concerning any ongoing evaluations they may be conducting. Support includes
such areas as:
· Conceptualizing measurable evaluation
goals and objectives;
· Writing project-specific evaluation questions;
· Developing broad indicators of program effectiveness and impact
that are responsive to the evaluation questions;
· Creating a data collection plan;
· Designing survey instruments, interview protocols, and other data
collection materials;
· Administering instruments and collecting data;
· Analyzing data;
· Interpreting conclusions; and
· Presenting findings in the form of reports, publications, and presentations.
AED also provides ongoing consultation
to the SC ATE management team in the design and development of the Center.
Evaluators Paul Bucci and Douglas Evans participate in weekly management
team conference calls, and periodic teleconferences on special topics. Drs.
Bucci and Evans also develop small-scale evaluation materials, such as chautauqua
comment forms for completion by chautauqua participants, and assist in large-scale
projects such as designing focus group research to collect project planning
information from Center stakeholders.
The summative evaluation determines the
extent to which anticipated and unanticipated outcomes of the Center's systemic
reform activities are attained. Both quantitative (survey research and analysis
of institutional and state-level data) methods and qualitative (case study)
methods are used to measure outcomes.
The implementation of SC ATE system reforms
is occuring in phases during which curriculum, faculty development, and
overall program reforms are instituted incrementally in the South Carolina's
16 technical colleges over time. This presents an opportunity to develop
naturally occurring comparison groups of implementing, partially implementing,
and non-implementing institutions. To capture this opportunity, the evaluation
employs a non-equivalent control group design for its formal studies in
each of the three project components. In this form of quasi-experimental
design, participants are not randomly assigned to intervention conditions.
Instead, naturally occurring experimental group (implementing) and control
group (partial and non-implementing) colleges are compared. This yields
a three-group comparison of participant and stakeholder change at annual
data collection points over the Center's five-year funding period.
In each component area, the group under
study-cadre faculty in the faculty development component, and student cohorts
in the program improvement and curriculum components-are tracked from study
year one through the conclusion of the Center funding period. The current
Exemplary Faculty Project will end in August 1998, at which time the three-year
cadre faculty study will be completed and data collection on Center faculty
development will be integrated into the overall SC ATE Center evaluation
(as described in the April 1997 Center Evaluation Plan).
E. Methodology
The Exemplary Faculty Project evaluation's
two main data collection instruments are: 1) an annual faculty survey and
2) a case study protocol. The annual faculty survey is administered in the
fall of each study year (1995, 1996, and 1997), while interviews and secondary
data are gathered in the winter and summer.
· Annual Faculty Surveys: An annual
survey of faculty knowledge, attitudes, and practices administered to participating
exemplary faculty and a matched group of peer faculty at the 16 technical
colleges. The survey covers pedagogy, technology, and faculty goals and
professional plans. There are exemplary faculty and comparison versions
of the survey.
· Exemplary Faculty Case Study Protocol: A structured interview protocol
used in the semi-annual interviews of eight exemplary faculty. It contains
items on faculty perception of project activities, attitudes toward SCATE
reform on their campuses, and use of what faculty have learned in the project.
In addition to these formal instruments,
evaluators also conduct secondary analysis of state data to determine large-scale
changes in institutional, program, and student indicators. Such measures
are also used extensively in the full Center evaluation. Finally, formative
evaluation techniques such as focus groups and satisfaction surveys are
used as tools for assessing current and planning future project activities.
· Analysis of Data Extracts from
the State Tech Data System: Annual analysis of institutional, ET program,
and student data to determine change in enrollment, completion, and related
patterns relevant to SCATE objectives.
· Formative Evaluation Techniques: Focus groups to gather stakeholder
input on project plans and areas of inquiry, quick-turnaround questionnaires
and interviews to measure participant interest and satisfaction with current
and future project activities.
Section V presents case study findings,
and focuses extensively on annual faculty survey results. The faculty survey,
and its companion control survey, contain a core of items included in each
year's survey in order to measure changes in participant and control faculty
attitudes, beliefs, practices, and plans over time. Other questions were
included on only one of the surveys. Table II-1 shows changes to the instrument
over time:
Table II-1: Annual Faculty
and Control Surveys By Study Year
|
| |
1995 (Year 1)
|
1996 (Year 2)
|
1997 (Year 3)
|
Survey Sections
|
Faculty
|
Control
|
Faculty
|
Control
|
Faculty
|
Control
|
Demographics
|
X |
X |
X |
X |
X |
X |
Exemplary Faculty Selection Process
|
X |
|
|
|
|
|
Attitudes about ATE program components
|
X |
|
X |
|
X |
|
Use of technology outside classroom
|
X |
X |
X |
X |
X |
X |
Use of technology inside classroom
|
X |
X |
X |
X |
X |
X |
Business and Industry Involvement
|
|
|
X |
X |
X |
X |
Professional activities and pedagogy
|
X |
X |
X |
X |
X |
X |
Professional goals and objectives
|
X |
X |
X |
X |
X |
X |
ATE Student Impact
|
|
|
|
|
X |
X |
The cadre faculty are already highly effective
educators whose role in the project is to become exemplars and lead instructors
in South Carolina's advanced engineering technology programs. Thus the surveys
are designed to determine if project activities enhanced faculty members'
application of such SCANS-related areas as problem solving, team building,
and technology use, and whether they integrated such techniques with professional
development and student-centered pedagogy. Additionally, the surveys ask
faculty to describe themselves professionally, to look forward down their
own career path, and to consider how participation in the project may affect
their future plans.
Did faculty enhance their ability to teach SCANS skills and competencies
to their students? Did they incorporate them in course content for student
benefit? Did they apply these skills themselves in their courses (e.g.,
through team-teaching techniques)? Which activities were most responsible
for this development? Which were least responsible? Answers to these questions
provide mid-program information on which skills are being acquired, used,
and to which activities their development can be attributed. Such information
assists project staff in deciding which activities are effective and which
need mid-course correction.
The surveys are supported by detailed case studies of randomly selected
participants who are interviewed to obtain self-reports on perceived program
effectiveness. Results of the case studies are primarily used for project
planning and to develop new activities.
F. Enhanced
Formative Assessment of Project Activities
In year one, as the evaluation was first being developed, AED formative
evaluation activities consisted primarily of case study interviews, observation,
and informal feedback to the SC ATE management team. In year two, the scope
of formative assessment expanded to include more formal measurements in
the following areas:
· Project activity surveys;
· Periodic participation in teleconferences for project planning;
and
· Focus group research.
The weekly planning sessions are conference
calls involving each member of the SC ATE management team. In January 1997,
AED evaluators Drs. Bucci and Evans began regular participation and provided
an evaluation perspective to ongoing project planning. Such input had been
previously provided, but with constant involvement in planning discussions,
the evaluators have been able to apply evaluation findings to guide management
decisions.
Starting in year two, chautauquas concluded
with a survey asking participants what the meeting had been designed to
accomplish, what they had learned, and their satisfaction with activities.
The evaluators worked with Ms. Lynn Mack, Co-PI for Faculty Development,
to develop, administer, and analyze results from these surveys. Findings
were used to modify the delivery of future chautauquas and to develop new
faculty development activities to meet participant needs.
As a result of the July 14 teleconference
on program improvement, Dr. Evans assisted the management team in developing
a quick-turnaround focus group research project. The aim of this research
was to determine technical college middle manager's knowledge, attitudes,
and problems they perceive to be facing the SC ATE initiative on their campuses.
Ultimately, the results of the research would be used to develop a plan
for encouraging middle management buy-in to SC ATE as a necessary step in
the process of integrating and sustaining the project over the long-term.
Dr. Evans developed the preliminary focus
group research design, and assisted Ms. Helen Edens, SC ATE curriculum specialist,
in developing a sampling plan and focus group sample. The following summarizes
the research design:
Design:
· Eligible participants are "middle managers" from the
16 technical colleges (e.g., CIOs, CSSOs, VPs, other administrators);
· Segment the middle manager population into "general education"
(GE) and "engineering technology" (ET) representatives;
· Draw a random sample of 16 GE and 16 ET managers (1 GE and 1 ET
from each college);
· Create 4 focus groups with 4 GE and 4 ET managers each, ensuring
no more than 1 manager from each college is in each group;
· To the extent possible, ensure that members of each focus group
are not very familiar with each other (i.e., colleagues who work closely
together should be separated); and
· Consider offering an "inducement" to attend the session
(e.g., a small gift).
Focus Group Sessions:
· Hold 1 session of approximately 2 hours with each focus group;
· Use a facilitator experienced in focus group research for each
session;
· Have a note-taker enter the discussion into a word processing file,
ensuring that all names are kept confidential (e.g., identify participants
in the transcript as #1, #2, etc., not by name); and
· Consider having 1 member of the management team present at each
meeting (consult with the facilitator before doing this).
Questions:
The following questions were used as guidelines. Focus groups were allowed
to digress and otherwise follow subtopics of interest.
1. Impact of SC ATE on production of engineering
technicians:
a. What is the extent of the need for engineering technicians in your service
area?
b. How well do you think your college is meeting this need?
c. What do you know about SC ATE that might have an impact on this?
2. SC ATE is directed at change in three
areas: Faculty development, curriculum reform, and program improvement (recruitment
and retention). In your opinion, how will SC ATE impact your campus in the
area of:
a. faculty development
b. curriculum reform
c. recruitment and retention
3. As the SC ATE project is implemented
on your campus:
a. What do see as: (i) the driving forces for implementation?
(ii) barriers to implementation?
b. How would you address the barriers?
In all, 29 middle managers participated
in the focus group sessions. Results are currently being employed in overall
Center program planning, especially in connection with an ongoing CIO-led
organizational change initiative.
III. YEAR TWO EVALUATION ACTIVITIES
A. Revisions to the Year One Survey
The first year faculty survey contained
a section entitled "Exemplary Faculty Selection Process" that
focused on faculty attitudes toward the process of identifying and selecting
project participants. Since these items were only designed to measure the
initial selection process, they were deleted from the year two survey. The
sequence of questions before and after this section remained unchanged in
year two.
Additionally, a new section entitled "Business and Industry" was
added to the year two faculty and control surveys. Questions focused on
faculty involvement with business and industry, plans for incorporating
work-based or "real world" lessons into courses, and perceived
impact of various kinds of business and industry involvement on students
and the educational process.
B. Administering the Year Two Survey
The second annual faculty survey was administered
in person at the fall chautauqua in October 1996. A total of 77 participating
faculty completed the survey. The control faculty survey was administered
by interoffice mail in October and November 1996. SC ATE Co-PI for Faculty
Development Lynn Mack sent the survey forms to each campus Chief Instructional
Officer (CIO) by interoffice mail for distribution to control faculty. The
CIOs collected the completed forms and returned them to Ms. Mack, who forwarded
them for AED for analysis.
C. Year Two Data Analysis: Comparisons Among Groups and Years
The first year survey data was taken as
the baseline to understand initial characteristics of the exemplary faculty
and to compare the faculty and control groups. Results were reported in
the first year evaluation report, dated March 1997. No significant differences
were found between the two groups, permitting comparison in year two.
In year two, survey data were used to compare
faculty change between the first and second project years, and between the
faculty and control groups. The paired samples t-test procedure was used
to compare the means from year one and year two for the exemplary group
and to compare the means from year one and year two for the control group.
This test reveals if there is a statistically significant difference between
the means of the same variable (in this case a response to a survey question)
before and after SC ATE implementation.
In order to compare the year two survey
responses of exemplary and control faculty, an independent samples t-test
was conducted. The independent samples t-test compares the means of a given
question from two separate and independent populations, in this case the
exemplary and control faculty. Results of this analysis are presented in
Section V.
D. Formative Evaluation: Feedback to ATE
Management Team
AED provides feedback to the ATE management team in a variety of formats,
including regular memoranda, informal feedback via e-mail, phone conversations,
and formal reports. During the project's second year, AED provided the following
specific input to the management team:
· Attended, observed, and provided
written and oral feedback on chautauquas in October 1996, February, and
April 1997;
· Participated in teleconferences on special project topics in July
and August 1997;
· Developed a middle manager focus group research plan in July 1997
(described above);
· Submitted NSF REPP program proposal (May 1997) to conduct a study
that:
Þ Identified the reform issues and progress results information that
key gatekeepers may find relevant;
Þ Designed and validated sample data collection, evaluation and presentation
tools tailored to build the early support of various gatekeepers; and
· Participated in weekly ATE management conference calls and carried
out specific assignments as a result of those discussions.
IV. PLANS FOR YEAR THREE EVALUATION
A. Response to Government Performance and Results Act (GPRA)
As a result of the congressional passage
of the Government Performance and Results Act (GPRA), beginning in FY 1999,
National Science Foundation budget requests must be outcomes-based. Focusing
on the number of people reached and program effectiveness, this new funding
structure reinforces the need for evaluations like the Exemplary Faculty
Study. Both the Year One and Year Two reports provide a strong foundation
for addressing the extent to which SC ATE is fulfilling the NSF goal of
providing, "excellence in science, mathematics, engineering and technology
education at all levels." With the Year Three survey, AED evaluators
will broaden SC ATE research efforts to understand some of the long-term,
systemic, and community wide effects of the Exemplary Faculty Project.
The Third Year Exemplary Faculty survey
includes a new section, "VIII. Outreach and Impact," to better
address these outcome measures. This section is designed to measure SC ATE's
effectiveness in not only improving the practices of participating faculty
and higher education institutions, but also increasing student achievement
and academic outcomes, enhancing the educational effectiveness of secondary
schools, and strengthening the role of community organizations and business
and industry. To do this, exemplary faculty are asked the following questions:
1. Since you first became involved in the
SC ATE project, with how many faculty members have you done the following?
(estimate the number next to each item below)
a. Trained in a new technique ____
b. Disseminated professional information to ____
c. Communicated with about the SC ATE project ____
d. Collaborated with on any kind of project ____
2. Please estimate the number of partnerships your institution has established
with the following types of organizations since the SC ATE project began
in September 1995 (estimate the number next to each item below)
a. Business and industry ____
b. Four-year higher education institutions ____
c. School systems ____
d. Community organizations (e.g., social services, churches) ____
e. Other (please specify and give number) _________________________
3. To what extent do you believe SC ATE has increased the educational effectiveness
of South Carolina's secondary schools? (Check only one response)
a. ( ) Not at all
b. ( ) A little bit
c. ( ) Somewhat
d. ( ) Quite a bit
e. ( ) A great deal
4. Have you, other SC ATE exemplary faculty
a your institution, or faculty with whom you or other SC ATE faculty have
worked produced materials for use in the classroom? If so, how have they
been disseminated or published? (Check all that apply)
a. ( ) No materials have been produced
b. ( ) In print
c. ( ) By word of mouth
d. ( ) By Internet or World Wide Web
e. ( ) By faculty-to-faculty information sharing
f. ( ) By computer-based media (e.g., diskette or CD-ROM)
g. ( ) Don't know
5. To what extent do you believe SC ATE
has improved student achievement and academic outcomes at your institution?
(Check only one response)
a. ( ) Not at all
b. ( ) A little bit
c. ( ) Somewhat
d. ( ) Quite a bit
e. ( ) A great deal
The Exemplary Faculty Project evaluation
has focused mainly on faculty development, rather than on the effects of
faculty development on students, institutions, and the technical college
system. We have examined attitudes, use of reform pedagogy, professional
use of technology, and professional development. The study has provided
in-depth answers to questions in the area of impact (effectiveness), as
defined in the NSF ATE program July 25 memorandum on ATE GPRA evaluation.
In the area of impact (numbers), we have
also made significant progress. After the ATE Center award to South Carolina
in 1996, one of the first evaluation tasks was to gain access to data from
the State Tech data system. State Tech data tracks student demographics,
enrollment, retention, and other major variables by area of academic concentration
(clusters). We have obtained access to data from 1992-1996, the four academic
years preceding the Center, and will use these as a baseline against which
to compare system-wide changes from 1996-2001.
The July 25 NSF memorandum also raises
several important evaluation questions in the impact (numbers) area for
which SC ATE has not collected significant data. The following questions
from the memorandum concern the numbers of organizations, people, and processes
affected by ATE Center reforms. They are followed by our planned or ongoing
response. Results will be reported in the Final Exemplary Faculty Project
Evaluation Report.
How many faculty and teachers have you
reached over the course of your project? We will collect data on faculty
and teachers through new items on the exemplary faculty survey as part of
our faculty development study. These will address direct participants by
self-report and others by report on interactions with them (e.g., another
teacher mentored in a particular pedagogy would be considered a secondary
participant).
How many partnerships have been established?
Partnerships will be measured through interviews with key staff at participating
technical colleges. Likely respondents include the ATE liaisons, Chief Instructional
Officers (CIOs), and ad hoc team members. A structured interview protocol
will be developed to collect data on numbers and types of partnerships established,
duration, purpose of the partnership, and other pertinent data elements.
How many business and industry people have
participated in terms of both time and financial support? The SC ATE Center
has had involvement from business and industry at many levels and in many
capacities. We will start by defining types of involvement in terms of time,
financial support, areas in which support has been provided, duration of
support, and related variables. We will use these to develop an interview
protocol to accompany the one developed for partnerships. Similar key staff
will be interviewed, with exceptions depending on business and industry
involvement.
How many other educational institutions
(and who) have participated? The SC ATE management team already has information
on participating institutions both within the technical college system (entire
16-college system) and external partner institutions (e.g., Clemson University).
These will be segmented by type of institution and type of participation.
A matrix of these two dimensions with counts of participation by type of
institution will be developed.
How many students are you impacting now
and how do you see this progressively developing in the future? Numbers
of students will be measured using data obtained from the exemplary faculty
survey (who are the teachers actually reaching students) combined with enrollment
data from the State Tech data system on students in those particular classes
(allowing us to determine counts of students reached by teachers participating
in the project).
B. Revisions to the Year Three Survey
Other than minor wording changes to reflect
faculty attitudes, activities, and plans since the previous survey, the
only survey revisions for year three are the addition of outreach and impact
questions. These are described earlier in Section IV.A.
C. Year Three Analysis and Dissemination
of Exemplary Faculty Data
The Final Exemplary Faculty Project Evaluation
Report, forthcoming in the Fall of 1998, will present longitudinal results
of the three-year survey research. Analysis of faculty change and conclusions
regarding project impact on faculty attitudes, practices, and future plans,
and their broader relation to broader SC ATE Center objectives will be presented.
The final report, along with special analyses,
graphic presentations of data, and data briefs, will be disseminated over
the SC ATE Web site. Results and conclusions will also be used to develop
SC ATE Center presentation materials and articles for publication. AED evaluators
and SC ATE management team members plan to make several presentations on
project and evaluation activities and outcomes over the coming year and
beyond. Several venues have already been identified for possible presentations:
· Other SC ATE Centers and project
sites (e.g., Center Director Ms. Elaine Craft and Evaluator Dr. Douglas
Evans have been asked by the New Jersey ATE Center to make a presentation
on student recruitment and retention research they are conducting for SC
ATE);
· National Education and Evaluation Research Conferences (e.g., American
Educational Research Association, American Institutional Research);
· Practitioner-oriented Conferences in Systemic Reform and Workforce
Development (e.g., League for Innovation); and
· NSF-sponsored Conferences (e.g., annual ATE project director's
meeting).
D. Incorporating Exemplary Faculty Evaluation into SC ATE Center Evaluation
The final Exemplary Faculty Project surveys
were administered in October 1997. In the fall of 1998, after the conclusion
of independent project funding, the project evaluation will be fully integrated
into the overall SC ATE Center evaluation. Activities will focus on the
following areas:
· Cadre faculty support for college
adoption and institutionalization of Center systemic reforms;
· Cadre faculty activities as "early adopters" and mentors
of fellow faculty in implementing the new curriculum, pedagogy, and other
professional practices;
· College and systemwide faculty development plans and activities;
and
· Faculty roles in pilot testing of the new engineering technology
Pre-ET and core curricula.
V. YEAR TWO RESULTS
A. Survey Results: Exemplary Faculty and Comparison Faculty
1. Sample and Methodology
Exemplary and Control Population Samples.
The SC ATE Exemplary Faculty project began with 63 participants in the exemplary
group and 53 in the control group. The year one report, dated March 26,
1997, compares the First Year Exemplary Faculty Survey responses of these
two groups. This section summarizes the results of the Second Year Exemplary
Faculty Survey administered to both the exemplary and control faculty. In
order to capture changes in faculty attitude and practice over the one year,
it was necessary to include only the individuals in each group who responded
to both surveys. The observations were matched based on birthday and middle
initial for both groups. As a result of this pairing, both cohort sizes
shrank due to both non-response and unmatched or incomplete cases. It is
important to note that the smaller sample size is not the result of attrition.
This procedure accounts for the reduced sample sizes for both the exemplary
and control faculty groups.
Statistical Procedures. In an effort to
garner as much information as possible from the survey data, two separate
statistical procedures were conducted:
· Paired samples t-test: The paired
samples t-test procedure was used to compare the means from year one and
year two for the exemplary group and to compare the means from year one
and year two for the control group. This test reveals if there is a statistically
significant difference between the means of the same variable (in this case
a response to a survey question) before and after SC ATE implementation.
In the following statistical tables, findings in bold represent statistically
significant differences between (1) exemplary faculty responses in year
one and year two, and, (2) control faculty responses in year one and year
two. Findings are all based on a paired-samples t-test, and are significant
at the p < .05 or p < .01 level.
· Independent samples t-test. In
order to compare the year two survey responses of exemplary and control
faculty, an independent samples t-test was conducted. The independent samples
t-test compares the means of a given question from two separate and independent
populations, in this case the exemplary and control faculty. In the following
statistical tables, findings in bold type and underlined represent statistically
significant differences between exemplary and control faculty responses
in year two. Findings are all based on an independent samples t-test, assuming
unequal variances, and are significant at the p < .05 or p < .01 level.
To illustrate significant results, this
section also includes selected graphic presentations. A complete set of
graphics for all year two results appears in the Appendix.
2. Faculty Characteristics
Table V-1 presents basic demographic information
about exemplary and control faculty for the second year of the project.
Faculty in each group are similar in age, teaching experience and gender
ratio. The consistency of these characteristics will ensure that any attitudinal
or pedagogical changes are not attributable to these differences.
Table V-1: Exemplary and Control Faculty
Characteristics
Characteristics Exemplary Faculty
(n=48) Control Faculty
(n=34)
Year 2 Year 2
Mean Age 44 45
Percent male 66.7% 61.8%
Mean years at current institution 7.0 9.6
Mean years of teaching experience 13.2
15.8
Table V-2 presents field of specialization
information for the exemplary and control faculty. Field of specialization
for exemplary and control faculty are similar, with the exception of mathematics.
Nearly twice as many exemplary faculty as control faculty teach in the mathematics
discipline, although this is not a statistically significant difference.
Table V-2: Field of Specialization
Field Exemplary Faculty
(n=48) Control Faculty
(n=34)
Year 2 Year 2
Communications, Business, and Social Science
20.8% 26.5%
Science 18.8% 17.6%
Mathematics 27.1% 14.7%
Engineering Technology (ET) 27.1% 32.4%
Other 6.3% 8.8%
Note: "Science" includes physics, chemistry, and biology specializations
reported by faculty. "Communications" includes English specialization
reported by faculty.
Table V-3 displays information on the position
or appointment held by members of each faculty group. Nearly 9% of control
faculty hold administrator positions compared with none of the exemplary
faculty. Again, this is not a statistically significant difference.
Table V-3: Position or Appointment at Home
Institution
Appointment or Position Exemplary Faculty
(n=48) Control Faculty
(n=34)
Year 2 Year 2
Instructor 62.5% 64.7%
Department Head 25% 17.6%
Program Coordinator 12.5% 8.8%
Administrator 0% 8.8%
|
