Six Steps to Continuous Improvement
The purpose of this guide is to help academic units develop and/or improve the process of assessing student learning.
Why is assessment of student learning important?In effect there are two reasons assessment of student learning is important.
Assessment is needed for improvement. Improvement, with its internal focus provides
How do we define the process of academic process?
The process of assessment, as it is understood in this guide, recommends that academic units
A comprehensive definition of assessment that portrays this process was proposed to the American Association for Higher Learning by Thomas Angelo, AAHE Bulletin, November 1995, p.7. The arrowed statements by ACAA following each component of the definition specify, broadly, actions to be taken.
â€œAssessment is an ongoing process aimed at understanding and improving student learning. It involves
What are the six steps to guide an assessment process?
1. Identify in broad terms what educational goals are valued
2. Articulate multiple measurable objectives for each goal
3. Select appropriate approaches to assess how well students are meeting the articulated objectives
4. Select appropriate measures that can be administered, analyzed, and interpreted for evidence of student learning outcomes
5. Communicate assessment findings to those involved in the process of assessment
6. Use feedback to make changes and inform curricular decisions and reevaluate the assessment process with the intent to continuously improve the quality of student learning.
These six steps can be applied to different levels or areas of student learning such as that of a program (e.g. graduate program or a program designed for a Learning Community), a major field of study (e.g. music), or a course (e.g. English 10001). The steps are useful for programs in academic units that are degree granting as well as for the wide range of programs and processes that influence learning, such as distance learning, library resources or liberal education courses.
Who is responsible for developing an assessment process for academic units?
What is the role of Advisory Committee for Academic Assessment?
The Office of the Provost appointed the Advisory Committee for Academic Assessment (ACAA) in Spring 2000 for the purpose of
This guide is a resource in support of assessment as are workshops and other events sponsored by ACAA.
A goal is a statement expressing what ideals are to be achieved. Goal statements tend to be broadly philosophical, global, timeless and not readily amenable to measurement. They capture the knowledge, skills, and values that students should acquire in a program by a course.
This first step in identifying goals requires faculty and others to reflect on questions such as the following:
The characteristics of goal statements should be the same whether the focus is at the level of the undergraduate major or minor, a specialized program, the graduate program, courses, or an entire unit. Several examples of broadly stated, philosophical, and hopeful goal statements follow.
Learning goals allow us to share with others the ideals of student learning we hope to achieve and to indicate the consistency of these goals with the mission of the university and its strategic planning. The above goal statements reflect the characteristics of such aims: they are general, they are ideals hoped for, they are not time-bound, and, unfortunately, they are not amenable, as stated, to being measured.
Herein lies the rub. Because the intent of academic assessment is to support continuous improvement of student learning, we must derive from these goals elements that can be measured. If this does not occur, we cannot evaluate how well students are learning what we expect of them. For this reason, the second task in the assessment planning requires the redefinition of goal statements as measurable objectives.
The task at step two is to redefine broad, global goal statements by specifying them in terms that allow for evaluation of how well students are meeting these learning goals.
List the student learning objectives for this major (course). Learning objectives should specify the activities, products, or performances to be measured and evaluated and the criteria they must meet for success. Learning objectives state what students will know, understand, and be able to do when they complete this major (course).
Defining objectives requires faculty and others to reflect on the questions below:
An example of the transformation of learning goal statements to learning objectives can be demonstrated using the examples of learning goals presented in Step One.
Goal: To develop â€œresponsible persons who will dedicate themselves to the . . . enhancement of the physical environment.
Learning Objective: Students will be able by their junior or senior years to critique various ethical and legal policies that impact the physical environment and defend, in both verbal and written work, their choices as to those that benefit this environment.
Goal: â€œTo foster literacy through the appreciation of the written word.â€
Learning Objective 1: Students will be able to master interpretive and analytical skills in writing about literature.
Learning Objective 2: Students will be able to critique and revise their own material.
Goal: â€œTo design and develop works of art that express ideas and personal feelingsâ€ . . .
Learning Objective: Students will provide a body of their artwork, accompanied with narrative, that demonstrates independent artistic development and self-reflection.
Restating learning objectives with as much specificity as possible by defining the criteria by which knowledge, performance, or values will be evaluated assures objectivity and makes required standards apparent. For example, in the second objective for English above, the faculty need to make explicit the level of performance students would attain for acceptable or unacceptable work. These revisions of objectives occur with discussion of what approaches or methods will be used as well as how objectives will be measured . . . the next steps in this process of assessment
Approaches define the procedures by which information is gathered; whereas, measures (in Step Four) are the specific instruments used to provide data. Some typical approaches (methods) used to gather information on student learning include portfolios, capstone courses, standardized achievement tests, external reviews, internship performances, focus groups, and so on. Multiple approaches (methods) and administration times are essential to ensure that students who may perform poorly with one method or at one time have other opportunities to demonstrate their learning.
More than one approach should be used to evaluate an objective. A valuable way to avoid the possible bias of using one method is to employ alternative methods at different points in time. For example, with regard to the learning objective for the School of Art on page 6, an approach might be student portfolios examined by external reviewers to judge if the artwork meets the criteria specified. An additional method is a student survey at graduation to evaluate to what degree students feel competent to perform the specified elements or criteria that reflect artistic development and self-reflection.
When selecting approaches to use, the following are some questions that need careful consideration.
Multiple approaches and times are essential to ensure that students who may perform poorly with one approach or at one time have an opportunity to demonstrate learning through multiple approaches and measures.
The task at this step is to identify and use measures appropriate for assessing the level at which students have achieved desired learning objectives. Needed now is agreement among faculty as to what evidence will assure that students are achieving the skills, knowledge, and values important to the academic unit. This moves the assessment process from a focus on intended results expressed as learning objectives to the level of achieved results.
Many measures can evaluate the objectives for learning, but it is important not to depend on a single measure to provide data about what and how well students are learning. Doing so can result in misinformation. Just as students learn in different ways, students respond differently with various evaluation tools. Using varied measures over time, including performance measures, more accurately affirms change and growth in learning. This allows greater confidence when recommending changes in the learning and assessment processes. Multiple measures to evaluate the learning objective for the School of Architecture (page 6) are offered below as examples of several ways to provide data about the level of student learning, each related to the same learning objective.
Capstone Experience Evaluation measures through explicitly defined criteria the competency level at which students have mastered the knowledge, skills, and values that define the major. To evaluate the level at which students have mastered one facet of this course, responsibility for enhancing the physical environment, seniors would write a paper. Faculty would establish a scoring guide of the essential elements used to judge this work. The elements might include knowledge of environmental policies, their historical development, their level of environmental impact, and studentsâ€™ understanding and valuation of policies that benefit or threaten the environment. Each criterion/element would have a subscale defined possibly as percentages or in specific terms that are agreed to measure the competency of student work.
Internally Developed Tests are established by consensus of members of the faculty to measure, as above, the level of knowledge students have about legal and ethical policies regarding the physical environment, how they would use policies to enhance the environment, and their personal values regarding this issue. To ensure questions focus on the objectives of the program, elements should be evaluated that nearly all faculty in the program agree should be known, applied, and show commitment to the learning objective.
Surveys garnered often from alumni, employers, and students indirectly measure through self-report the competency of student learning at various times during their academic career or after completion of their studies. The School might choose a survey of graduates three years out to evaluate what responsibilities they are taking with regard to enhancing the environment, the degree to which the believe their studies fostered the direction they have taken, and the value of this objective to their job opportunities or community service.
Some questions that need to be considered when selecting measures include the following.
The task at this step is three-fold. First is to collect the information from measures that have been chosen to provide evidence of how close studentsâ€™ actual learning comes to meeting the expected outcomes faculty and others have for a course, a major, or a program. Second is to evaluate what is found. Third is to share the findings.
Some questions that need consideration when planning this step follow.
In connection with the collection of measures
In connection with the interpretation of measures,
In connection with the communication of results,
Sharing results leads to the reevaluation of the assessment process to improve student learning.
Academic assessment is an ongoing process that requires continuous reevaluation as to whether teaching and learning processes achieve the goals and objectives defined by faculty in the academic unit. When students succeed in achieving those goals and objectives, one might assume that the teaching and learning processes are functioning well. When students do not achieve those goals and objectives, changes should be made in teaching and learning processes. Reevaluation after changes are made will suggest if those changes were helpful to student learning. In this way, assessment creates a continuous cycle through these six steps in the assessment program and teaching/learning processes.
In making changes, faculty should consider the following two questions:
What elements of the teaching/learning processes should be added, deleted, or changed to improve student success?
Did the assessment plan for the academic unit produce results that have face validity? If not, why not?
Making changes to enhance student success requires reflection and thoughtful analysis foreshadowed in the actions suggested in the section on â€œHow do we define the process of academic assessment?â€
Some believe when the words â€˜improvementâ€ or â€œenhancementâ€ are used that something is wrong. That is not the case. Most faculty, for example, are accustomed to reviewing and looking to improve what occurs during class time, at the end of a course, or in committees that discuss curriculum, pedagogy, and other educational matters. The intent of step six is the same -- to plan, often with others, new ways to accomplish their goals for students.
Some questions that need consideration at this juncture follow.
In order to develop this guide, it was necessary to define and explain terms as they seemed appropriate for the purposes of this guide. ACAA recognizes that some terms are understood in ways different from that used here, concepts such as assessment, methods, measures, goals, objectives, and values to cite a few. In an effort to be helpful, not arbitrary, the following explanations of terms as used in this guide are offered.
Approaches are the procedures used to gather the information needed to assess how well students have met the learning objectives. They are the course of action through which evidence about courses, programs, majors and the like will be gathered. To provide quality information, multiple approaches should be used.
Assessment refers to a continuous process instituted to understand and improve student learning. While academic units may find alternative pathways to arrive at this goal, this process needs to begin with articulation of educational goals for all programs and courses. These goals should be expressed as measurable objectives followed by the selection of reliable and valid methods and measures. After collecting, interpreting, and sharing findings, the aim is to use these learning outcomes to better understand how and what students learn, how well students are meeting expected objectives, as well as to develop strategies to improve the teaching and learning processes.
Benchmark is the actual measurement of group performance against an established standard or performance, often external.
Criterion is the standard of performance established as the passing score for a performance or other measures such as a test. The performance is compared to an expected level of mastery in an area rather than to other studentsâ€™ scores.
Cross-Sectional Studies provide information about a group of students at one point in time.
Evaluate and Evaluation are terms used in this guide to indicate the interpreting of findings and are used as synonymous to the term assess and assessment. ACAA is aware that many make a distinction between evaluation and assessment with the difference that assessment is a process predicated on knowledge of intended goals or objectives while, in contrast, evaluation is a process concerned with outcomes without prior concern or knowledge about goals. That distinction is not used in the guide.
Goals are statements about the general academic aims or ideals to which an educational unit aspires. Goal statements allow us to share with others our hopes in regard to the learning achievements of our students. Further, goals at the unit level should align with the mission of the university. Goal statements are not amenable, as stated, to measurement.
Longitudinal studies provide information from the same group of students at several different points in time.
Measures are the specific instruments or performances used to provide data about learning. They are the tools that are to provide information as to the level of achieved results or outcomes. To avoid systematic bias in findings, multiple measures are required.
Methods - see approaches.
Objectives are the redefinition of learning goals in a way that permits their measurement. Objectives express the intended results or outcomes of student learning and clearly specify the criteria by which student knowledge, performance, or values will be evaluated.
Process is a method generally involving steps or operations that are ordered and/or interdependent.
Qualitative and Quantitative Research describe two research methods. Both are valuable as a means to assess student learning outcomes. In a practical and somewhat philosophical sense the difference is that quantitative research tries to make use of objective measures to test hypotheses and to allow for controlling and predicting learning. Qualitative research makes use of more subjective observations of learning.
Reliability is the extent to which studies or findings can be replicated.
Sampling consists of obtaining information from a portion of a larger group or population. When the selection of a sample is randomly chosen there is greater likelihood that the findings from the sample will be representative of the larger group.
Validity depends on demonstrating that a measure actually measures what it is purported to measure.
There are many sources of information and assistance available to help with the tasks suggested in this guide. A full listing of these with brief comments as to their purpose will be available at the ACAA website. This page suggests a limited number of resources that may be valuable to be aware of immediately.
The Advisory Committee for Academic Assessment, the Office of Research Planning and Institutional Effectiveness, and the Faculty Professional Development Center are all resources on campus that can assist in certain areas of academic assessment planning.
The Office of Academic Assessment has a small library focused on academic assessment, with books, articles, and journals as well as various sourcebooks regarding methods and measures.
From a multitude of internet resources, a few that may be valuable sources to begin with follow.
North Carolina State University - University Planning and Analysis
California Academic Press - Student Outcomes Assessment: Opportunities and Strategies
American Association for Higher Education
The Higher Learning Commission