Assessment

Assessment Standards

1. Student learning outcome Statement

a. Describes the change in students' knowledge, attitude, and/or behavior.

Rationale. Assessment looks at what the program does for students, not what students do in the program. The learning outcome should describe what happens to the students’ set of skills, beliefs, and knowledge over the course of the program; in other words, how efficient is the program in what it claims to do.

Incorrect: Students will write a thesis.

Correct: Students in the (name of the program) will demonstrate the ability to present defensible conclusions from an investigation of pertinent primary and secondary sources.

In the example above, the learning outcome specifically refers to the ability that students will get from the program: “the ability to present defensible conclusions”. Then, in the method of assessment section you may state that students will write a thesis to demonstrate the above SLO.

b. Is not stated in terms of assignment.

This standard is an extension of the previous one. Do not state what students will do in the program.

Incorrect: Write a thesis, complete a course, or take an exam.

These are assessment instruments and belong in the Method of Assessment section. In the SLO section, state skills students will acquire from the program.

For example, a thesis is not an end goal of getting a degree; the end goal of the degree program includes the knowledge, skills, attitudes and habits of mind that students take with them when they successfully complete a program. A thesis may serve as a tool to demonstrate the knowledge, skills, attitudes and habits of mind.

c. Is unique to the program.

The learning outcome should be specifically tied to the program. If you have two programs with identical learning outcomes, the implication is that these are two identical programs. If students learn identical things in programs A and B, then program A is identical to program B; and thus, one of the programs should be eliminated.

Rationale. Each program is designed to give students a unique set of skills and abilities. For example, although undergraduate degrees in biomedical science and chemistry have the majority of shared courses, these degrees prepare students for different careers; therefore, they should have different content knowledge learning outcomes. Communication skills and critical thinking skills may be shared.

d. Is observable and measurable.

The learning outcome should be stated in a manner that facilitates measurement. You cannot measure what you cannot see; therefore, the SLO should be stated in terms of students demonstrating some skill, behavior, and/or knowledge.

Incorrect: Students will be good citizens.

The above example is an example of a degree goal. Consider instead the following example.

Correct: Students will be able to apply the Amendments to the Constitution of the United States in various situations.

This skill constitutes the fact of being a good citizen, but it is also observable and measureable. As a result, this outcome naturally lends itself for the assessment method. You may ask students to write an essay asking students to apply their knowledge of the amendments to their life; or you may design embedded exam questions that present a case and ask students how amendments can be applied to that case.

e. Employs a verb from Bloom's taxonomy.

Please select the most appropriate verb from Bloom’s taxonomy.

Bloom’s taxonomy is the hierarchical model of classifying cognitive skills in terms of complexity.

Rationale. Understanding the material is better than just knowing the facts, being able to apply the knowledge is better than having the understanding of the material, and so on. Choose the most appropriate cognitive skill for the level of your program (Bachelor’s vs. Master’s vs. Doctorate), and for the level of the class within the program (Introductory course vs. capstone course). Students in the beginning of their education may focus more on knowing and understanding the material, while more advanced students should apply, analyze, and synthesize the material; not the other way around!

Correct: Undergraduate students in physics in their freshman year will demonstrate knowledge of basic laws of electricity and magnetism.

Correct: Undergraduate students in physics in their senior year will be able to apply laws of electricity and magnetism to a wide range of situations.

Correct: During their third year, doctoral students in physics will be able to produce a scientific paper of a publishable quality that constitutes an original contribution to their chosen field of specialization.

Note that as students advance through their educational careers, they are required to demonstrate cognitive skills of higher complexity. From knowledge to application to synthesis of new knowledge.

f. States specific group of students who will perform the SLO.

Indicate to which students the stated SLO is directed. For example: sophomore students, graduating students, students taking a required capstone course, students completing core sequence of the courses, students entering their senior year, etc.

Rationale. Students achieve different learning outcomes and skills in different points of time during their educational career, and some learning outcomes are stepping stones for others. For example, there is a huge difference between assessing graduating students and assessing students entering their junior year. If you assess graduating students, you only obtain a summative snapshot of their progress over the course of the program. You may only get the following information: 20% of graduates could not apply some specific skill. On the other hand, if you assess students entering junior year, or in other words, students that are about to take upper level courses, you will get different kinds of information. It may turn out that students do not get sufficient training in the lower courses, and that is why they fail in upper level courses! Having this kind of information will give faculty an opportunity to intervene, make changes to the lower level curriculum, and make sure that students understand the basics before taking upper level courses.


2. Method of assessment

a. States and describes the assessment instrument.

In as much detail as possible, describe assignment, activity, and so on, that will be used to assess the stated learning outcome. Common assessment instruments include: essays, written student work including discussion board responses, theses/dissertations, presentations, oral reports, performances, portfolios, open-ended (or multiple choice) embedded test questions, lab reports, internship or practicum evaluation forms, exams or standardized tests.

b. Indicates how the instrument specifically measures the stated SLO.

Justify how selected assessment instrument specifically addresses stated SLO. Is it one of the criterion in the rubric? Are there specific embedded questions? Please see an example below

For example, the SLO states that student will be able to do x, y, z. If the assessment instrument is a multiple choice test, please provide the statement that aligns x, y, z to specific questions in the test (i.e. questions 2, 5, 11, 13, 16, 23, etc.). If the assessment instrument is term paper and rubric is used to assess the paper, please specify which rubric components measure x, y, z.

c. Is distinct from grading.

Grades are inappropriate for continuous quality improvement; they summarize overall performance of the student.This type of assessment will not necessarily yield data that can be used for improvement. A student with a 70% overall test score may fail in one objective which may need to be improved. One option is to measure each aspect separately, report those ratings, and then average them together.

Take a look at the following example of a mediocre assessment. Students will write theses, professor will then assign grades to each thesis. A year later you can see the following distribution: 20 students received A, 40 students received B, 15 students received C. What can you do with this information? How can you improve the program? What are common problems? What difficulties do students encounter while writing a thesis? Which skills are underdeveloped? Is it writing, research skills, ability to defend an argument, or gaps in the knowledge of the discipline? If there are gaps in discipline specific knowledge, then in what area? Letter grades do not give useful information that can be used to make adjustments to the curriculum.

d. Describes the assessment context.

A statement which delineates the course(s) in which the assessments were administered. If administered outside of a course, under what circumstances the assessment was administered.

Context for assessment can be divided into two types: (1) Course-embedded assessment and (2) assessment outside of the course. Examples of the former include a project in a capstone course, final exam in one of the core courses; examples of the latter include qualifying oral exam at the end of the program, portfolio of student work drawn from multiple classes, internship evaluation forms, licensure exam administered outside of the coursework. Regardless of what assessment type you selected, please provide explanation of the assessment context.

Example of an assessment outside of the course: “Assessment instrument is an oral qualifying exam that students have to pass to complete a degree program. An Oral Examination Committee comprised of three professors drawn from the student’s core [ABC XXXX, ABC XXXX, ABC XXXX, ABC XXXX] and elective [varies year to year] coursework will conduct the evaluation.”

Example of a course-embedded assessment: “Final project in the ABC XXXX capstone course will be used as an assessment instrument”.

e. Indicates the sample.

Will you assess all the students or sample of the students? Please provide all the relevant statistical information and sampling techniques employed.

f. Addresses all the rubric's requirements (if used).

(1) What are the criteria of the rubric?

For example, criteria in the Oral Communication VALUE Rubric are: Organization, Language, Delivery, Supporting Material, Central Message. When student gives a presentation for example, criteria stated above are all assessed separately; scoring student work on a single scale is unacceptable, as it does not show a full picture.

(2) What is the range of the rubric score and what does each rating mean?

For example, if you state that students will be evaluated on a scale from 1 to 5, please elaborate what is meant by each rating (getting 1 means this, getting 2 means that, and so on). A rubric is a subjective assessment instrument and should be rigorously defined.

(3) Who will be evaluating the students?

Please state who the raters are, as well as a number of raters (at least two). Due to the fact that rubrics are inherently subjective, only individuals who know the subject well should evaluate the students. Usually, these individuals are faculty members; sometimes professionals outside of the university are allowed, however, you need to justify it. Students are not allowed as evaluators.

Rationale. Compared to the use of test questions that can be either right or wrong, the use of rubrics has some degree of subjectivity. One rater may think that the student deserves a 3 out of 5 on this criterion, another rater thinks that the student deserves a 4 out of 5; different raters may interpret aspects/criteria of scoring differently. Multiple raters are needed to improve the reliability of the measurement.

(4) How is inter-rater reliability (IRR) addressed?

Inter-rater reliability is the degree of agreement between raters. In other words, the method of assessment needs to state how drastic differences in scores (if any arise) between two reviewers would be addressed. Common methods of addressing the inter-rater reliability include: (1) raters discuss until they reach agreement regarding the rating, (2) if raters cannot agree on the rating, then a third rater is utilized. SACSCOC is looking for percent agreement stemming from rubric calibration to ensure reliability in rubric scoring across multiple raters. 


3. Performance Target

a. Is quantifiable.

Performance target should be stated in terms of the assessment instrument. For rubrics, it should be stated in terms of overall (or components) rating. For embedded questions, it should be stated in terms of number of questions answered correctly.

b. Specifies the threshold of success and indicates the percentage of students that will reach the threshold.

Performance targets are internal predictions made by the program regarding the level of student achievement for that SLO. For example: “Program implementation will be considered a success if 90% of the sample will achieve a final score of 4 or higher for this assessment.” This is the extension of the previous standard. In addition to specifying the benchmark result, specific how many students (percentage) will reach that threshold.

c. Aligns with learning outcome and method of assessment.

Performance target should be related to the method of assessment and learning outcome. Please verify that there is a common thread throughout your assessment plan: this is what we want students to know, here is how we will measure it, here is the numerical target that would indicate the program is actually successful in providing knowledge and skills to its students.

d. Stated in reasonable terms.

Setting performance targets is up to the program; however, the benchmark should be meaningful and appropriate for making decisions regarding the program. Saying that 100% of students will reach the threshold may not be realistic. Additionally, stating that 30% of students should get at least passing results may be too low. Please note, assessment should produce results that will help improve curriculum and/or instruction.

 

4. assessment Results

a. Mirrors the wording of the performance target section.

Results should be worded in terms of the performance target.

b. Includes total number of students assessed.

How many students were assessed? If you were using a sample, what proportion of the total population is the sample?

c. Includes number of student that reached the benchmark.

How many students actually reached stated benchmark?

d. Provides sufficient statistical information about the results.

Provide as much statistical information as possible (mean, median, standard deviation, etc.). 

 

5. Use of Assessment results

a. Interprets and analyzes the results. Provides reflection.

This is the most important section of the assessment – this is the reason why assessment practice is required by regional and specialized accreditors throughout the world. As a university, we should continuously improve: each program should look for weak areas in the curriculum and address them. There is an important distinction to note: this is an assessment of the program, not its participants.

For this section, please look at your results and interpret them. What do the data show? Are there any anomalies? Does anything standout? Provide as much narrative as possible.

Reflection: What insights arose from this process? What did you learn?

b. Includes actionable “next steps” the program will take.

Based on the interpretation of the results, state actionable changes. Programs should look at and think about what improvements or developments will be implemented at the program level in light of the assessment results. This section is not for programs to describe how they will change their assessment plan to yield greater levels of student achievement, or to elaborate on the assessment results in any way. In addition, this section is not meant for programs to relay how they will work with students differently to achieve greater results (e.g., advising students to seek tutoring).

Example: If the Critical Thinking assessment resulted in a significantly lower number of students achieving at the performance target, than the Use of Results section could include how and where the program will reinforce critical thinking skills, what adjustments will be made to the curriculum, how program faculty will address the deficiency and other future improvements or developments.

c. Does not include phrases such as “we will continue to monitor…”.

The use of the above phrase violates the continuous quality standard found in most accreditation principles. Assessment is not linear and finite; it is continuous and seeks to assess program development on an annual basis (continual improvement). If all performance targets have been met within a plan, the program is asked to develop SLOs that improve new areas aside from what has already been “perfected.”

Example: If the Critical Thinking assessment resulted in sufficient scores to indicate that the measured SLO had been met, then the program should include a statement that the program is functioning well in this area, as well as a statement of the projected area of concentration for the subsequent year’s assessment.