Using Test Results: A Four-Step Process
By Dr. James Cox
July, 2000
Copyright 2000 JK Educational Associates, Inc. All rights reserved
While working with schools the past four to five years I have observed that on a school or district site rarely is there a process in place for working with high stakes test results, once those results arrive at the doorstep. The political arena has so overpowered the educational intent of testing that typically we jump into the numbers, see how we did, celebrate or wring our hands, and then begin writing the dreaded, but always mandated, improvement plan.
Sadly, when we allow the political intent of a test to rule our professional lives, we lose much of the genuine potential that test scores have in assisting to elevate the quality of our instructional programs. When we focus on "raising test scores " rather than teaching kids, our goals, also, shift to the political arena ... and teaching suffers as a result.
Major issue: Raising test scores and improving the quality of instructional programs are not the same thing!When we work with high stakes test scores, there is one courageous belief we must embrace. That is, we must believe if we improve the quality of our instructional programs, achievement, including test scores, will also improve.
If we cannot embrace this "cause and effect relationship" between the quality of our work and the results we get in the form of higher achievement including test scores, then we end up keeping our eyes on the scores, and take our eyes off the kids. Guess who loses?
Resolving the dilemma
I believe that to work with test scores effectively requires that educators progress through a prescribed set of steps, keeping the political arena as far away as possible. I'd like to present for your consideration the four steps to which I subscribe. They are no panacea, but they do keep us focused, while at the same time keeping the scores in perspective.
Step 1 Establishing the Environment
Step I is less a step, than a starting point, but I believe it is so important that I have included as Step 1. As leaders we must establish an environment in which our staff knows and believes that working with high stakes test scores is very important work. Taking a proactive approach rather than reacting to external forces is the goal of Step 1. When we're told that something is important, but deep down we don't really believe it, we're going to end up giving it a puny effort.
I know of no educator who is thrilled at the notion of being judged solely by the results of a test. We do so much complaining we often fail to acknowledge that "that's the way it is." Reacting in such a way often results in our stakeholders' viewing us as cry babies, afraid to be held accountable for doing good work with their kids. When our stakeholders view us this way, all we're doing is widening the credibility gap between us and those whom we serve. Accountability is here to stay and regardless of how unfair we believe some of these practices to be, we've got to ready ourselves for the challenge.
The attitudes we carry with us will go far in determining the manner in which we do this work. The leader cannot give a half-hearted effort to encouraging his or her staff to take this work seriously. I don't believe that creating a positive environment when working with test results happens by osmosis, or because the boss deems it so. A healthy environment must be planned and initiated. A healthy environment means no excuses or "cop outs;" A healthy environment acknowledges that improvement is not only possible.....it is a professional obligation.
But there will be those who resist becoming part of what appears to be an overwhelming endeavor. While such resistance is understandable, interventions to get them "over the hump" become necessary. Dr. Spencer Johnson has written a book titled Who Moved My Cheese? It takes 30 minutes to read this national best seller, and anyone who is having difficulty accepting the reality of educational accountability is encouraged to learn from this simple parable dealing with the inevitable reality of change. (www.whomovedmycheese.com).
Step 2 Working With the Data
Step 2 is the step in which the numbers are at center stage. This is the number crunching step and consists of three processes, data summary, data analysis, and establishing need. These are addressed below.
Summarizing the Data
Summarizing the data means nothing more than putting the data in a user-friendly format. Folks who don't work with data for a living and are exposed on occasion to information, such as test scores, need to feel comfortable with the numbers. Test scores normally come to the educator in the form of computer printouts of various types with the accompanying admonition to "do something with it." I've been in the data business close to 35 years and I've never seen computer printouts that were user friendly. Alternatively, they may arrive bound inside a pretty cover, full of graphs and tables suitable for display ... but equally unusable as a tool for analysis. Normally, there are too many numbers, too many graphs, and too many pages. Thus, "doing something with it," is often met with "???????"
Peter Drucker, a big name in business management, says this: " The fewer data needed, the better the information. An overload of information, that is, anything much beyond what is truly needed, leads to information blackout. It does not enrich, but impoverishes." He suggests that we limit the amount of information we work with, but, with certainty, what we have is the most important.
Because there is so much data potential from a testing program (multiple content areas; multiple grades, multiple years; multiple groups and subgroups; etc.), we must first consider what is the most important to know and deal only with that. When we screen out that which is less important to us, we can eventually hold the important stuff in the palm of our hand. These are the data that promote understanding and utility.
Data can be presented or formatted in two ways, for display and for utility. Data display promotes clarity for oral or written reports and clear communication is the goal. In this context we often see graphs and charts. Data formatted for utility produces action. In this context we see a step-by-step structure and school wide patterns of achievement. Understanding, not display, is the goal.
The presentations of data intended for display and data intended for utility look different. Too often staffs attempt to use display data to generate action, which ultimately results in shallow understanding and patchwork plans for improvement
Analyzing the Data
The second process in working with the data is analysis. Analyzing the data simply asks us to answer the question, "What do these data tell us about the achievement levels of our students?" Though we often think so, this is not a complex step. When we hear the word, "analysis" we tend to think we need five courses in statistics to do the work. Such is not the case [five courses in statistics hurt you!]. Data analysis is a very common sense issue; for example, we look at the data and conclude that we're doing better in primary reading than in intermediate reading, but we're performing better in intermediate math than in primary math; our girls are performing consistently higher than our boys; students who have been with us at least two years are performing 10-12 percentile ranks higher than those who have not been with us as long. That's all there is to it. Analyzing the data prepares us to move to the next point, establishing a priority of student need.
Establishing need
Once a staff understands what the data say, we draw a conclusion that says, "These are our students' needs." Needs can be content, grade, and/or subgroup. We make statements such as, "We have a need in primary reading; we have a need in intermediate math; we have a need in reading with our boys; we have a need in Grade 6 across the board."
Establishing need does not result in problem solving or solutions such as a need for more materials or for additional staff development. That comes later. Establishing need focuses on students.
Step 3 Establishing Cause
Establishing cause is by far the most important step in the four-step process.
It is also the step that is most often neglected. At this point we answer the question, "Why do we have a need in primary reading? .... in intermediate math?....in reading with our boys?.... in Grade 6?"
In our best professional judgment we identify those causes that are within our control and that by addressing them would impact student achievement and test scores. Causes for needs are of two types, pollutants and program quality.
Pollutants
You will recall from the paper titled, "Why Test Scores Result the Way They Do," that scores can be caused in part by four pollutants. I say, pollutants, because these four variables pollute the accuracy of your results, while having nothing to do with the students' achievement level in the particular area being tested. These four include (1) the physical environment in which testing occurs; (2) the attitudes of staff and students toward the testing program; (3) the degree to which students have test taking skills; and (4) the degree to which content of the instructional program and content of the test are aligned.
A pollutant analysis considers each of the four and reports the degree to which the staff believes the pollutants played a part. You may not have a need in primary reading, for example, nearly as much as you have a need to clean up your pollutant act in the primary grades. Of course, if there is considerable pollutant contribution, this is something that must be changed.
Program Quality
Once you have considered pollutants as a cause, your needs can also be attributed in part to the overall quality of your instructional program. Once again I refer you to the paper titled, "The Quality of an Instructional Program (the oft used phrase with questionable meaning)" This article suggests that the quality of an instructional program is defined largely by 15 factors, some of which are no doubt stronger than others. The paper refers to these as "program supports" and "program deterrents. In order to establish cause, a staff has the task of identifying which of the 15 elements they believe currently support and which deter the effort.
Important: Supports and deterrents are relative. You can have a high quality reading program, for example, and you will still have relative weaknesses. All instructional programs have supports and all programs have deterrents. Identifying them is crucial to the success of working with test scores effectively.Following this line of thought, an improvement plan will focus upon eliminating or drastically reducing the deterrents while maintaining the quality of the supports. Logic suggests that when the deterrents are eliminated, at least in part, the quality of the program will improve and the achievement of our students will go up.
Bottom Line: The reason why Establishing Cause is the most important of the four steps is that if a staff accurately identifies the causes of need, then the improvement plan will naturally follow. If the staff inaccurately identifies the causes, then your improvement plan will be little more than an academic exercise pointing itself in the wrong direction.
Step 4 Developing the Improvement Plan
The most important thing to remember about step 4 is this:
NEVER WRITE A PLAN TO IMPROVE TEST SCORES. WRITE A PLAN TO ADDRESS CAUSE. THE FOCUS IS ON DOING SOMETHING BETTER, NOT DOING SOMETHNG DIFFERENT.
A plan to improve test scores keeps us in the political arena. A plan to address cause puts us into the educational arena. If continuously improving the overall quality of what we do is our goal; we must be in the educational arena to accomplish it. If the causal factors we identified in Step 3 are the right ones (at least partially); we will be headed in the right direction. If the factors we identify are not the right ones, our plan will probably go up in smoke.
From Step 3 identify the major causal factors that you believe are the most important realizing that you can't deal with all of them. Your improvement plan will focus on eliminating or drastically reducing each of these causal factors. Some of the factors will focus on pollutants while others must be quality based.
During this phase a staff wants to do all in its power to avoid the catchy solutions, the ones that might "look good," but ones for which there is little supporting evidence or research. "Research based" is an over-used phrase, but it is important to be cautious. Addressing some causal factors will, therefore, require that current research findings be considered before arriving at a solution. Remember, an intended improvement plan may not be an improvement plan at all. You may just end up doing something different, not something better. To affect student achievement, what we do has got to be better than what we did. These are important divisions; a research base can give us a head start.
Final Considerations
Be Creative
When you develop your plan, be creative. As I have said in seminars, you and I have written 487 school plans, and we've become pretty hardened to the idea of writing another one. Therefore, we tend to just write something down to get done. "Run it up the flagpole and see if the boss salutes it."
One way to liven up a school plan is put some creativity into it. For example, what might you do to provide time to work with low achieving students individually? How about using the senior center? Or maybe tutoring from high school kids? You get the idea. We tend to box ourselves in by writing an improvement plan that looks like all the other improvement plans. Give yourself permission to do things a bit differently.
Consider the Whole
When you develop your improvement plan, immediately assess what impact the plan will have upon the rest of your curriculum. You may have created the best improvement plan in reading that's ever been done, but in the process may have ruined your science or social studies program. Know right away that that's what you've done and accept or reject it rather than discovering it six months from now.
Closing Note
In all the years I have worked with test results and tried to make meaning of them, the one truth for me that looms above all else is that focusing on the quality of our work has a greater effect upon student achievement than focusing on our test scores. If we focus on the quality of our work, the test scores will take care of themselves. If we focus on raising our test scores, we may show test gains, but to what end if the gains don't reflect the quality of our instructional work? Long term results require quality.
The stakes are too high to allow panic to overcome judgment. Methodically progressing through a school's test results will reduce the probability of panic and increase the probability of improved program quality and improved student achievement. Thus, I ask you to consider the four steps in your own work.
