"The Quality of an Instructional Program"s
By Dr. James Cox
July, 2000
(The oft used phrase with questionable meaning)
Introduction
You find yourself at a social gathering, a very pleasant affair, and are approached by a supportive constituent. The person, who has absolutely no ax to grind, makes the following comment, followed by a question: "I have heard you say on several occasions that you believe you have a very high quality reading program on your campus. You know how supportive I am, and I have absolutely no reason to doubt you. But for my understanding, when you say you have a high quality instructional program, what do you mean by that?"Pause, followed by possible embarrassment.Too many of us in the profession when, posed with such a question, are not sure how to respond. Part of the reason, I believe, is that when an important concept, such as "a high quality program," becomes jargonized, it loses its meaning, and there are as many meanings for the phrase as there are people who use it. We have been so barraged of recent, both internally and externally, with the indictment of not providing high quality instructional programs and "You've got to improve," that the important concept of quality has been lost along the way.Thus, there is a real need for us, I believe, to revisit the phrase, "quality of an instructional program," and give it real meaning. After all, when our stakeholders see our test scores in the newspapers, they tend to attribute the cause solely to the quality of our work. While we know this may not be the case, it is of little consequence to them. Because they tend to attribute scores to program, we'd better be very well versed with the quality concept and know what we're talking about.What do we mean when we say, "quality of an instructional program?"
The quality of an instructional program is comprised of three elements, materials (and equipment), activities, and people. The three make for an interesting acronym, MAP. I often refer to the quality of an instructional program as the quality of one's MAP. Incidentally, when I use the phrase, instructional program, I'm talking about a school's reading program, math program, writing, physical education, business, Tide 1, the program for English Language Learners, and the like. We have a bunch of instructional programs on a campus and each one has its MAP.
When we speak of materials and equipment, we are referring to the quality of materials, the quantity, the accessibility, the appropriateness. We are talking about materials for students, for teachers, for parents; we're taking about computers and software;Activities are of two types, instructional and support. Instructional activities are those activities that we do 'in the presence of kids and support activities are those things we do in the absence of kids. Instructional activities include such things as instructional strategies, grouping techniques, assessment activities, reteaching strategies. Activities such as curriculum planning, curriculum development, staff development are support activities. The people part of a school's MAP includes the roles and responsibilities of the adults involved in making the program work. These are the teachers, instructional assistants, instructional specialists, administrators, district office staff. The people part of one's MAP also deals with the knowledge, skills, experiences, and attitudes of the people responsible for the instructional program.If I were to figuratively take all the materials, all the activities and all the people in an instructional program and toss them in a barrel, mix them up and pour them out, I would have the instructional program.
An instructional program, the MAP, has quality attached to it. Consider this. If the vertical axis is the program quality axis from high to low and the horizontal axis is the dimension of time, then our goal would be, while operating at a given level of quality, to work to improve that level of quality. The rationale for this is that we believe that there is a direct relationship between the quality of what we do and the level of students' learning. The better we are, the higher level of achievement the students will attain. If we do not work under this premise, I would wonder why we're in the business.
Reading the MAP
Two sets of elements comprise the quality of any instructional program. (represented by the x's in the figure). The set above the quality line are the stronger parts of your MAP. These are the elements that pull the quality of your program up. I call them the program "supports." The set below the quality line are the barriers or weaker parts of a program. These are the things that are keeping your program from being all it can be. These are the "deterrents."Supports and deterrents are relative. Every instructional program on a campus has supports and every program has deterrents. You can have the highest quality program in the world and that program still has deterrents. Likewise, a weak program has supports-Note the following: In one case the quality is very high; in the other, very low; both have supports and both have deterrents.
If we can remove or significantly reduce the effect of the deterrents, the quality of our program will be enhanced. And thus, student achievement will be enhanced. First, however, we must accurately (as well as we can) identify the nature of the deterrents. To clarify this notion, consider a scenario.
A Scenario: Part I
It was the mid 1980's. A high school achieved less than positive results on a high stakes writing test. Staff scurried around in an effort to find the solution to the dilemma. The decision to implement Writing Across the Curriculum in each classroom was quickly made. Training of all teachers in the school was conducted and writing commenced to be a part of each classroom, regardless of the subject being taught. Subsequent test results failed to achieve the desired goal.
Same Scenario: Part 11
In a very positive, proactive environment, the school tried again. However, in this case, supports and deterrents were assessed by staff. Here were the results of this program diagnostic analysis: The school had a veteran staff and a very stable staff. They'd been around for a while and they were aging. It was discovered that the median age of a very competent group of English teachers in the school (those staff primarily responsible for teaching writing) was between 45 and 50 years old. If an English teacher were somewhere around 45 years old in the mid 80's, chances are he or she had never been trained to teach writing. Teaching of writing as a process didn't surface as a technology until the late 70's or early 80's, and therefore, the teachers had never been exposed to the newly developed strategy. Typically, they equated the teaching of writing with the assignment of writing. "Do you teach writing?" "Yes I teach writing". "What do you do?" "Two papers a week. "
An additional discovery was that no part of the curriculum supported the teaching of writing. The curriculum was a traditional curriculum, literature, grammar and the like, with no direction for teaching writing.
Same Scenario: Part III
The school could have been wrong with this assessment. But they weren't. They were correct. English teachers were trained and supported. Curriculum was developed and, need I become Paul Harvey to tell you the rest of the story? Writing achievement, as measured by the important exam, began to go up.
In this scenario, Writing Across the Curriculum was a garbage solution. It related to little going on in the writing program. It may have had some benefit, but it didn't touch the real issue. When the deterrents were discovered, and plans implemented to eliminate the deterrents, the quality of the program had been enhanced, and the kids responded in kind.
Bottom line
Writing Across the Curriculum was initiated as a means to improve test scores. Teachers being trained and curriculum developed were a means of improving the quality of the instructional program. The school set out to improve test scores and it didn't work. Then they set out to improve program and the scores went up. There's quite a difference between the two scenarios.
Unless educators are able to analyze the quality of a program's MAP, it is doubtful that substantive improvement will take place. As a profession we are very solution focused. "We have a need. Let's try this!" We must become much more analytical regarding the instructional programs we provide.
The Analysis of Process (AOP)
Several years ago I developed an instrument called the "Analysis of Process" (AOP). The purpose of the instrument is to help a staff or leadership team identify the supports and deterrents of any instructional program. It is no panacea, but it does provide direction and helps a group stay structured as they progress through the diagnostic activity. For reference, a completed AOP is included at the back of this paper.Let's start by stating what the AOP is not. It is not a questionnaire. You don't put this in teachers' mail boxes and say, "Return by Thursday." The intention is that the form be a catalyst for dialog, that it guide a group through the diagnostic process.A group of 5-7 educators, teachers and administrators, come together to discuss a particular program. To be a part of the group requires meeting two criteria. First, each member has a very good knowledge of the program being addressed, and, second, each member has a vested interest in the program.The process is intended to last one hour, AND IT IS TIMED. Thus, from among the group, you will assign a facilitator and a time-keeper. The roles of these two individuals are to keep the process moving and to avoid bird walking. A very important aspect of the roles of the facilitator and time-keeper is to keep the group from getting into a solution mode. We are so quick to want to solve the problem, we tend to solve too quickly, before we really have had a chance to look at all sides of the program.
Down the left column is a set of 15 elements that can support or deter any instructional program. These are all elements of a program MAP. The set of 15 do not carry a research base as in, "These are the ones you MUST be concerned about," but they have withstood the test of time for me, and I believe you will find them quite credible. The second column, labeled, "Description," gives a brief description of each element.
The Activity
The group comes together, after having become familiar with the process, and sets out to "discover" the supports and deterrents of the program under scrutiny. They address each of the elements in order. First, is the materials and equipment for teachers. They dialog and discuss this issue. According to the simple rating system presented in the directions on the form, they determine if this element is a program support or a deterrent. Consensus is the goal. If the element is a support place a (+) in the "Rating" column. If it is a deterrent, place a (-). A third option is to place a (0), indicating an irrelevant issue. Some of the 15, though very few, may not be relevant for a particular program. For example, materials and equipment for parents (the third element listed) may produce a (0) when a high school staff is assessing its writing program.The last column offers the opportunity to briefly comment about an element. It is important to use this column if a particular rating produced a (-), because the group will want to document why that rating was given.
Once in a while, hopefully not often, a rating of (+/-) will be given. This happens when the group cannot reach consensus for a particular element or when there are particular strengths and, at the same time, particular weaknesses that the group wishes to cite. When this occurs, definitely explain your rating in the comment column.
Owning the Problem
Proceeding through the process you will discover two delicate but critical elements on the reverse side of the form. These are item 12, teachers' knowledge and skills, and item 13, teachers' receptivity to the program. These are sensitive items because they are asking the group to self-reflect and own up to possible deterrents of which they may be a part. For many, this is uncomfortable. However, it is essential that the elements be addressed.I'm going out on a limb on this one, but I've been known to do that on more than one occasion. There aren't many things I'd bet my life savings on, but this is one of them. I've been in the program evaluation business for about 30 years so I've been around the block a few times, and with conviction I make the following statement: There will never be school improvement on any campus whose staff, teachers and administrators, are unable or unwilling to self-reflect. Because if a person is not willing to self-reflect, they will automatically be into denial and blame. And an organization can never improve by blaming others for their condition.Understand that I'm not suggesting that one must identify teachers' knowledge and skills or teachers' receptivity to the program as deterrents in order to be honestly self reflecting. Most of the time, an honest and objective (+) would result when considering these two issues. But what of the few times when a (+) is not the "correct answer?" Will a staff be insightful and courageous enough to uncover the deterrent? In the scenario about Writing Across the Curriculum, who do you suppose identified teachers' knowledge and skills as a primary deterrent? Was it the superintendent or other figures in the power base of the organization? No, it was the teachers, themselves. When they identified it, the school was on its way to improvement.Notice also on the reverse side of the form that there are two spaces for an organization to add customized elements of their particular program to the list, if they deem them important. If you choose to do this, fine. Just make sure that what you add is something over which you have control or can do something
about. Adding something over which you have no control disempowers the organization that is working toward improving program quality.
Closure of the Analysis of Process Activity
When the group is finished and there are about 10 minutes or so left (the exercise may go a bit over the established one hour limit, but as long as they have stayed on task the whole time and have not contracted a case of analysis paralysis, that's okay), close the exercise in the following manner: Under the heading, "Setting Priorities," copy your ratings from above. There's no new work here. When that has been completed, consider all your ratings and identify what you believe to be your top three supports and, in rank order (1st, 2nd & 3rd) your top three deterrents, and don't hurry through this last portion. This aspect of the exercise is critical because you are identifying those program elements that you will be addressing in an effort to improve the quality of your existing program.
The activity is now completed.
Suggestions for Using the AOP
If you like this process, there are some suggestions I would offer to enhance the activity. These comments and suggestions come from a considerable amount of experience that I've had with using the instrument.Consider these, in no particular order of importance:
• Make sure you are working with an ongoing program
Some teams have tried to use the form during the planning process but before it becomes implemented; that doesn't work. You can't assess something that isn't happening.
• Be Patient..... Trust the process. Remember, this is not a questionnaire!
Diagnosing a program in a structured way is not something we typically do. Therefore, while the exercise may look good on paper, it probably won't work very well the first time or two you try it; when talking to your staff about the intended use, my suggestion is to teach your teachers by using a relative strong program in which teachers' knowledge and skills are a significant support. Don't start with a troubled program in which teachers' lack of skills is a deterrent. Teachers don't need to feel like they're on the hot seat when learning something new.
• Be sure to assign the facilitator and time keeper roles; it's what makes the activity work. Do not get into a solution mode.
Using the AOP, it is so common for a group to have identified a "big ticket" deterrent, and rather than marking it a minus and moving on, to get into a solution discussion. When this happens, we have just diluted the power of the diagnostic activity. You can't diagnose and solve simultaneously.
Focus on supports and deterrents being relative. All programs have supports and all programs have deterrents. The goal is not to obtain all (+)'s.
When you begin using the AOP, staff, quite understandably tend to treat it as an evaluation rather than a diagnostic activity. Thus, any minus serves as some kind of blight upon the quality of the staff. Emphasize the word, "relative."
Do not have the group fill out the form alone and then bring their ratings to the group.
To save time, sometimes this is done. I advise against it. We want the ratings to surface as a result of group dialog and discussion in which ideas build on each other. When we bring our ratings to the table, we tend to try to build our case so that we can be right, rather than making it a group decision.
• Do not use "language arts" as a program.
Language arts has too many parts for the exercise to work well. We have reading, writing, speaking, listening, along with language mechanics, spelling, etc. Using Language arts as a program will result in an unsatisfactory experience.
• Focus solely on the program being assessed. Don't allow the "halo effect" to take over.
In research lingo, the "Halo effect" occurs when a rating is given because of our general feelings rather than the specifics which are being addressed. A good example would be a very competent staff giving itself a (+) because overall, they are very good, but failing to realize that for the specific program being assessed, staff knowledge and skills would really be a (-).
Customize the form to fit your situation. But be certain to include only those elements over which you have control.
I alluded to this earlier. This form is for you. Change it any way you like. But to include elements over which you have little control (like monetary issues), reduces the impact of the exercise. The message then becomes "We have a significant deterrent over which we have no control that is preventing us from continuously improving. Therefore, we can't improve." We really don't want to send ourselves that message.
An Invitation
Of all the above suggestions, the call for patience in learning this process is probably the most important. You will discover when working with the AOP some things that work well and other things that don't work too well. Hopefully, the AOP will not be seen as just another thing to do, among the myriad of other things we are doing. If you want to get philosophical about it, diagnosing program supports and deterrents before arriving at a solution is really changing the way we do business. Jumping to solutions without analyzing your program first will, more often than not, produce "writing across the curriculum"I would be pleased to have an extended conversation with you regarding the use of the AOP or any other aspect of defining what we mean by "the quality of an educational program." Call anytime.
Jim Cox
714-281-7150
KeniC@aol.com
