It is an honor to address you this morning. I hope our time together today is productive for us. My aim in the first part of my presentation is to offer some perspectives on the topic of consistency in accreditation recommendations. In the second part, I would like to offer some strategies to foster consistency. We will increasingly be interested in this topic as the processes of accreditation in our associations become more established and mature. We will increasingly be asked to defend our practices and procedures for dependable monitoring, intelligent scrutiny, and consistency. Accreditation is all about evaluation with a view to continuous improvement; it is peer reviewers evaluating programs’ achievements of missions and purposes, and providing consultative advice to promote continuous improvement in time toward declared goals.

We must ask three questions when talking about consistency: The first question we must ask is "Are you making decisions based on your criteria or standards?" The second question we must ask is "Are you interpreting the criteria according to the intent of the criteria or standards, rather than from a personal perspective?" The second question we must ask is "Are your accrediting recommendations defensible over time?"

We agree that accreditation criteria or standards are designed to be met. We expect them to be met, and we are vigilant to make sure that bias does not enter. We agree that we want consistency in making recommendations on a single set of policies because when we talk about consistency, we mean that our accreditation recommendations are credible, and are appropriate to their contexts.

Let me consider with you for a while what we don’t mean when we say "consistency." We do not mean unanimity of opinion. Evaluative processes in accreditation are primarily deliberative in nature and do not lend themselves to a strictly numerical scoring system. Therefore, we don’t mean unanimity of opinion when we talk about consistency. This would make for centralized control of education, and that kind of centralization militates against the diversity in American education

When we say we want consistency, then, we mean that we want team judgments to be credible – judgments that are appropriate because they have been arrived at by a reliable application of the criteria. Earlier I posed the issue of consistency as the question "Are you interpreting the criteria according to the intent of the criteria or standards?"

With that in mind, let me take, for instance, the inconsistency in judgment of two teams each reviewing different programs that have similar issues. If the team judgments are supported by solid rationales based on the criteria or standards, and reflect the contexts of the programs – that is to say, they are mission-based -- the variability, or what we may perceive as inconsistency, in judgment may arise just because of the different contexts (missions) of the program. On the other hand, if two teams, each reviewing programs that have widely different issues, arrive at same or similar judgments, and at the different review levels in the Accrediting Association the judgments are seen not to be based on the criteria, then we have a situation where the criteria or standards are applied minimally. Consistency, then, is not applying criteria or standards as a formula, or as imperatives, but about making decisions based on the intent or educational value of a criteria and applying it to the program being evaluated in its context, so that the judgment made is credible. Consistency is more easily understood then as credibility because when we view the word from that perspective it reflects our desire that judgments be appropriate to the mission-based contexts in which they are made. The challenge is to ensure that the intent – the educational value – of a specific criteria or standard is reflected in the team’s evaluation of the program. When we explain and nurture common understandings of the intent, or value, of criteria among programs and among peer reviewers, a team can look at how well a program has addressed the intent of the criteria within its own mission and purposes. When we fail to explain the intent of the criteria, we apply only the language of the criteria (which often is interpreted differently by different people) not their intent. The educational intent of the criteria is essentially a guide, and the context of a program the specific instance in which it is to be interpreted.

How do we explain and nurture common understanding of the intent of criteria so that we can foster increasing consistency in their application? First, there is a need for consistency of definition or clarification of terms. Let us take "assessment" as an example. Is there consistency in what programs mean by "objectives" and "outcomes" of assessment? "Objectives" are what you want graduates to be able to do two to three years out of graduation. They are related to academic competencies. "Outcomes," on the other hand, are related more to the learning achieved in the program. If we find that objectives are not being met, then we must go back and take a look at the program’s outcomes and make adjustments there. Many programs often think about assessment only in terms of surveys. Others have being doing good assessments, but are not systematic about it. Definitions or clarifications of key terms, then, are absolutely necessary for promoting consistency over time.

We can also look at the issue of consistency from the perspective of what can cause inconsistencies. Inconsistencies arise when there is:

• Failure to assess if criteria are fully met. The Higher Learning Commission of the North Central Association’s new team report format requires a team to write an Assurance section where a team provides evidence that an institution meets the criteria for accreditation.

• Failure to discover that a program has not followed its own stated policies and procedures.

• Uneven interpretation of criteria because they are unclear. Different teams may evaluate differently because the criteria are ambiguous.

• Requiring criteria or standards that are impossible to achieve, so peer evaluators compensate for the difficulty.

• Using one criterion in preference to another because a team member argues for its dominance.

• Widely different processes followed in different programs with state, federal, or licensure requirements. This could be a problem if it is claimed that a graduate doesn’t meet a specific required level of competency.

• Accreditation criteria that are internally inconsistent, or inconsistent with federal regulations. Peer review teams may interpret to compensate for that discrepancy.

Now let me talk about evaluative processes. Accreditation evaluative processes are primarily deliberative in nature, so we have to allow for a certain amount of latitude in interpreting accreditation criteria. We can do a more effective job of training our evaluators and institutions to better understand the intent or clear educational value of criteria, and, equally importantly, the importance of context in making recommendations. Peer reviewers on evaluation teams have legitimate differences of opinion and we have to provide some latitude for those differences, but always within the intent of the criteria or standards.

It should also be noted that accreditation systems have a multi-tiered structure. This ensures that each evaluation is exposed to a variety of perspectives and thus the system guards against bias for or against a program or school on any basis. I see our task as marshalling our efforts to nurture common understanding among programs, staff, and evaluators of the intent of accreditation criteria. Achieving consistency is a slow and deliberate process.

Let me give you an example of a recent first-hand experience relating to a panel’s consideration of monitoring alternatives. An evaluator’s panel on which I served was discussing the difference between requiring a Progress Report or a Focused Visit to address a concern. One member on the panel felt that a Progress Report focusing on how the institution was addressing the concern would provide the accrediting commission and the staff a good idea of the progress being made on the issue of concern. Another member felt that the concern was of sufficient importance to a specific criterion that a Focused Visit was the more appropriate monitoring procedure to recommend. However, it wasn’t until a member of the panel pointed out that the concern was of sufficient importance to one criterion, and that the intent of a Focused Visit is to get an on-site evaluation of how well the institution was addressing this concern, whereas a Progress Report provides only the institution’s perspective on its own progress on the issue of concern, that the panel saw the rationale (and value) for recommending a Focused Visit. Both would address the progress made, but one was preferred because it underscored the importance of the issue of concern to meeting the criterion. Moreover, a Focused Visit team would be in a better position, on-site, to evaluate the progress made on the issue of this concern.

In the second half of my presentation I’d like to talk about some strategies to foster consistency. Many members in the audience probably have in their accrediting associations a Review Committee or Actions Committee whose task is to review evaluation reports for consistency against criteria. Such a committee with a charge to focus on consistency and application of criteria may recommend making some clarifications or adjustments in team recommendations. Another function of such a committee may be to provide consultative advice to the program which the Executive Director of the accrediting association provides to the program or an institution. The committee as well may use what can be called an Action Letter to the program. Many members here may have something similar in place. The Action Letter, while communicating the final accreditation decision of the accreditation Board or Commission, may also highlight issues of concern that a team has indicated. It can include a specific paragraph communicating the Review Committee’s own concerns. The purpose of an Action Letter may be several-fold. It is to assure that the program meets accreditation criteria; it is also to point out concerns or deficiencies to the program, and equally important, it is to consult with the program about continuous improvement over time against its own stated goals. It serves to "Close–the–Loop," holistically, so to speak. As the Review Committee reviews team reports submitted over a year or two, it may find that certain issues surface because of inconsistent application of the criteria. Issues of consistency are generally very subtle. Those issues then can become the focus of peer reviewer or evaluator training, and of consultative assistance to programs coming up for review. The Action Letter, therefore, becomes a more malleable tool in speaking to programs. It facilitates building, over time, common understanding among programs and evaluators of the intent, or educational value, of accreditation criteria or standards. Thus the focus remains, as it should be, on future oriented continuous improvement in programs.

We must recognize the pressing need for more consistency in accrediting recommendations. Every member here uses training as a way to inform and educate consultatively its accreditation association’s objectives through workshops, listserves, focus groups on recurring topics, and guidance on developing judgment. We have to bring people along – staff, evaluators, decision makers –to show that we are making decisions based on the criteria, and that there is a pressing need for common understanding of the intent of the criteria. Let me, therefore, now describe some strategies for you to consider for promoting consistency.

• Consider for example, putting up electronic sample questions on topics of discussion on important areas such as administration, finance, student support, distance learning.

• The team report format or general requirements for the accrediting association’s report could be made available electronically. This would help the staff to track recommendations over time -- rather than just anecdotal evidence -- and find issues of inconsistency, which then can become the focus of a "Nuts & Bolts" session at the Association’s annual workshop. The objective of a "Nuts & Bolts" session may be to find a norm – what issues are important to have consensus about? From the evaluator’s perspective, the question would be "Is this important in how to think about the program?" From the decision makers’ perspectives -- the Commission or the Board -- the evidence may show that team recommendations are increasingly defensible and inconsistencies on the important issues – governance, planning for the future, finances, for example – are increasingly reduced.

• Take the thorny area of evidentiary proof as another area where we can promote a consistent approach:

Some other questions to consider are:

• Does a criterion have minimum mandatory or core requirements for accreditation? Is the floor for expectations commonly understood? If the language of the criteria is ambiguous on this point, then we may wish to revisit how the criteria are written.

• Does a criterion define or describe good educational practice? What is the intent – the educational value – implicit in it?

• Does a criterion or standard have an "end point" (defined parameter) which is clear and unambiguous? Is it written so that a team can judge that a program meets or does not meet the criteria? Is the intent of the criteria met in the program’s context? I should note here that a program’s context is essentially its mission, purposes, and resources.

• How much is too much latitude in interpreting guidelines when they are used?

• How do we avoid using guidelines as imperatives?

• Do we have some common understanding of the different triggers for different monitoring processes?

Let me continue to present some additional strategies to improve consistency. You may be familiar with many of them, but presenting them together should show the very close relationship among them. I believe several if not all of them have to be done in some dimension when our objective is to increase consistency. We may not have complete consistency, but we will certainly increase and continuously improve in achieving it. We must take the long view in increasing it.

• Consider a web site that posts important and frequently asked questions. Questions may be drawn from one year’s evaluations. Visitors to the site may be invited to comment.

• Consider providing information about trends in higher education with links to significant documents -- for example, innovations in collaborations, partnerships. ASPA could play a very effective role in facilitating a small clearing house of information about national trends focused on specialized and professional accreditation.

• Review some areas for improvement discovered in the past year’s team reports as critical questions. These may be particular issues about which peer reviewers and programs coming up for review should become increasingly aware.

• Address some report writing pitfalls. For example, when communicating a concern or challenge, it is the "so what" that points to the failure of not meeting the intent of the criteria. If the concern statement does not include a "so what’" it may not be actionable by the program.

• Provide electronic tutorials on specific issues through examples of good responses as opposed to vague, non-constructive responses.

• Foster more useful and constructive report feedback without getting into an inspector mode of evaluation. Focus on what is easily doable –- continuous improvement.

• Develop a general outline for all reports. A general outline helps to promote consistency.

• Create sample questions on important topics such as program administration, funding, student services, distance learning.

• If there is a workbook, ensure that criteria and workbook reflect the same intent, or educational values to avoid ambiguity and different interpretations.

• Revisit the explanations of terms, procedures, and expectations that we have in our Handbooks for programs and peer-evaluators, checking clarity and internal inconsistencies. There is a limit, of course, to the extent to which we want to formalize the process. We would not want to take any steps that may obstruct the deliberative nature of the peer review process that lies at the heart of the accreditation process.

Our accreditation systems, as I noted before, are multi-tiered structures that expose each level of evaluation to a variety of perspectives and therefore guard against the possibility that final judgments/ recommendations may be biased for or against at any one level. Any single set of peer reviewers -- other than Boards and Commissions -- are not decision makers in accrediting associations.

I strongly believe that when we want to increase consistency in judgments we mean that we want team judgments to be credible – judgments that are credible because they are based on an informed application of criteria.