All posts by Alan Broomhead

The Accreditation-Ready Program

There are few obligations for faculty and staff that cause knots in the stomach and departmental wrangling than preparing the accreditation self-study. It is often viewed as a burden, a distraction from everyone’s ‘real’ work, and a process of bureaucratic box-checking or of trying to fit the round peg of the program into the square hole of accreditation requirements.

In Five Dimensions of Quality, Linda Suskie draws on years of experience with accreditation, institutional and program assessment, and accountability to re-frame the role of accreditors as “low-cost consultants who can offer excellent collegial advice” (p. 245) to schools and programs seeking to demonstrate their value to stakeholders in an increasingly competitive market.  Accreditation should be viewed not as an imposition of alien practices on an established program, but as a way for a school or program to gain  external affirmation of already-existing quality. The challenge is not to make the program ‘fit’ accreditation standards, but actually to be a quality program and demonstrate that quality.

Accreditation success, then, flows naturally from the pursuit of quality, and is not an end in itself. But what is quality? Suskie breaks it down into five dimensions or ‘cultures’:

A Culture of Relevance
Deploying resources effectively to put students first, and understand and meet stakeholders’ needs.

A Culture of Community
Fostering trust among faculty, students, and staff, communicating openly and honestly, and encouraging collaboration.

A Culture of Focus and Aspiration
Being clear about school or program  purpose, values, and goals.

A Culture of Evidence
Collecting evidence to gauge student learning and program or school effectiveness.

A Culture of Betterment
Using evidence to make improvements and deploy resources effectively.

Fostering these cultures is the work of leadership, since they require widespread buy-in from all stakeholders. The challenge in many institutions is institutional inertia, as Suskie points out in her chapter, “Why is this so hard?” Faculty, staff, and governing boards may feel satisfied that the school’s reputation is sufficient for future success; resources – especially money and people’s time – may not be forthcoming; faculty and staff may live in comfortable isolation from the  real-world needs of students; there may be an ingrained reluctance to communicate successes; there is frequently resistance to change; and siloed departments in programs and institutions make across-the-board cultural change difficult to pull off.

The question administrators and faculty should ask themselves is, “Do we put our efforts into pursuing quality, or into maintaining our accreditation?” Suskie’s book presents a convincing case that working on the former will make the latter much easier and will result in quality rather than box-checking. For its straightforward prose (including jargon alerts scattered throughout), its sound advice, and its call for schools to demonstrate quality in a highly competitive environment, Five Dimensions of Quality should be a go-to resource on the reference bookshelf of decision-makers and leaders in higher education programs.

Suskie, L., Five Dimensions of Quality, Jossey-Bass 2015

More of my education-related book reviews are at Amazon.

Challenge and change in intensive English programs

From left: Bill Hellriegel, Carol Swett, Michelle Bell, Amy Fenning, Alan Broomhead

Challenges over the past few years have deeply impacted intensive English programs, forcing irreversible changes in their organizational cultures that result in anxiety and tension, but also innovation and adaptation. That was the theme of a panel session, “Organizational Culture in University and Proprietary IEPs: Challenges and Changes,” presented by Michelle Bell (University of Southern California), Amy Fenning (University of Tennessee at Martin), Bill Hellriegel (Southern Illinois University), Carol Swett (ELS Language Centers at Benedictine University, Illinois) and myself at the TESOL International Convention on March 28. Recognizing the cultural types of IEPs and how they are affected by changes is the first step in adapting and surviving in an increasingly competitive field.

IEP cultures can roughly be divided into collegial and managerial types, following Bergquist and Pawlak’s (2007) typology of academic cultures. A collegial culture, more likely to be found in a university-governed IEP, is faculty-focused, with faculty scholarship and teaching, academic autonomy and freedom, and faculty ownership of the curriculum as the organizing principle. A managerial culture is administration-driven, motivated by considerations of fiscal responsibility and effective supervision, and organized by systems, processes, and standards.

The massive shift to accreditation in IEPs has moved collegially-oriented programs in a managerial direction. Faculty are required to plan, teach, and assess in compliance with program-wide student learning outcomes; policies and procedures have to be written and followed; and program success is measured by data, which has to be systematically collected, analyzed, and evaluated. Proprietary IEPs are seeing a a shift in the other direction: faculty standards require minimum levels of certification, experience, and ongoing professional development, and these are affecting faculty hiring and employment practices in many proprietary programs.

The severe enrollment challenge of the past two years has also affected both types of program. University IEPs are becoming more revenue-driven and entrepreneurial, actively seeking new recruitment partnerships and designing new programs – such as short-term high school programs – to respond to changing demand. Faculty may have little say in these initiatives. Meanwhile, proprietary IEPs are increasingly developing conditional-admit and TOEFL-waiver agreements with partner universities, requiring them to make programs more academically-focused and hire masters-level teachers who are qualified to teach English for academic purposes.

These are ground-shifting developments, and program leaders who recognize the need to address profound cultural change in their organizations – and not just surface-level adjustments – will be in the strongest position to navigate these challenging times.

Reference
Bergquist, W.H. & Pawlak, K., Engaging the Six Cultures of the Academy, Jossey-Bass 2007

Is language proficiency measurable?

For the past few years, ESL teachers have been pushed to focus their efforts on helping their students achieve ‘measurable objectives.’ Am I the only person who finds this a strange idea? Measuring something is a matter of determining how much of something there is. To measure, we need a unit of measurement: inches and feet, centimeters and meters, pounds and ounces, grams and kilograms. By agreeing on standardized units of measurement, we can determine, objectively, the quantity of something.

Could we use this approach to evaluate a work of art or a piece of music? What would Van Gogh’s Starry Night or Mozart’s Eine Kleine Nachtmusik measure? What unit of measurement would you use? Impossible, because we are not dealing with quantity (the amount of paint or size of canvas, or the number of notes) but with quality. There is no unit of measurement for quality.

When we want to assess and evaluate language are we dealing with something more like a distance or weight, or like a work of art? Is it the quantity of language or the quality of language we want to know about?

If you believe it is quantity, then you might say we can ‘measure’ the number of words a student has learned, or the number of grammar points. These don’t work as units of measurement, though, because defining exactly what is meant by ‘learning a word’ is complicated, as language teachers know. A student may be able to say it but not spell it, may use it but in an inappropriate context, may not recognize it when written down but may hear it in another person’s speech, may forget it on one occasion but recall it on another.

The psychometric testing tradition has given us tests which appear to measure learners’ language ability by assigning a score and thus appear to be objective. Classroom assessments by teachers are often regarded as a subjective second-best.

We should move away from the notion that language proficiency is measurable, and that test scores give us an ‘objective measure’ of a learner’s ability.  Language should be evaluated qualitatively, by people, using rich description rather than fantasy units of measurement that give the false impression of objectivity.

The Inadequacy of “ESL” for International Student Preparation

Wrapped up in the term ESL (English as a Second Language) is an assumption that language, above all, is what students need to succeed in an English-speaking environment. The same kind of assumption can be found in the name of the most popular standardized U.S. admissions test for international students, the TOEFL (Test of English as a Foreign Language). The CEFR (Common European Framework of Reference for Languages) lists levels of language proficiency by skill, and many ESL programs continue to organize their curricula on the basis of Listening, Speaking, Reading, and Writing skills. The field of SLA (Second Language Acquisition) is a major feeder discipline in ESL teacher preparation programs.

A focus on the acquisition of language skills gets us only so far if we are preparing an international student for academic work in an English-speaking setting.  One thing among very many that this student needs to do is to read a text critically and offer an original, well-thought-out, supported, and argued response. The student may need to argue that response in class, and defend it against other points of view, in an assertive yet diplomatic manner. To be taken seriously, the student will need to behave in what is recognized as a normal and appropriate manner in that environment – and know when and how to revert to a more informal style when class ends. All of this goes far beyond language skills.

What this student needs to learn is what James Paul Gee in Social Linguistics and Literacies refers to as Discourse (with a capital D). Discourse “is composed of distinctive ways of listening/speaking and often, too, writing/reading coupled with distinctive ways of acting, interacting, valuing, feeling, dressing, thinking, believing with other people and with various objects, tools, and technologies, so as to enact specific socially recognizable identities engaged in specific socially recognizable activities” (p. 152). These are less language skills than “social practices into which people are apprenticed as part of a social group” (p. 76). As we move in different Discourse communities, we need to know how to play our part and be recognized as a legitimate member of each community. Discourses are mastered by “enculturation…into social practices through scaffolded and supported interaction with people who have already mastered the Discourse” (p. 168).

This helps us understand why any program of learning that reduces preparation to language skills is inadequate. Students need to learn the ways of interacting, believing, valuing, and effectively being in the academic Discourse community. University IEPs (intensive English programs) teach English for academic purposes, but they still largely identify as English language programs with language-based missions, their faculty members have degrees in teaching English, and classes are often language skill-specific. They are often isolated from the rest of the campus, and therefore don’t allow for the kind of apprenticeship into the social practices of the campus that would make international students full members of the Discourse community.

In order to address this wider understanding of international student preparation:

  • Intensive English programs should ensure their missions, their curricula and teaching, and their names, encapsulate the full meaning of international student preparation – not simply ESL.
  • University administrations should make international student preparation a task for the whole university, supported by, but not the sole responsibility of, an intensive English program. The IEP’s efforts should be integrated into a campus-wide strategy for international student preparation.
  • Universities should not expect that simply raising the required TOEFL scores will improve international student outcomes – students need induction into the Discourse community, not just a higher TOEFL score.
  • ESL teacher preparation programs need to include coursework on social literacy and in preparing students to enter and successfully navigate their target Discourse communities.

Some of this has already been achieved. Many IEPs recognize their wider mission of orienting students into academic culture, and more recently,  pathway programs have been structured to provide ESL support alongside credit-bearing classes that, in theory at least, offers an apprenticeship into the academic community. But there is a long way to go before the notion of Discourse communities drives international student preparation beyond the inadequacy of “ESL.”

Reference
Gee, J.P., Social Linguistics and Literacies, 5th Ed., Routledge 2015

Working with the tension between language test validity and reliability

The combination of validity and reliability is the holy grail when it comes to language assessment, yet these two qualities are always in direct tension with each other. This can create a challenge when English language programs try to put in place effective measures of language learning, and especially when they have to convince their accreditors that they’ve done so. Student achievement standards are frequently not met in accreditation reviews for precisely this reason.

An assessment is valid if it measures what it is supposed to measure. So, a multiple choice test is generally not a very valid means of testing speaking ability; nor is a gap-fill test a very valid way to determine whether a student has learned to use a grammar structure in communication. On the other hand, a student presentation might serve as a useful basis for a valid assessment of speaking ability, and a speaking or writing test that elicits a target grammar structure would bring to light a student’s ability to use grammar.

An assessment is reliable if would yield the same results for that student if administered by a different person or in a different location. An in-class presentation or role-play assessed by the class teacher is vulnerable to having a low level of reliability, since the test conditions would be difficult to reproduce in another class. The TOEFL iBT is probably the gold standard for test reliability, with extremely detailed protocols for ensuring the uniformity of the test-taking experience for all students, and two-rater grading of written and spoken assignments.

You can probably see the tension: the greater the validity, the harder it is to attain reliability; the greater the reliability, the harder it is to make the test valid (in the three-hour iBT, the test taker is not required to interact with a single human being).

To increase the reliability of valid assessments, programs can:

  1. use a common set of learning objectives across the program and hold teachers accountable for teaching to them
  2. use standard assessment rubrics across the program
  3. calibrate grading through teacher training
  4. have more than one person assess each student’s performance.

These measures might generate pushback among faculty in some university programs.

Unfortunately, there aren’t any great ways to increase the validity of highly reliable of achievement tests. Doing so would require standardizing the teaching – teaching directly to the test – which nobody in an IEP wants, except in a course specifically for test preparation. Programs that use external standardized tests for level promotion are not using a valid means of assessing what was taught (since the test makers don’t know what was taught).

Instead of seeking the absolute standard of ‘assessments that are valid and reliable,’ we need to

  1. start by creating assessments that are valid – that measure precisely what was taught and was supposed to be learned; and then
  2. design and implement measures to reach as high a level of reliability of those assessments as is possible and practical.

Using this approach is a recognition that you can’t have it all, but you can work within the tension of validity and reliability to reach a satisfactory compromise.