Category Archives: For Schools

I can advise on program management, curriculum, faculty issues, and business development.

The baby and the bathwater

If you read anything about curriculum design these days, or attend a presentation or workshop, you will learn only one model. Backward design starts at the end, defining student learning outcomes, then working backward through assessment, teaching and learning objectives, content and sequencing, and finally teaching and learning. This approach to curriculum design is so pervasive that anyone new to education might think there is no other way.

Thanks to the recently published second edition of Jack Richards’ book Curriculum Development in Language Teaching, we can learn about, or be reminded of, the alternative. Although it was never recognized with the name, forward design took the opposite approach: decide on your content and sequence it, teach, assess, and evaluate with a grade. This used to be the standard way to design  and teach a course in higher education. And the fact is, many teachers who used this approach continue to do so, or try to do so, in tension with an institutional backward design ethos promoted by accrediting organizations.

Backward design has the benefit of identifying (in theory anyway) student needs, developing measurable learning outcomes, and demonstrating program quality through analyzing and publishing student success rates against those outcomes. There is accountability for student success, which is important in an increasingly competitive environment in which customers (students and their sponsors) demand transparency in results. Yet as we are reminded in Richards’ book:

“The experience of language teachers today is often one of diminished classroom autonomy and of being managed by business-savvy administrators.” (Hadley, 2014, cited in Richards, 2017, p. 228)

This is because teachers are increasingly told to work with standardized outcomes and learning objectives, demonstrate that their assessments address the student learning outcomes, and use textbooks that deskill teachers by driving many of their instructional decisions (Richards, p. 247).

If backward design has introduced an obsessive focus on outcomes, or product, forward design was always much more about the educational process. A process curriculum:

“…is person-centered, considers users’ needs, identifies problems rather than rushing to solutions, and does not rely on top-down mechanistic models but is a process that works towards interaction between participants at all levels.” (Kennedy, 2013, cited in Richards, 2017, p. 227)

The emphasis is more on what teachers might call the art of teaching, making meaningful experiences for students, and letting the teaching and learning follow, to an extent, students’ needs and interests as they arise during the course. In a process approach, students and teachers form a learning community that explores together, often with an unclear destination. While backward design requires assessment of students’ achievement of learning outcomes exclusively, forward design grades participation, homework, and attendance, because these are indicators of students’ engagement in the process of learning.

While I don’t think we can or should return to a purely process-focused approach, we should consider what is lost when we throw out the process approach baby with the forward design bathwater, and embrace a product-based approach too strongly. Now that backward design is established as the accepted way to design curriculum, I hope that we can start to talk about how to maintain art in the process of teaching, one that recognizes teacher creativity, responds to the students in the class and the needs of the particular group, and provides a unique and unrepeatable experience for learners.

Thanks to Richards’ book, we may be able to start having that conversation.

Jack Richards’ book Curriculum Development in Language Teaching is published by Cambridge University Press.

The jury trial and the language test

Background vector created by macrovector – http://www.freepik.com

I recently had the opportunity to serve on a jury in a case in which a young man was accused of ‘operating under the influence,’ i.e. driving a car while under the influence of alcohol. There was only one witness, the police officer who arrested the young man. In order to establish that the young man was under the influence of alcohol after pulling him over, the officer had conducted what is known as a field sobriety test, which usually has three components:

  1. The horizontal gaze nystagmus test, in which a person’s eyes have to track a moving object.
  2. The walk-and-turn test, in which the person walks heel-to-toe for nine steps, pivots on the left foot and walks back heel-to-toe for nine steps.
  3. The one-leg stand test, in which the person stands on one leg and counts to 30.

Each test has its criteria for success and failure. For example, you may fail the one-leg stand test if you have to use your arms to balance yourself, or if you put your foot down before counting to 30.

In the jury room, we six jury members found that the prosecutor’s evidence was based almost entirely on the results of the field sobriety test, which the young man failed. We had been instructed by the judge that we should presume innocence and find the man not guilty unless the prosecution could prove ‘beyond reasonable doubt’ that the young man had been driving under the influence of alcohol.

We jury members were not permitted to do any research on the field sobriety test, and none of us had any information about how useful it was, though we knew that it is widely used. Most jurors had doubt about the evidence, but one juror believed we should base our verdict on the result of the field sobriety test.

I found myself in the position of explaining that we cannot assume that tests give us useful information and I used the notions of validity and reliability to explain it. In this case:

  1. Is the field sobriety test valid? Does it measure what it purports to measure? (Note that the officer conducted only the walk-and-turn test and the one-leg stand test.) Were there other reasons why the young man – or anyone – might fail these tests other than being under the influence of alcohol? (It was claimed by the defense that the young man had ‘an issue’ with his left leg.)
  2. Is the field sobriety test reliable? Can we be sure another officer would have reached the same conclusion based on the young man’s performance on the test?

Given these questions, could we determine ‘beyond reasonable doubt’ that the young man had been under the influence of alcohol?

I am writing about this because in language teaching, we tend to place a lot of trust in the tests we use, whether made by an individual teacher, a program, or an international testing organization. Any really useful test should help us determine whether the test taker has attained certain knowledge or is capable of performing certain functions in the language. Think about any test you know – the TOEFL, TOEIC, IELTS, or those home-grown tests used in your program. To what extent are you able to say ‘beyond reasonable doubt’ that the person taking the test has acquired certain knowledge or skills? We know for sure that plenty of students with high TOEFL scores are not well-prepared linguistically to succeed on an American college campus, and this is a test that has a huge research effort behind it.

So I would ask that you advocate for more evidence than a single test to determine students’ language proficiency. Decision-makers such as college admissions staff should seek multiple sources of evidence that converge on a conclusion, not just a single test. They should have more than one person or entity reviewing students’ language ability. We should maintain a healthy doubt about the tests we use.

What was the verdict in this case? In the end, no matter what the truth was, we could not return a guilty verdict because the state had not proved beyond reasonable doubt that the young man had operated a motor vehicle under the influence of alcohol.

Naturally I rushed to my computer after the trial and googled ‘reliability of the field sobriety test.’ What do you think I found?

Forwards design, backwards design

Backwards-design curriculum is a relatively new approach to curriculum design that is finding its way into many disciplines. In English language teaching, backwards design originated with English for Specific Purposes courses (such as English for pilots or English for the food service industry) where it was important to specify what learners should be able to do following the course or program. In the US, it was given a boost by the accreditation requirements of ACCET and CEA, themselves subject to the mandates of a federal Department of Education that sought greater accountability from educational institutions, starting with the No Child Left Behind Act of 2001.

To understand backwards design, it helps to understand what it sought to replace. In a forwards-design approach to curriculum, the subject matter is broken down into its component parts and taught. At the end there is a test or other form of assessment to check what the students learned. The most obvious example in English language teaching is the traditional grammar syllabus, which organizes teaching grammar point by grammar point, and is still popular in many schools. In early versions of the communicative approach, grammar was replaced by communicative functions, but the approach was essentially the same. When I started teaching, I followed this approach: teach the points, then make a test.

Backwards design turns the process around. In this approach, you first analyze the needs of the students. What are they learning for? What do they need to do? This results in an overall goal for the course or program, and learning outcomes (often referred to as Student Learning Outcomes or SLOs) that state what students are expected to know or be able to do after completing it. Then you determine what would be acceptable evidence of achievement of this knowledge or ability, and design a means of assessment. Following that, you break the outcomes down into teaching and learning objectives and order them into a logical sequence. Finally, you decide how you are going to teach the knowledge and skills – your methodology.

Backwards design has been disruptive in many schools. Faculty with many years of experience are used to a forwards-design approach, and have developed their preferred ways of teaching around this approach. They may feel that the classroom is a place to explore new knowledge with students, they may want to meet students where they are, and may not want to define in advance exactly where those students will end up. Many are uncomfortable with trying to shoehorn their established teaching practices into a backwards-design course. Newer faculty are more likely to have been trained in backwards-design principles and accept them as natural.

However we feel about backwards design, it responds to a demand for greater accountability in education. This is the result of rising tuitions, a value-for-money orientation among students and their parents and sponsors, and a greater demand for demonstrable practical skills resulting from education. Love it or hate it, teachers have to embrace backwards design and incorporate it into their professional practice.

What makes a school?

We are used to talking a lot about quality in education. It used to be normal to describe quality in terms of inputs: faculty to student ratios, faculty degrees, school facilities, test scores of the incoming class, and so on. More recently, we have been pressured by government departments, funding agencies, and accreditors to prove our quality in terms of outcomes: can-do statements, demonstrable skills gained, behavioral changes in our students at the end of their course or program.

The input-outcome paradigm for determining quality is adequate enough if we are in a production mindset. In this mindset, education is analogous to the production of goods or services. “Our shoes are made of the finest Italian leather” is an input-based quality claim. “Kills 99.9% of bacteria” is an outcome-based quality claim. Similarly, “highly qualified and friendly teachers” is an input-based quality claim. “Our Academic English program will prepare you to succeed at a university” is an outcome-based quality claim.

I wonder if this paradigm tells the whole story about the quality of an education? This is important to consider for intensive English programs (IEPs), because they are increasingly competing against other models of English language education and training, such as in-country classes, online tutoring and courses, even apps on devices. The producers of these alternatives can point to their inputs and outcomes and on that basis apparently offer a viable alternative to an intensive English program.

But I want IEPs to revive the notion of the school, a concept that is far broader than mere production or educational delivery. If you think about your own educational experiences which were the most memorable? Which shaped you most as a human being? Were you most influenced by a program that had clearly defined student learning outcomes? Did you learn the most from the teacher who was the most highly qualified? Unlikely.

A school is a place where community is formed. Diverse (however you choose to define diverse) students, teachers, and staff, come together in the shared enterprise of teaching and learning. There is social interaction, friction, the challenging of dearly-held beliefs. There is laughter, disappointment, joy, and frustration. Students encounter teachers with idiosyncrasies that they will never forget. A story heard sticks in the mind forever. A kindness is extended and remembered.  All of this is the quality of an education that is not recognized by the production paradigm of educational quality. It cannot be measured. You cannot really put a price on it.

Think about this kind of quality when  you go to your school in the morning. You still have to hire qualified teachers and measure student learning. But you have the opportunity to create unique, precious, and lasting experiences for your students, staff, and faculty. This is what really makes a school.

Outcomes are fine, but inputs make the difference

Alan speaking at the 2019 TESOL International Convention in Atlanta, GA

At the TESOL International Convention in March, I participated in a panel presentation on the future of intensive English programs. One of the themes I asked attendees to consider was user experience design, an increasingly popular concept in industry that emphasizes the creation of meaningful and relevant experiences for a product’s users. We are about to see, for example, the widespread introduction of self-driving cars. What will we do in our vehicles when we no longer have to concentrate on driving? Car manufacturers are increasingly turning their attention to promoting the experience users of their vehicles will have, rather than the car’s technical features.

The intensive English program (IEP) field can take something useful from the notion of user experience. In recent years, educators have been pressured to focus their efforts on student learning outcomes, as governments seek greater accountability from educational institutions. The obsession with outcomes has unfortunately led to the neglect of the quality of the educational experience, and ‘non-essential’ programs such as sports and the arts have been cut back in many school districts. IEPs have been swept up in the outcomes obsession, primarily through the requirements of their accreditors, who need to see measurable evidence of outcomes but have no standards relating to the quality of the students’ daily experience in their programs.

But this is a rough time for IEPs in the U.S. Student numbers are falling because of changing demographics in their sending countries, stronger English language programming in public schools across the globe, and above all, competition from other countries (such as the Philippines and Malaysia) and formats (apps and online learning). Simply focusing on outcomes is not the answer for U.S. IEPs. Each IEP can offer a unique experience to its students, an experience that can be personally enriching and be life-changing, can create life-long international friendships and networks, and can teach much more than language: intercultural communication and understanding, adaptability, and resilience.

If you booked a package vacation with a tour company, you would not expect the company to describe to you the anticipated outcomes of the vacation. You would expect that the elements you purchase, or the inputs – the destination, the tours, the hotel, the attractions – would offer an enriching and enjoyable experience. In their marketing strategy and program delivery, IEP leaders should pay close attention to user experience design, thinking about every aspect of the program from the users’ (the students’) point of view and working to make it the best possible experience for them. This is one way IEPs can distinguish themselves from the competition in an increasingly crowded global English language marketplace.

The panel presentation, “IEP? What will Intensive English Programs Look Like in the Future?” was devised and chaired by Jodi Nelms (University of Houston), and included contributions from Mary Reeves and Heidi Villenga (Commission on English Language Program Accreditation), Mark Algren (University of Missouri) and Scott Stevens (University of Delaware). 

Putting Students First

At a recent professional development session at Stafford House Boston, Miyo Takahashi Le and I presented some principles and practices of good service to intensive English program (IEP) students. The simple mantra “Students come first” particularly resonated with many in the audience. It means that in any policy or practice consideration, staff should always prioritize what is best for the students. Putting students first may seem obvious, but although individual staff and faculty approach their work with their students’ best interests in mind, in practice there can be institutional or structural impediments that inhibit good service. Here are three examples.

Silos
At one IEP, the Admissions team was responsible for first-day check-in of students for its semester-length program. The team had made the process – which included scanning passports and I-20s, checking students’ insurance, and ensuring that tuition was paid – highly efficient. There was no involvement from the Academic team in first-day check-in, because it was viewed as an Admissions process. Yet many students had academic-related questions when they came in on the first day, and there was no process to get those questions answered. Worse, for new students who had traveled thousands of miles to come and study at the program, there was little in the way of a warm welcome, no chance for students to meet their teachers, and only limited opportunities to start bonding with other students.

This was changed by having Admissions and Academics collaborate to develop a first-day check-in experience that included a warm welcome and conversation with faculty, and advisors on hand to answer students’ questions. It resulted also in the Academic staff and faculty being able to take care of some academic procedures (such as elective class selection) on check-in day, which was more efficient and of better service to students.

Breaking down silos and seeing first-day check-in as an institutional effort rather than the activity of just one department led to better service to students.

Prioritizing Staff or Faculty Interests
In an IEP that ran three semesters per year, the summer semester was set up differently from the fall and spring semesters. The daily schedule was shorter, with all days ending at 1:00 instead of 3:30, because students were restricted to one elective rather than two. The summer was divided into two six-week sessions, giving faculty the opportunity to teach less or concentrate their teaching into one half of the semester. This also meant that electives that were designed to be taught over 60 hours were crunched into 30 hours. None of this was great for students.

The original rationale given for the different summer schedule was that the IEP was running its main semesters in the fall and spring, and summer was seen as just an extra that was not taken as seriously. But students wished to study year-round, and there was no reason why the program should be any different in the summer. In fact, it looked suspiciously as though the summer had been designed for the convenience of the faculty rather than the good of the students.

After much discussion and a faculty vote, the summer semester was brought into line with the fall and spring semesters, creating a smoother study experience for students studying over several semesters.

Institutional Inertia
At a residential program, students were required to sign out when they left campus and sign back in when they returned. This was an onerous process that began twenty or thirty years before, and was intended to increase the safety of the students by enabling staff to check who was on campus at any time. Upon review, it was found that many students failed to sign out and sign in correctly, making the system ineffective. Given that the Student Service team’s mission had recently been updated to include empowering students and helping them to be more independent, the sign-out/sign-in system seemed outdated and intrusive. And on reflection, staff realized that the system had been introduced before the age of smart phones, which students now all carry at all times, making them easier to reach than ever before. The reason the sign-out/sign-in system continued was simply because that’s how it had always been done. While some staff had reservations initially, the burdensome sign-out/sign-in books were finally removed, and in a subsequent survey, students overwhelmingly supported the change.

Serving Better
No matter how much you may want to provide top-notch service to your students, impediments – such as silos, prioritizing the interests of faculty or staff over those of students, and institutional inertia – can get in the way of great service to students. Do you recognize any of these impediments in your program? How can you serve better?

 

 

Thank you to Ece Gürler of Stafford house Boston for devising and publicizing the session ‘How Can We Serve Better?’ 

What you should know about language placement tests

Most intensive English programs offer a placement test or a set of procedures to try and ensure that students are receiving instruction at an appropriate level. Minimally, a multiple-choice test is offered at the school on the first day,  or online. Other procedures typically include an interview and a written assignment that are assessed by the program’s teachers.

If your program is introducing, reviewing, or revising placement procedures, here are some important considerations.

Placement tests are not tied to your program’s objectives
If a program uses a commercially available test such as the Oxford Placement Test or the Michigan English Placement Test, the test may give some indication of students’ overall proficiency, but will not tell you which of your program’s learning objectives students have achieved. Even if you include a writing assignment based on a prompt or an interview, these procedures will not cover the range of learning objectives for your program. The results of placement tests are, therefore, highly inferential. They may gain face validity if used over a long period of time, if program staff can say, “Students with x score typically do well in y level,” but they do not tell which level is the correct one based on achievement of your program’s objectives.

Placement tests are not diagnostic
Because placement tests generally yield limited data about students’ proficiency, they they can be used to broadly categorize students into your program’s levels, but they won’t tell you much about each student’s ability on the four skills in a variety of discourse settings. This is why you sometimes find very quiet and hesitant students in a class with fast talkers, which can lead to student frustration – they placed at the same overall level but their skills vary. To serve students effectively, schools need to build in additional procedures (such as a needs analysis or separating skill classes by level) to ensure that students’ individual needs are understood and can be addressed.

Placement tests don’t tell you a student’s level
This may seem counter-intuitive, but in fact there are no ‘levels’ in language learning. Language proficiency improves on a continuum. Levels are imposed by programs as a way to group students (and each program has its own system of levels and grouping). ‘Level’ for a language program means its curricular level – what is specified to be taught to (and presumably learned by) a categorized group of students. Placement is the process of deciding which level a student should be placed in – but it doesn’t tell you ‘the student’s level.’

Placement procedures rarely ask the student’s opinion
Some students are ambitious and want to be challenged. Others want to spend time reviewing and consolidating what they know. Some students lack confidence and want time and space in the classroom to get comfortable with themselves as language learners in an English-only environment. Students’ own learning preferences are not usually taken into account in placement procedures; they are told what level they will be placed in and that this is the right level for them ‘based on the placement test,’ which, as I’ve tried to show above, is may be limited in its effectiveness. Placement procedures should take students’ preferences into account.

The only relevant information you need is…
‘what is the level of our program at which this student is likely to thrive and make the best progress?’ All other considerations are secondary.

So, if you are introducing, reviewing, or revising your placement testing procedures, consider the following

1. Improve the validity of your procedures by linking them directly to your program’s learning objectives.

2. Take proficiency on individual skills into account when placing students.

3. Avoid concluding that a student must be in a particular level because of the placement result – build in procedures for flexibility.

4. Ask students about their preferred level of challenge – if your program is ‘student-centered,’ you should be doing this anyway.

5. Finally, if a student is unhappy with his or her placement, be willing to make a change – understand that the placement test gave you limited information and that adult students have valid opinions about what works best for them.

 

Are adult learning principles at odds with accreditation requirements?

Created by Katemangostar – Freepik.com

Malcolm Knowles’ seminal text The Adult Learner sets out the principles of andragogy, an approach to teaching and learning which recognizes that children and adults learn differently. In pedagogy, the teaching of children, a relationship of dependency is assumed:

“The pedagogical model assigns to the teacher full responsibility for making all decisions about what will be learned, how it will be learned, when it will be learned, and if it has been learned.”
(Knowles, Holton, & Swanson, 2015, p. 41)

In practice, learners study to pass the course, not to apply their learning to their lives; they become dependent on the teacher’s decision-making; what they bring to the classroom is subordinate to the requirements of the curriculum and textbook; they are ready to learn when the teacher or the system deems them ready; and they are motivated by external motivators such as grades.

Based on this pedagogical model, the No Child Left Behind Act of 2001 sought to improve accountability in K-12 schools, in part by requiring states to develop ‘measurable objectives’ and assessments to measure student achievement. This meant that curricula were to be specified by each state in advance, and all students were to follow the curriculum on a pre-determined timeline.

In the early 2000s, this way of thinking bled into Department of Education requirements for accrediting agencies, including the two agencies that accredit intensive English programs, ACCET and CEA.
IEPs found themselves having similarly to specify learning objectives and timelines and to demonstrate that learners were achieving the specified objectives.

This pedagogy-based approach (whatever you may think of its effectiveness in the public schools) is inconsistent with the andragogical model proposed by Knowles, which makes the following assumptions:

  • Adults need to decide for themselves that they are ready to learn something; they must see its practical application to their lives.
  • Adults are self-directed and resist others imposing their will on them.
  • Adults bring a large volume of experience to the learning environment with them, that teachers must integrate as part of the teaching and learning.
  • Adults bring a problem-solving orientation to learning, and curriculum must address their life issues.
  • Adults are more likely to be motivated by internal factors such as satisfaction and self-esteem, rather than by grades.

If the assumptions of the andragogical model are correct, then the direction most IEPs have moved in as a result of accreditation requirements may not be appropriate for many of their adult students. Curricular items delivered on a specific timeline do not speak to adult students’ own readiness to learn. Talk of externally imposed ‘student learning objectives’ does not interest them. Imposition of topics by the teacher and/or textbook often fails to engage their self-directed nature.

I know of only one IEP that systematically addresses the principles of andragogy through customized, mutually agreed syllabi, assessments, and evaluations. It is very effective. Needless to say, it is also expensive and it requires enormous time and effort on the part of teachers and administrators. The IEP also has a hard time making its case to an accreditor which has curriculum and achievement standards that are based on a pedagogical, rather than an andragogical approach.

Reference
Knowles, M.S., Holton III, E.F., Swanson, R.A., The Adult Learner, Routledge, 2015

Do’s and don’ts of delegation

Many IEPs are staffed by people who started out as classroom teachers. This can be a positive thing, but management skills – especially the skills of managing people – have to be learned. One important skill that can be challenging to learn is delegation. Knowing when and how to delegate is important for all academic directors, student services managers, and program coordinators. Here are some tips for delegating.

  1. Delegate responsibilities, not tasks. True delegation isn’t just asking someone to do something; it is entrusting a person with a responsibility. It could be a project with a deadline such as writing a curriculum, or it could be an ongoing responsibility such as overseeing program assessment. Find an area of work that someone could take on and let a person do it. That’s delegation.
  2. Give ownership. Once you delegate, step back and let the person get on with it. They may not do it the same way as you; they will often find a better way of doing it. Owning an area of responsibility gives employees pride in their work.
  3. Explain your goal, not the process. Tell the person what you are trying to achieve, such as conversion of inquiries into registrations. Let them know how it’s been done before. Then let them figure out how they are going to meet your goal.
  4. Hold the person accountable. Agree on criteria for success, and arrange to check in on a regular basis to discuss progress and challenges. Make the person understand that although they have ownership of this area of work, they are accountable to you and the organization for results.
  5. Anticipate and tolerate mistakes. Employees need space to make mistakes, especially if they are trying out new methods. Agree with yourself that you won’t stress out if the person makes mistakes as they learn to do what you have asked. Repeating the same mistake over and over is a problem, but mistakes in learning are natural.
  6. Support, support, support. Make clear to the person that you are there to support them. When you delegate anything, consider this your primary role. Ask repeatedly, “What can I do to support you? Is there anything I can do to help you do this more effectively?”
  7. And remember: you are still responsible. Although you have delegated, the responsibility is still ultimately yours. If something goes wrong, don’t blame the person you delegated to. When explaining the situation to your boss, take responsibility. Conversely, if your employee did a good job, be quick to praise and advertise his or her accomplishment to your colleagues.

These are my do’s and don’ts of delegation. Do you have any others?

What’s your pricing strategy?

As competition for students increases, intensive English programs should consider the price of their program. The pricing decision must take into account the overhead and operating cost of the program, as well as revenue and margin goals. But a conscious pricing strategy also positions the program in relation to competitor programs. Prospective students evaluate the program price against the perceived value the program will have for them. Here are five examples of intensive English program pricing strategies I have encountered over the past few years, each of which exemplifies a pricing strategy that worked, or didn’t, for the program. Prices are for tuition for a four-week general English program.

Program A was part of a for-profit language school chain, located in the downtown area of a major city. At $2,200 (this was in 1998), it was priced significantly higher than similar programs. The price was intended to communicate high quality, but although this company had developed its own proprietary language learning software and hired only teachers with master’s degrees, it struggled to communicate added value to students. Student feedback included comments such as, “Why is this school so expensive?” This language school chain went out of business.

Takeaway: If you charge a higher price, you must be able to communicate the value you offer beyond the competition.

Program B, also part of a for-profit chain, marketed itself as a kind of ‘business class’ English school. It charged anything from $4,000 to $8,000 in the early 2000s, much more than any other program. Its facility was designed with business executives in mind, classrooms were fitted out as high-end conference rooms, and classes were very small, with additional one-on-one options available for personal attention (which most students opted for). Teachers were trained to push students hard for rapid progress. This school had a clear differentiating factor, and there was an elite clientele willing to pay the price, including corporate HR managers seeking professional development for employees. This was a low-volume, high-margin approach that was successful as long as the school could find and reach its market, which was and remains a challenge.

Takeaway: If you price high to attract a small number of well-heeled students, you must have a strategy and the capacity to reach those students or decision-makers, and clearly describe your difference.

Program C is a university-based program charging  almost $3000, which is about twice what the typical proprietary program in the same city charges. The university and the program have a strong reputation, and the program has a highly qualified and experienced faculty teaching an academically-based curriculum. The program saw a significant drop in enrollments in the past few years. While its price may reflect an ‘elite’ status as a university program and continues to attract students who seek a superior program, its net income is allocated to fund other university programs, and the university has demanding revenue expectations. Students are paying above what the program might need to charge if it were an independent entity. On the other hand, if they choose, students can take advantage of the university’s facilities and its student community.

Takeaway: A university-based IEP must be able to communicate the value inherent in its position on a campus to potential students who have academic ambitions.

Program D is a stand-alone proprietary program outside the main business district of the city, charging $750, or as little as $500 for students who make a long-term commitment. This program has made price its main differentiating factor, and is experiencing growth even at a time of overall declines. This is a high-volume, low-margin strategy that works if enrollment is strong, as it is. The downside is that it attracts a population largely from one country, many of whom stay with relatives who have immigrated from that country, and whose priority may be to stay in the U.S. rather than a strong urge to learn English. This has quality implications in the classroom.

Takeaway: A low price can be successful if you are able to find large numbers of students who may be satisfied with a no-frills program, and recruit them directly rather than through commission-based agents.

Program E: This program is a downtown branch of an international chain of English schools, and charges around $1700, which is about the same as other chain schools in the same city. This school emphasizes its location, its modern facility and technology, and friendly, welcoming atmosphere. It recruits primarily through an extensive network of agents, to whom it pays commissions.

Takeaway: When your price is the same as your direct competitors, you must build and maintain strong relations with your sales network, and develop in your agents a strong brand loyalty.

Information about intensive English programs is ubiquitous, and in an era of high student mobility,  you might say that every program is competing with every other program for students. Price is one of the major factors in positioning your program. Which of the above strategies is yours closest to? Is it appropriate now, and will it remain so in the future?

The push and pull of power in intensive English programs

…a review of Organizational Power Politics: Tactics in Organizational Leadership (2nd Edition) by Gilbert W. Fairholm

Power has a bad reputation in educational environments. In many people’s minds, it is associated with terms like ‘power-hungry’ or ‘greedy for power,’ and yes, I’ve heard it used by faculty to describe what they see as overreaching administrators trying to control their work.

But power, according to Fairholm, is ethically neutral; it is the  motives of the individuals who use it that determine whether its outcomes are positive or negative. And power is intrinsic to any group of people that aims to get things done, so reading a book about organizational power politics can give you insights not only into your own power and how to use and increase it, but also into the power tactics of those around you. In turn, you will gain a deeper understanding of how your organization works, and especially why some people or groups are more powerful than others.

In any group or organizational setting, power is the ability to control scarce resources in order to achieve your aims, even if others oppose you. While you may think of power as coercive (hence its reputation), power is exerted in a number of ways, listed by Fairholm on page 12, ranging from coercive at the top, to consensual at the bottom:

Force
Authority
Manipulation
Threat/Promise
Persuasion
Influence

Those who are higher in the organizational hierarchy are more likely to be able to access the direct forms of power at the top of this list, while those lower down may exercise the indirect power types at the bottom. Yet power is not only about where you are in the hierarchy, and other sources of power include expertise, criticality to the organization, and group solidarity. This means that the exercise of power can be inverted, and power can be exerted upward. Fairholm describes “the power exercised by… lower-level workers who dominate their superiors through their control over resources (e.g., skill) the leader needs” (p. 55).

In many educational settings, especially in universities, there is a tension between faculty and administrators over the use of power. While it may be inappropriate to talk about who is higher and lower in the hierarchy, this struggle often manifests as one between authority legitimized by formal position among administrators, and the threat exercised by faculty who know that their expertise and skills are critical to the institution. Faculty in many university intensive English programs are challenged by the administrative invocation of the authority of non-negotiable accreditation standards. Fairholm sums this situation up concisely: “Promulgation of standard operating procedures, requiring prior (or post) approval of subordinate decision or actions and an over-adherence to organizational traditions, exemplify this tactic” (p. 125). Note that in the case of IEPs, it is often ‘organizational traditions’ that are promoted by the faculty in opposition to the ‘standard operating procedures’ imposed by the administration.  Note also that teachers in proprietary IEPs have long been subject to organizational standard operating procedures, and are unlikely to be able to draw on institutional tradition as a counter-weight. This is one reason why proprietary IEP teachers have relatively less power in their organizations than their university counterparts.

If you want to understand the workings of power in your organization, and gain insight into your current power and how to increase it, Fairholm’s book – with its sharp analysis, questionnaires, and lists of strategies – is a good place to begin.

Fairholm, G.W., Organizational Power Politics: Tactics in Organizational Leadership, 2nd Ed., Praeger 2009

Faculty freedom and curriculum design in intensive English programs

How much freedom do intensive English program (IEP) teachers have to design their courses, choose their materials, and teach to their interests? How much should they? These questions become ever more compelling as accreditation standards push programs to be accountable for their outcomes.

Teachers in proprietary (non-university-governed) IEPs have long been used to teaching within a structured framework, using prescribed textbooks and curricula that map out what is to be covered by the week or even by the day. This has been necessary because, adopting a customer-centric and profit-maximizing approach, they allow students frequent – weekly or monthly – entry and exit points. Students staying for a short program jump in and then out of existing classes with longer-term students. Those long-term students need to be able to move through a defined program of work and progress to the next, and then the next, level. This means that all teachers need to be on the specified part of the curriculum – in some cases on the specified page of the textbook – at all times.

Many university IEPs have inherited the university tradition of faculty autonomy, giving faculty the freedom to write their own syllabi, choose their own materials, and generally teach to their own interests. Under the influence of CEA accreditation standards, faculty are losing some of this autonomy, as student achievement standards require them to teach to a program-wide set of learning objectives. Student promotion to the next level must be based on student achievement of objectives, so faculty have to conform to standard assessment, evaluation, and grading practices. In order to ensure all students are getting the same course, university programs are increasingly prescribing textbooks. As a result, university IEP curricula and faculty work are looking more like those of proprietary programs.

This trend has caused much tension between faculty and administration at IEPs where faculty have fought to retain autonomy in their teaching. Some faculty claim that students are losing out because, being close to the students, they know what is best for them. Administrators charged with implementing accreditation standards argue in turn that students gain when there is a program-wide system that smooths out the differences between faculty styles and preferences.

In proprietary programs, curriculum can be imposed by administrative fiat. This is harder in university programs. Those that have adapted best are the ones where administrators and faculty have a trusting relationship and can jointly respond to the new requirements in a collaborative way that reconciles the divergent demands of individual autonomy and program standardization. Some programs continue to struggle.

 

 

 

Unintended Consequences? Effects of the 2010 Accreditation Act on Intensive English Programs

The Accreditation Act passed in 2010 required that F-1 students pursuing an English language training program must attend a program that is accredited by a Department of Education recognized accrediting agency. University-governed programs were covered by their university’s regional accreditor, which meant that for them, an additional specialized accreditation was optional. All proprietary programs – mostly for-profit language schools – were required to seek and gain accreditation.

The Accreditation Act was supported, and its passage celebrated by, program directors and leaders at university-governed and well-established, already-accredited for-profit language school companies. They were motivated by a strong desire to bring greater professionalism to the field and to weed out a significant number of unscrupulous and fly-by-night operators who had cleared the relatively low bar for entry into the industry and whose low standards were tainting the field as a whole. Since the passage of the Act, the two specialized accrediting agencies for intensive English programs, CEA and ACCET, have added hundreds of intensive English programs to their rolls. Plenty of programs that sought accreditation have been denied, and the weeding out process has been largely successful.

But some consequences are not so unequivocally positive for the field:

  • The accreditation process costs up to $10,000, plus annual sustaining fees. This is a significant financial burden on programs, especially during a time of enrollment challenges. While university-governed programs have the option of sheltering under their institution’s accreditation and avoiding these costs, proprietary programs have no choice but to pay up or cease doing business.
  • The requirement for an IEP to be accredited creates a Catch-22 for potential new entrants into the market. A proprietary program has to be in business for two years (ACCET) or one year (CEA) before it can apply for accreditation. The accreditation process itself takes around 18 months, and if it succeeds, the program must then wait for F-1 issuing approval from the federal government. In the words of one IEP administrator in this situation, “It felt like being choked to death for four years.” During this time, the program has to survive on non-F-1 students. The near-impossibility of this makes the price of entry extremely high for those wanting to enter the field. While there were always requirements to become an F-1 school, the Accreditation Act raised almost insurmountable barriers to new proprietary players.
  • A consequence of this is greater consolidation in the proprietary IEP market. If you cannot start a new school, you have to purchase an existing one. Inevitably, those with the resources to do this are large companies seeking to develop branded chains of English schools. Further, accrediting agencies make it relatively easy for existing schools to open new branches through a simplified accreditation process for the new branch, thus allowing existing companies to expand while new entrants continue to struggle to gain entry.
  • Accreditation likely has the effect of curbing innovation in the field. Adherence to accreditation standards tends to result in institutional isomorphism (the phenomenon of institutions of a certain type looking the same), and programs are reluctant to launch anything radically different for fear of not complying with accreditation standards. Aside from surface details (number of levels, number of weeks per session, etc.), IEPs can be quite difficult to tell apart. This, combined with the lengthy SEVP approval process for new programs, in turn leads to commodification in the industry: potential students have difficulty telling one program apart from another, and use price, location, and established brand reputation to make their choice rather than any specific features of a program.

Overall, the benefit to the field has been positive. Students can apply to U.S. IEPs with the knowledge that their chosen program has been verified by an accreditor to meet high standards. The price to the industry as a whole is high though, and we should look for ways to mitigate the downsides – in particular to find ways to foster innovation and be open to new models – as we continue to face challenging market conditions in the years to come.

How valid is that speaking test really?

students in testing lab

Language learners who take an online language test usually expect to receive an evaluation of their speaking ability in the results. But online tests don’t do a very good job of assessing speaking ability because they lack construct validity: they cannot create the type of conditions the learner will be speaking the language in, such as a conversation or presentation. The iBT TOEFL has speaking components, but the test taker has no interlocutor, creating a highly unrealistic speaking situation – a monologue spoken into a microphone with no audience – on which speaking ability will be evaluated. Some online tests contain no speaking component at all; claims about the test taker’s speaking ability is even more inferential than those of the iBT. None of this prevents test makers from making confident claims about their test’s ability to measure learners’ speaking ability.

Speaking is a particularly difficult skill to test properly, especially the ‘spoken interaction’ described in the Common European Framework of Reference. Research has shown that learners perform differently under different conditions. For example, a test taker scored more highly when paired with another learner in a conversation than when assessed by interview with an examiner (Swain, Kinnear, & Steinman, 2011). Conversation is co-constructed by participants, who build on and scaffold each other’s utterances. Conversation requires cooperation, the successful negotiation of meaning, strategies to understand the other person, asking questions, requesting clarification, affirming, and paraphrasing. Is it likely that any of this can be evaluated by an assessment that does not require the learner to do any of these things?

Online tests have emerged from the psychometric testing tradition, which assumes that an ability is stable in an individual, and therefore requires isolation of the individual in order to avoid extraneous influences. This is the opposite of most spoken language in use. We should call into question the usefulness of tests that make claims based on a lack of validity.

The best way for spoken language to be assessed is by an expert interlocutor interacting with and observing learners in interactions with others over a period of time. Language teachers – trained and experienced in assessment and evaluation techniques, and in many cases able to assess learners over the course of a session or semester – are best placed to offer this kind of assessment.

Reference
Swain, M, Kinnear, P., & Steinman, L., Sociocultural Theory in Second Language Education, Multilingual Matters 2011

Picture credit https://tc.iupui.edu/

How SWBATs and can-do statements shortchange language learners

“Can keep up with an animated discussion, identifying accurately arguments supporting and opposing points of view.” “Can tell a story or describe something in a simple list of points.” If your program is using Common European Framework of Reference (CEFR) descriptors as its outcomes statements, you’ll be familiar with ‘can-do’ statements like these.

The CEFR was developed as a means to assess and describe language proficiency. It was built on the European tradition of communicative language teaching (CLT), which emphasized the performance of language tasks. Since language performance can be observed, the CEFR’s can-do statements were a perfect match for the measurable-outcomes-based accountability initiatives that came in the wake of No Child Left Behind. Many teachers have been trained, encouraged, or badgered to plan their lessons and courses around SWBAT (‘students will be able to’) or can-do statements.

There is a persuasive case to be made that CEFR (and similar) performance statements are a useful way to describe language proficiency. Employers, for example, what to know what a potential employee can do in a language – what practical uses the employee can use the language for. Language educators are not employers, though. What language educators need to know is whether and to what extent learning has taken place, and here’s the problem.

Broadly speaking, two educational traditions have informed language teaching: the behavioral, and the cognitive. Behaviorists see learning as a change in behavior, one that can be observed or measured. Cognitivists see learning as acquiring and understanding knowledge. The cognitivist tradition fell out of fashion with the demise of the grammar-translation method and the rise of behavior-based approaches to language teaching. These days, we can probably all agree that in language learning, we need to refer to both traditions: the acquisition or construction of a mental representation of the language, and the skill required to be able to use it in practice. When our outcomes are can-do statements, we focus on observable or measurable behaviors, but tend to pay less attention to acquired or constructed knowledge. We want to know if the learner ‘can tell a story,’ or ‘keep up with an animated discussion,’ for example.

If you have taught students from various countries, you know that some are great performers even if they lack a solid language base – somehow, they manage to draw on sparse linguistic resources to communicate. And on the other hand, you know that some learners have extensive language knowledge, especially grammar and vocabulary knowledge, but have a great deal of difficulty ‘performing.’ Hence, Chomsky wrote of language proficiency, “behavior is only one kind of evidence, sometimes not the best, and surely no criterion for knowledge,” (as cited in Widdowson, 1990). The one is not necessarily indicative of the other.

If you are an educator (as opposed to an employer), you are interested in student learning in any form. You want to know what progress a learner has made. From a cognitive point of view, that includes changes in the learner’s mental representation of the language – a clearer understanding of the form, meaning, and use of the present perfect, for example – even if that has not yet resulted in a change in behavior, such as the ability to use that tense easily in a conversation. A learner who has made great strides in his/or mental representation of the language but is still speaking in telegraphic speech may be of little interest to an employer, but should be of great interest to an educator, because learning has taken place that is a basis for future teaching. Assessment and description of the learner’s language should address this type of progress. The behavioral tradition, with its can-do outcomes statements have no interest in such cognitive development – it is not interested until there is a change of behavior, an observable, measurable performance.

This approach to assessment shortchanges learners who may have made real progress on the cognitive side. So, I’m calling on language educators not to accept uncritically the use of CEFR and similar performance-based descriptors as measures of language learning.

Reference
Widdowson, H.G., Aspects of Language Teaching, Oxford University Press, 1990

The Accreditation-Ready Program

There are few obligations for faculty and staff that cause knots in the stomach and departmental wrangling than preparing the accreditation self-study. It is often viewed as a burden, a distraction from everyone’s ‘real’ work, and a process of bureaucratic box-checking or of trying to fit the round peg of the program into the square hole of accreditation requirements.

In Five Dimensions of Quality, Linda Suskie draws on years of experience with accreditation, institutional and program assessment, and accountability to re-frame the role of accreditors as “low-cost consultants who can offer excellent collegial advice” (p. 245) to schools and programs seeking to demonstrate their value to stakeholders in an increasingly competitive market.  Accreditation should be viewed not as an imposition of alien practices on an established program, but as a way for a school or program to gain  external affirmation of already-existing quality. The challenge is not to make the program ‘fit’ accreditation standards, but actually to be a quality program and demonstrate that quality.

Accreditation success, then, flows naturally from the pursuit of quality, and is not an end in itself. But what is quality? Suskie breaks it down into five dimensions or ‘cultures’:

A Culture of Relevance
Deploying resources effectively to put students first, and understand and meet stakeholders’ needs.

A Culture of Community
Fostering trust among faculty, students, and staff, communicating openly and honestly, and encouraging collaboration.

A Culture of Focus and Aspiration
Being clear about school or program  purpose, values, and goals.

A Culture of Evidence
Collecting evidence to gauge student learning and program or school effectiveness.

A Culture of Betterment
Using evidence to make improvements and deploy resources effectively.

Fostering these cultures is the work of leadership, since they require widespread buy-in from all stakeholders. The challenge in many institutions is institutional inertia, as Suskie points out in her chapter, “Why is this so hard?” Faculty, staff, and governing boards may feel satisfied that the school’s reputation is sufficient for future success; resources – especially money and people’s time – may not be forthcoming; faculty and staff may live in comfortable isolation from the  real-world needs of students; there may be an ingrained reluctance to communicate successes; there is frequently resistance to change; and siloed departments in programs and institutions make across-the-board cultural change difficult to pull off.

The question administrators and faculty should ask themselves is, “Do we put our efforts into pursuing quality, or into maintaining our accreditation?” Suskie’s book presents a convincing case that working on the former will make the latter much easier and will result in quality rather than box-checking. For its straightforward prose (including jargon alerts scattered throughout), its sound advice, and its call for schools to demonstrate quality in a highly competitive environment, Five Dimensions of Quality should be a go-to resource on the reference bookshelf of decision-makers and leaders in higher education programs.

Suskie, L., Five Dimensions of Quality, Jossey-Bass 2015

More of my education-related book reviews are at Amazon.

Challenge and change in intensive English programs

From left: Bill Hellriegel, Carol Swett, Michelle Bell, Amy Fenning, Alan Broomhead

Challenges over the past few years have deeply impacted intensive English programs, forcing irreversible changes in their organizational cultures that result in anxiety and tension, but also innovation and adaptation. That was the theme of a panel session, “Organizational Culture in University and Proprietary IEPs: Challenges and Changes,” presented by Michelle Bell (University of Southern California), Amy Fenning (University of Tennessee at Martin), Bill Hellriegel (Southern Illinois University), Carol Swett (ELS Language Centers at Benedictine University, Illinois) and myself at the TESOL International Convention on March 28. Recognizing the cultural types of IEPs and how they are affected by changes is the first step in adapting and surviving in an increasingly competitive field.

IEP cultures can roughly be divided into collegial and managerial types, following Bergquist and Pawlak’s (2007) typology of academic cultures. A collegial culture, more likely to be found in a university-governed IEP, is faculty-focused, with faculty scholarship and teaching, academic autonomy and freedom, and faculty ownership of the curriculum as the organizing principle. A managerial culture is administration-driven, motivated by considerations of fiscal responsibility and effective supervision, and organized by systems, processes, and standards.

The massive shift to accreditation in IEPs has moved collegially-oriented programs in a managerial direction. Faculty are required to plan, teach, and assess in compliance with program-wide student learning outcomes; policies and procedures have to be written and followed; and program success is measured by data, which has to be systematically collected, analyzed, and evaluated. Proprietary IEPs are seeing a a shift in the other direction: faculty standards require minimum levels of certification, experience, and ongoing professional development, and these are affecting faculty hiring and employment practices in many proprietary programs.

The severe enrollment challenge of the past two years has also affected both types of program. University IEPs are becoming more revenue-driven and entrepreneurial, actively seeking new recruitment partnerships and designing new programs – such as short-term high school programs – to respond to changing demand. Faculty may have little say in these initiatives. Meanwhile, proprietary IEPs are increasingly developing conditional-admit and TOEFL-waiver agreements with partner universities, requiring them to make programs more academically-focused and hire masters-level teachers who are qualified to teach English for academic purposes.

These are ground-shifting developments, and program leaders who recognize the need to address profound cultural change in their organizations – and not just surface-level adjustments – will be in the strongest position to navigate these challenging times.

Reference
Bergquist, W.H. & Pawlak, K., Engaging the Six Cultures of the Academy, Jossey-Bass 2007

Is language proficiency measurable?

For the past few years, ESL teachers have been pushed to focus their efforts on helping their students achieve ‘measurable objectives.’ Am I the only person who finds this a strange idea? Measuring something is a matter of determining how much of something there is. To measure, we need a unit of measurement: inches and feet, centimeters and meters, pounds and ounces, grams and kilograms. By agreeing on standardized units of measurement, we can determine, objectively, the quantity of something.

Could we use this approach to evaluate a work of art or a piece of music? What would Van Gogh’s Starry Night or Mozart’s Eine Kleine Nachtmusik measure? What unit of measurement would you use? Impossible, because we are not dealing with quantity (the amount of paint or size of canvas, or the number of notes) but with quality. There is no unit of measurement for quality.

When we want to assess and evaluate language are we dealing with something more like a distance or weight, or like a work of art? Is it the quantity of language or the quality of language we want to know about?

If you believe it is quantity, then you might say we can ‘measure’ the number of words a student has learned, or the number of grammar points. These don’t work as units of measurement, though, because defining exactly what is meant by ‘learning a word’ is complicated, as language teachers know. A student may be able to say it but not spell it, may use it but in an inappropriate context, may not recognize it when written down but may hear it in another person’s speech, may forget it on one occasion but recall it on another.

The psychometric testing tradition has given us tests which appear to measure learners’ language ability by assigning a score and thus appear to be objective. Classroom assessments by teachers are often regarded as a subjective second-best.

We should move away from the notion that language proficiency is measurable, and that test scores give us an ‘objective measure’ of a learner’s ability.  Language should be evaluated qualitatively, by people, using rich description rather than fantasy units of measurement that give the false impression of objectivity.

The Inadequacy of “ESL” for International Student Preparation

Wrapped up in the term ESL (English as a Second Language) is an assumption that language, above all, is what students need to succeed in an English-speaking environment. The same kind of assumption can be found in the name of the most popular standardized U.S. admissions test for international students, the TOEFL (Test of English as a Foreign Language). The CEFR (Common European Framework of Reference for Languages) lists levels of language proficiency by skill, and many ESL programs continue to organize their curricula on the basis of Listening, Speaking, Reading, and Writing skills. The field of SLA (Second Language Acquisition) is a major feeder discipline in ESL teacher preparation programs.

A focus on the acquisition of language skills gets us only so far if we are preparing an international student for academic work in an English-speaking setting.  One thing among very many that this student needs to do is to read a text critically and offer an original, well-thought-out, supported, and argued response. The student may need to argue that response in class, and defend it against other points of view, in an assertive yet diplomatic manner. To be taken seriously, the student will need to behave in what is recognized as a normal and appropriate manner in that environment – and know when and how to revert to a more informal style when class ends. All of this goes far beyond language skills.

What this student needs to learn is what James Paul Gee in Social Linguistics and Literacies refers to as Discourse (with a capital D). Discourse “is composed of distinctive ways of listening/speaking and often, too, writing/reading coupled with distinctive ways of acting, interacting, valuing, feeling, dressing, thinking, believing with other people and with various objects, tools, and technologies, so as to enact specific socially recognizable identities engaged in specific socially recognizable activities” (p. 152). These are less language skills than “social practices into which people are apprenticed as part of a social group” (p. 76). As we move in different Discourse communities, we need to know how to play our part and be recognized as a legitimate member of each community. Discourses are mastered by “enculturation…into social practices through scaffolded and supported interaction with people who have already mastered the Discourse” (p. 168).

This helps us understand why any program of learning that reduces preparation to language skills is inadequate. Students need to learn the ways of interacting, believing, valuing, and effectively being in the academic Discourse community. University IEPs (intensive English programs) teach English for academic purposes, but they still largely identify as English language programs with language-based missions, their faculty members have degrees in teaching English, and classes are often language skill-specific. They are often isolated from the rest of the campus, and therefore don’t allow for the kind of apprenticeship into the social practices of the campus that would make international students full members of the Discourse community.

In order to address this wider understanding of international student preparation:

  • Intensive English programs should ensure their missions, their curricula and teaching, and their names, encapsulate the full meaning of international student preparation – not simply ESL.
  • University administrations should make international student preparation a task for the whole university, supported by, but not the sole responsibility of, an intensive English program. The IEP’s efforts should be integrated into a campus-wide strategy for international student preparation.
  • Universities should not expect that simply raising the required TOEFL scores will improve international student outcomes – students need induction into the Discourse community, not just a higher TOEFL score.
  • ESL teacher preparation programs need to include coursework on social literacy and in preparing students to enter and successfully navigate their target Discourse communities.

Some of this has already been achieved. Many IEPs recognize their wider mission of orienting students into academic culture, and more recently,  pathway programs have been structured to provide ESL support alongside credit-bearing classes that, in theory at least, offers an apprenticeship into the academic community. But there is a long way to go before the notion of Discourse communities drives international student preparation beyond the inadequacy of “ESL.”

Reference
Gee, J.P., Social Linguistics and Literacies, 5th Ed., Routledge 2015

Working with the tension between language test validity and reliability

The combination of validity and reliability is the holy grail when it comes to language assessment, yet these two qualities are always in direct tension with each other. This can create a challenge when English language programs try to put in place effective measures of language learning, and especially when they have to convince their accreditors that they’ve done so. Student achievement standards are frequently not met in accreditation reviews for precisely this reason.

An assessment is valid if it measures what it is supposed to measure. So, a multiple choice test is generally not a very valid means of testing speaking ability; nor is a gap-fill test a very valid way to determine whether a student has learned to use a grammar structure in communication. On the other hand, a student presentation might serve as a useful basis for a valid assessment of speaking ability, and a speaking or writing test that elicits a target grammar structure would bring to light a student’s ability to use grammar.

An assessment is reliable if would yield the same results for that student if administered by a different person or in a different location. An in-class presentation or role-play assessed by the class teacher is vulnerable to having a low level of reliability, since the test conditions would be difficult to reproduce in another class. The TOEFL iBT is probably the gold standard for test reliability, with extremely detailed protocols for ensuring the uniformity of the test-taking experience for all students, and two-rater grading of written and spoken assignments.

You can probably see the tension: the greater the validity, the harder it is to attain reliability; the greater the reliability, the harder it is to make the test valid (in the three-hour iBT, the test taker is not required to interact with a single human being).

To increase the reliability of valid assessments, programs can:

  1. use a common set of learning objectives across the program and hold teachers accountable for teaching to them
  2. use standard assessment rubrics across the program
  3. calibrate grading through teacher training
  4. have more than one person assess each student’s performance.

These measures might generate pushback among faculty in some university programs.

Unfortunately, there aren’t any great ways to increase the validity of highly reliable of achievement tests. Doing so would require standardizing the teaching – teaching directly to the test – which nobody in an IEP wants, except in a course specifically for test preparation. Programs that use external standardized tests for level promotion are not using a valid means of assessing what was taught (since the test makers don’t know what was taught).

Instead of seeking the absolute standard of ‘assessments that are valid and reliable,’ we need to

  1. start by creating assessments that are valid – that measure precisely what was taught and was supposed to be learned; and then
  2. design and implement measures to reach as high a level of reliability of those assessments as is possible and practical.

Using this approach is a recognition that you can’t have it all, but you can work within the tension of validity and reliability to reach a satisfactory compromise.