Category Archives: For Schools

I can advise on program management, curriculum, faculty issues, and business development.

Is language proficiency measurable?

For the past few years, ESL teachers have been pushed to focus their efforts on helping their students achieve ‘measurable objectives.’ Am I the only person who finds this a strange idea? Measuring something is a matter of determining how much of something there is. To measure, we need a unit of measurement: inches and feet, centimeters and meters, pounds and ounces, grams and kilograms. By agreeing on standardized units of measurement, we can determine, objectively, the quantity of something.

Could we use this approach to evaluate a work of art or a piece of music? What would Van Gogh’s Starry Night or Mozart’s Eine Kleine Nachtmusik measure? What unit of measurement would you use? Impossible, because we are not dealing with quantity (the amount of paint or size of canvas, or the number of notes) but with quality. There is no unit of measurement for quality.

When we want to assess and evaluate language are we dealing with something more like a distance or weight, or like a work of art? Is it the quantity of language or the quality of language we want to know about?

If you believe it is quantity, then you might say we can ‘measure’ the number of words a student has learned, or the number of grammar points. These don’t work as units of measurement, though, because defining exactly what is meant by ‘learning a word’ is complicated, as language teachers know. A student may be able to say it but not spell it, may use it but in an inappropriate context, may not recognize it when written down but may hear it in another person’s speech, may forget it on one occasion but recall it on another.

The psychometric testing tradition has given us tests which appear to measure learners’ language ability by assigning a score and thus appear to be objective. Classroom assessments by teachers are often regarded as a subjective second-best.

We should move away from the notion that language proficiency is measurable, and that test scores give us an ‘objective measure’ of a learner’s ability.  Language should be evaluated qualitatively, by people, using rich description rather than fantasy units of measurement that give the false impression of objectivity.

The Inadequacy of “ESL” for International Student Preparation

Wrapped up in the term ESL (English as a Second Language) is an assumption that language, above all, is what students need to succeed in an English-speaking environment. The same kind of assumption can be found in the name of the most popular standardized U.S. admissions test for international students, the TOEFL (Test of English as a Foreign Language). The CEFR (Common European Framework of Reference for Languages) lists levels of language proficiency by skill, and many ESL programs continue to organize their curricula on the basis of Listening, Speaking, Reading, and Writing skills. The field of SLA (Second Language Acquisition) is a major feeder discipline in ESL teacher preparation programs.

A focus on the acquisition of language skills gets us only so far if we are preparing an international student for academic work in an English-speaking setting.  One thing among very many that this student needs to do is to read a text critically and offer an original, well-thought-out, supported, and argued response. The student may need to argue that response in class, and defend it against other points of view, in an assertive yet diplomatic manner. To be taken seriously, the student will need to behave in what is recognized as a normal and appropriate manner in that environment – and know when and how to revert to a more informal style when class ends. All of this goes far beyond language skills.

What this student needs to learn is what James Paul Gee in Social Linguistics and Literacies refers to as Discourse (with a capital D). Discourse “is composed of distinctive ways of listening/speaking and often, too, writing/reading coupled with distinctive ways of acting, interacting, valuing, feeling, dressing, thinking, believing with other people and with various objects, tools, and technologies, so as to enact specific socially recognizable identities engaged in specific socially recognizable activities” (p. 152). These are less language skills than “social practices into which people are apprenticed as part of a social group” (p. 76). As we move in different Discourse communities, we need to know how to play our part and be recognized as a legitimate member of each community. Discourses are mastered by “enculturation…into social practices through scaffolded and supported interaction with people who have already mastered the Discourse” (p. 168).

This helps us understand why any program of learning that reduces preparation to language skills is inadequate. Students need to learn the ways of interacting, believing, valuing, and effectively being in the academic Discourse community. University IEPs (intensive English programs) teach English for academic purposes, but they still largely identify as English language programs with language-based missions, their faculty members have degrees in teaching English, and classes are often language skill-specific. They are often isolated from the rest of the campus, and therefore don’t allow for the kind of apprenticeship into the social practices of the campus that would make international students full members of the Discourse community.

In order to address this wider understanding of international student preparation:

  • Intensive English programs should ensure their missions, their curricula and teaching, and their names, encapsulate the full meaning of international student preparation – not simply ESL.
  • University administrations should make international student preparation a task for the whole university, supported by, but not the sole responsibility of, an intensive English program. The IEP’s efforts should be integrated into a campus-wide strategy for international student preparation.
  • Universities should not expect that simply raising the required TOEFL scores will improve international student outcomes – students need induction into the Discourse community, not just a higher TOEFL score.
  • ESL teacher preparation programs need to include coursework on social literacy and in preparing students to enter and successfully navigate their target Discourse communities.

Some of this has already been achieved. Many IEPs recognize their wider mission of orienting students into academic culture, and more recently,  pathway programs have been structured to provide ESL support alongside credit-bearing classes that, in theory at least, offers an apprenticeship into the academic community. But there is a long way to go before the notion of Discourse communities drives international student preparation beyond the inadequacy of “ESL.”

Reference
Gee, J.P., Social Linguistics and Literacies, 5th Ed., Routledge 2015

Working with the tension between language test validity and reliability

The combination of validity and reliability is the holy grail when it comes to language assessment, yet these two qualities are always in direct tension with each other. This can create a challenge when English language programs try to put in place effective measures of language learning, and especially when they have to convince their accreditors that they’ve done so. Student achievement standards are frequently not met in accreditation reviews for precisely this reason.

An assessment is valid if it measures what it is supposed to measure. So, a multiple choice test is generally not a very valid means of testing speaking ability; nor is a gap-fill test a very valid way to determine whether a student has learned to use a grammar structure in communication. On the other hand, a student presentation might serve as a useful basis for a valid assessment of speaking ability, and a speaking or writing test that elicits a target grammar structure would bring to light a student’s ability to use grammar.

An assessment is reliable if would yield the same results for that student if administered by a different person or in a different location. An in-class presentation or role-play assessed by the class teacher is vulnerable to having a low level of reliability, since the test conditions would be difficult to reproduce in another class. The TOEFL iBT is probably the gold standard for test reliability, with extremely detailed protocols for ensuring the uniformity of the test-taking experience for all students, and two-rater grading of written and spoken assignments.

You can probably see the tension: the greater the validity, the harder it is to attain reliability; the greater the reliability, the harder it is to make the test valid (in the three-hour iBT, the test taker is not required to interact with a single human being).

To increase the reliability of valid assessments, programs can:

  1. use a common set of learning objectives across the program and hold teachers accountable for teaching to them
  2. use standard assessment rubrics across the program
  3. calibrate grading through teacher training
  4. have more than one person assess each student’s performance.

These measures might generate pushback among faculty in some university programs.

Unfortunately, there aren’t any great ways to increase the validity of highly reliable of achievement tests. Doing so would require standardizing the teaching – teaching directly to the test – which nobody in an IEP wants, except in a course specifically for test preparation. Programs that use external standardized tests for level promotion are not using a valid means of assessing what was taught (since the test makers don’t know what was taught).

Instead of seeking the absolute standard of ‘assessments that are valid and reliable,’ we need to

  1. start by creating assessments that are valid – that measure precisely what was taught and was supposed to be learned; and then
  2. design and implement measures to reach as high a level of reliability of those assessments as is possible and practical.

Using this approach is a recognition that you can’t have it all, but you can work within the tension of validity and reliability to reach a satisfactory compromise.

Teacher Leadership for Program Improvement and Development

“The function of leadership is to produce more leaders, not more followers.” – Ralph Nader

If you’ve read anything from the popular leadership literature, you’re probably familiar with the prescriptions for strong leadership: confidence, vision, integrity, charisma, and so on. Analyses like these are premised on positional leadership, or what James Spillane called ‘the heroics of leadership’ – the notion of the strong leader standing at the head of an organization, leading the way. Think Jack Welch, Jeff Bezos, Larry Page.

There is an alternative to this view of leadership. Distributed leadership begins with the idea of leadership tasks and asks how these tasks can be accomplished by individuals across an organization. In an intensive English program (IEP), leadership tasks can include curriculum development, teacher professional development, and materials writing, among others. In many management-driven IEPs, these tasks are still carried out by non-teaching staff. But there are advantages to bringing teachers into leadership tasks and distributing them more widely.

Bringing teachers into leadership roles – whether formally or informally – is great professional development for teachers in a profession that otherwise may have no upward career path. It can give teachers a sense of fulfillment from helping other teachers or the program as a whole, and can serve to retain talented teachers by more thoroughly networking them into the program. And it serves the program by drawing on the enormous pool of talent that teachers bring, and which is otherwise confined to their classrooms.

There are challenges to introducing teacher leadership. The role of leader is not a part of the ‘role schema’ of a teacher – teachers are socialized to be followers and may not see themselves as leaders. Teachers need to reframe their self-identity to include leadership. External constraints include how teachers are perceived by other teachers when they take on leadership roles – there is often a culture of egalitarianism among teachers, and teacher leaders may draw disapproval from their teacher peers. Finally, structuring  teacher jobs to include leadership can be a challenge: teachers may lack the time or energy to take on leadership roles, and the program may be limited in its ability to compensate them or structure their jobs appropriately to accommodate leadership tasks.

For teacher leadership to succeed, therefore, it requires strong support from the program director or academic director. Teachers need to be empowered to take on leadership roles, and they need to be given the time and resources to succeed. They may also need coaching on how to manage their identity as teacher leaders, and they may need to be protected from resentment that may occur among peers as they take on leadership roles.

IEPs serve their students best when everyone has an opportunity to contribute to decision-making and program improvements. Here are some examples of teacher leadership I’ve seen over the past few years:

  • A teacher took the initiative to start a reading corner to encourage extensive reading among students. With management support, she built up a collection of books for ESL learners at all levels, and now runs the reading corner as part of her job.
  • A teacher started a peer observation group to encourage teachers to visit each other’s classes and give feedback.
  • Teachers took on program coordinator roles for which they were given release time from their teaching.
  • Teachers played a role in the selection and hire of new teachers.
  • Teachers delivered professional development workshops for their peers, or organized a professional development program.
  • Teachers self-organized into ‘level’ teams to collaborate on in-class projects and assessments.
  • Experienced teachers mentored newer teachers.
  • Teachers became subject or skill experts in the program.

Think about how teacher leadership can be extended in your program.

Can an intensive English program go virtual?


Business image created by Jcomp – Freepik.com

Along with continuing enrollment challenges for university and proprietary intensive English programs (IEPs) comes a demand for fresh ideas, re-thinking the model, and new types of programs that meet the needs of today’s learner. Given the rise and ubiquity of online learning, many IEP leaders are asking whether and how they might take their programs online.

Online ESL is already big business, with many startup companies connecting students and teachers in different parts of the world through synchronous online lessons. The first challenge for IEPs thinking about breaking into this market is how to devote the resources to develop and market an online program while not diverting resources from their current on-ground operations. But the greater challenge is how to take a model that has developed and established its value in one format (on-ground) over many years and adapting that model to a new, online format.

In their book on academic cultures, Bergquist and Pawlak identify the ‘tangible’ culture and the ‘virtual’ culture as two cultural types that may be in tension with each other. IEPs have developed around a tangible culture that emphasizes location, student life, interaction with local people, institutional facilities – the whole student experience. Additionally, as a result of visa regulations, they have built curricula and weekly schedules that prioritize compliance over the needs of students (example: there is no strictly educational reason why students should spend 18 hours in class in order to learn a language). This model has been valuable to the many thousands of students who have attended IEPs. But how much of this on-ground value can an IEP retain when it puts its programs online? And with many providers in the online market, most of which are specialized, agile, not tied to an on-ground model, and highly entrepreneurial, how feasible is it for established IEPs to make significant inroads into this market?

My prediction is that most intensive English programs will not play a significant role in the online ESL market, nor will they want to break from the on-ground model they have spent years nurturing. To survive, they will need to continue adapting to the needs of current students who want to travel for an education. Right now this means offering short, specialized programs, and pathways into universities. While the demand for intensive English programs is currently in a slump and may never bounce back to the numbers of recent years, the tangible academic culture is not going away, and there will always be value in traveling for a global, intercultural, and language education. IEPs need to continue working to demonstrate that value to tomorrow’s students.

Reference
Bergquist, W.H. & Pawlak, K., Engaging the Six Cultures of the Academy, Jossey-Bass 2008.