Category Archives: Other news

How valid is that speaking test really?

students in testing lab

Language learners who take an online language test usually expect to receive an evaluation of their speaking ability in the results. But online tests don’t do a very good job of assessing speaking ability because they lack construct validity: they cannot create the type of conditions the learner will be speaking the language in, such as a conversation or presentation. The iBT TOEFL has speaking components, but the test taker has no interlocutor, creating a highly unrealistic speaking situation – a monologue spoken into a microphone with no audience – on which speaking ability will be evaluated. Some online tests contain no speaking component at all; claims about the test taker’s speaking ability is even more inferential than those of the iBT. None of this prevents test makers from making confident claims about their test’s ability to measure learners’ speaking ability.

Speaking is a particularly difficult skill to test properly, especially the ‘spoken interaction’ described in the Common European Framework of Reference. Research has shown that learners perform differently under different conditions. For example, a test taker scored more highly when paired with another learner in a conversation than when assessed by interview with an examiner (Swain, Kinnear, & Steinman, 2011). Conversation is co-constructed by participants, who build on and scaffold each other’s utterances. Conversation requires cooperation, the successful negotiation of meaning, strategies to understand the other person, asking questions, requesting clarification, affirming, and paraphrasing. Is it likely that any of this can be evaluated by an assessment that does not require the learner to do any of these things?

Online tests have emerged from the psychometric testing tradition, which assumes that an ability is stable in an individual, and therefore requires isolation of the individual in order to avoid extraneous influences. This is the opposite of most spoken language in use. We should call into question the usefulness of tests that make claims based on a lack of validity.

The best way for spoken language to be assessed is by an expert interlocutor interacting with and observing learners in interactions with others over a period of time. Language teachers – trained and experienced in assessment and evaluation techniques, and in many cases able to assess learners over the course of a session or semester – are best placed to offer this kind of assessment.

Reference
Swain, M, Kinnear, P., & Steinman, L., Sociocultural Theory in Second Language Education, Multilingual Matters 2011

Picture credit https://tc.iupui.edu/

Keep Calm and Dance

Jerome Murphy of Harvard’s Graduate School of Education has made a career of educational leadership, and has first-hand experience of the stress, burnout, and even despair that can come with a lifetime of trying to satisfy the needs and demands of faculty, staff, students, and a host of other stakeholders. “Honking and hissing like geese,” goes Murphy’s goose theory of leadership, “faculty and staff members will cruise into the boss’s office, ruffle their feathers, poop on the rug, and leave” (p. 44), expecting a solution to whatever problem they brought in. The unskillful response of many leaders under these conditions is to obsessively ruminate, resist the discomfort and try to escape it, and rebuke themselves for not measuring up. The more they try to escape their discomfort, the more entangled they become in it. Can anyone relate yet?

Murphy’s answer in Dancing in the Rain: Leading with Compassion, Vitality, and Mindfulness in Education, is learn to live with the emotional discomfort and get it to work for you. When it rains, don’t run for cover; learn to dance in it. His formula for doing this, developed over a career, is summed up by the acronym MYDANCE:

Mind your values: Take action inspired by what matters most to you
Yield to now: Slow down and focus on the present moment
Disentangle from upsets: Mentally step back, observing and making room for upsets
Allow unease: Open up to upsets even if you dislike them
Nourish yourself: Engage in activities that replenish your energy and restore your perspective
Cherish self-compassion: Give yourself the kindness you need and deserve
Express feelings wisely: Carefully reveal your human side so that you can build trusting relationships (p. 41)

Murphy takes the reader through these Buddhist-inspired precepts chapter by chapter, and includes many easy-to-do exercises. For example in the Mind Your Values chapter, we are invited to call to mind a favorite leader, reflecting on the person’s values and how the person makes us feel. In Yield to Now, a 5-minute exercise suggests focusing in turn on the five senses, bringing attention back gently each time the mind wanders.

This definitely isn’t your typical educational leadership book. It’s more of a handbook on surviving and thriving amid the slings and arrows of academic administration. If your professional life seems to be a constant struggle, this may be the therapy you need.

(This review was also posted on Amazon)

IEPs and the Rise of Pathway Programs

This week my colleague from the University of Kansas, Deborah Osborne and I led a discussion on the future of

Deborah Osborne and Alan at the TESOL Convention, Seattle

university intensive English programs at the TESOL Convention in Seattle. Many university IEPs reported enrollment declines of between 10% and 70% last fall, and some are really struggling at this point. Meanwhile, pathway programs resulting from partnerships between universities (such as American University, Oregon State University, and George Mason University) and corporate partners (such as INTO, Navitas, and Shorelight) continue to proliferate, suggesting that English language and academic preparation for international students is undergoing a major shift.

For prospective students, the major attraction of a pathway program is the word ‘guaranteed.’ If they complete the program successfully, they will matriculate into the partner university. Many university IEPs can offer college advising and assistance with applications, but are not able to offer that guarantee. For universities, pathway programs can offer a fast track to increasing international student enrollments with little upfront investment or the need to build their own international student recruiting capacity.

There are two main choices for university IEPs. First, they can try to compete on the pathway providers’ territory by setting up their own pathway programs. The challenge here is that they generally don’t have the recruitment network to make that happen quickly, and there are very few instances of a university recruiting significant numbers of students via a home-grown pathway. Second, they can adapt to provide programming that pathway programs can’t. Examples include shorter programs, custom programs, specialized programs, and programs to support admitted or matriculated international students on their campus. This will require flexibility and an innovative and entrepreneurial spirit.

These are challenging times for IEPs, and it remains to be seen whether they can adapt to the challenge of the pathway model.