Category Archives: For Schools

I can advise on program management, curriculum, faculty issues, and business development.

Working with the tension between language test validity and reliability

The combination of validity and reliability is the holy grail when it comes to language assessment, yet these two qualities are always in direct tension with each other. This can create a challenge when English language programs try to put in place effective measures of language learning, and especially when they have to convince their accreditors that they’ve done so. Student achievement standards are frequently not met in accreditation reviews for precisely this reason.

An assessment is valid if it measures what it is supposed to measure. So, a multiple choice test is generally not a very valid means of testing speaking ability; nor is a gap-fill test a very valid way to determine whether a student has learned to use a grammar structure in communication. On the other hand, a student presentation might serve as a useful basis for a valid assessment of speaking ability, and a speaking or writing test that elicits a target grammar structure would bring to light a student’s ability to use grammar.

An assessment is reliable if would yield the same results for that student if administered by a different person or in a different location. An in-class presentation or role-play assessed by the class teacher is vulnerable to having a low level of reliability, since the test conditions would be difficult to reproduce in another class. The TOEFL iBT is probably the gold standard for test reliability, with extremely detailed protocols for ensuring the uniformity of the test-taking experience for all students, and two-rater grading of written and spoken assignments.

You can probably see the tension: the greater the validity, the harder it is to attain reliability; the greater the reliability, the harder it is to make the test valid (in the three-hour iBT, the test taker is not required to interact with a single human being).

To increase the reliability of valid assessments, programs can:

  1. use a common set of learning objectives across the program and hold teachers accountable for teaching to them
  2. use standard assessment rubrics across the program
  3. calibrate grading through teacher training
  4. have more than one person assess each student’s performance.

These measures might generate pushback among faculty in some university programs.

Unfortunately, there aren’t any great ways to increase the validity of highly reliable of achievement tests. Doing so would require standardizing the teaching – teaching directly to the test – which nobody in an IEP wants, except in a course specifically for test preparation. Programs that use external standardized tests for level promotion are not using a valid means of assessing what was taught (since the test makers don’t know what was taught).

Instead of seeking the absolute standard of ‘assessments that are valid and reliable,’ we need to

  1. start by creating assessments that are valid – that measure precisely what was taught and was supposed to be learned; and then
  2. design and implement measures to reach as high a level of reliability of those assessments as is possible and practical.

Using this approach is a recognition that you can’t have it all, but you can work within the tension of validity and reliability to reach a satisfactory compromise.

Teacher Leadership for Program Improvement and Development

“The function of leadership is to produce more leaders, not more followers.” – Ralph Nader

If you’ve read anything from the popular leadership literature, you’re probably familiar with the prescriptions for strong leadership: confidence, vision, integrity, charisma, and so on. Analyses like these are premised on positional leadership, or what James Spillane called ‘the heroics of leadership’ – the notion of the strong leader standing at the head of an organization, leading the way. Think Jack Welch, Jeff Bezos, Larry Page.

There is an alternative to this view of leadership. Distributed leadership begins with the idea of leadership tasks and asks how these tasks can be accomplished by individuals across an organization. In an intensive English program (IEP), leadership tasks can include curriculum development, teacher professional development, and materials writing, among others. In many management-driven IEPs, these tasks are still carried out by non-teaching staff. But there are advantages to bringing teachers into leadership tasks and distributing them more widely.

Bringing teachers into leadership roles – whether formally or informally – is great professional development for teachers in a profession that otherwise may have no upward career path. It can give teachers a sense of fulfillment from helping other teachers or the program as a whole, and can serve to retain talented teachers by more thoroughly networking them into the program. And it serves the program by drawing on the enormous pool of talent that teachers bring, and which is otherwise confined to their classrooms.

There are challenges to introducing teacher leadership. The role of leader is not a part of the ‘role schema’ of a teacher – teachers are socialized to be followers and may not see themselves as leaders. Teachers need to reframe their self-identity to include leadership. External constraints include how teachers are perceived by other teachers when they take on leadership roles – there is often a culture of egalitarianism among teachers, and teacher leaders may draw disapproval from their teacher peers. Finally, structuring  teacher jobs to include leadership can be a challenge: teachers may lack the time or energy to take on leadership roles, and the program may be limited in its ability to compensate them or structure their jobs appropriately to accommodate leadership tasks.

For teacher leadership to succeed, therefore, it requires strong support from the program director or academic director. Teachers need to be empowered to take on leadership roles, and they need to be given the time and resources to succeed. They may also need coaching on how to manage their identity as teacher leaders, and they may need to be protected from resentment that may occur among peers as they take on leadership roles.

IEPs serve their students best when everyone has an opportunity to contribute to decision-making and program improvements. Here are some examples of teacher leadership I’ve seen over the past few years:

  • A teacher took the initiative to start a reading corner to encourage extensive reading among students. With management support, she built up a collection of books for ESL learners at all levels, and now runs the reading corner as part of her job.
  • A teacher started a peer observation group to encourage teachers to visit each other’s classes and give feedback.
  • Teachers took on program coordinator roles for which they were given release time from their teaching.
  • Teachers played a role in the selection and hire of new teachers.
  • Teachers delivered professional development workshops for their peers, or organized a professional development program.
  • Teachers self-organized into ‘level’ teams to collaborate on in-class projects and assessments.
  • Experienced teachers mentored newer teachers.
  • Teachers became subject or skill experts in the program.

Think about how teacher leadership can be extended in your program.

Can an intensive English program go virtual?


Business image created by Jcomp – Freepik.com

Along with continuing enrollment challenges for university and proprietary intensive English programs (IEPs) comes a demand for fresh ideas, re-thinking the model, and new types of programs that meet the needs of today’s learner. Given the rise and ubiquity of online learning, many IEP leaders are asking whether and how they might take their programs online.

Online ESL is already big business, with many startup companies connecting students and teachers in different parts of the world through synchronous online lessons. The first challenge for IEPs thinking about breaking into this market is how to devote the resources to develop and market an online program while not diverting resources from their current on-ground operations. But the greater challenge is how to take a model that has developed and established its value in one format (on-ground) over many years and adapting that model to a new, online format.

In their book on academic cultures, Bergquist and Pawlak identify the ‘tangible’ culture and the ‘virtual’ culture as two cultural types that may be in tension with each other. IEPs have developed around a tangible culture that emphasizes location, student life, interaction with local people, institutional facilities – the whole student experience. Additionally, as a result of visa regulations, they have built curricula and weekly schedules that prioritize compliance over the needs of students (example: there is no strictly educational reason why students should spend 18 hours in class in order to learn a language). This model has been valuable to the many thousands of students who have attended IEPs. But how much of this on-ground value can an IEP retain when it puts its programs online? And with many providers in the online market, most of which are specialized, agile, not tied to an on-ground model, and highly entrepreneurial, how feasible is it for established IEPs to make significant inroads into this market?

My prediction is that most intensive English programs will not play a significant role in the online ESL market, nor will they want to break from the on-ground model they have spent years nurturing. To survive, they will need to continue adapting to the needs of current students who want to travel for an education. Right now this means offering short, specialized programs, and pathways into universities. While the demand for intensive English programs is currently in a slump and may never bounce back to the numbers of recent years, the tangible academic culture is not going away, and there will always be value in traveling for a global, intercultural, and language education. IEPs need to continue working to demonstrate that value to tomorrow’s students.

Reference
Bergquist, W.H. & Pawlak, K., Engaging the Six Cultures of the Academy, Jossey-Bass 2008.

Keeping your Intensive English Program Relevant on Campus


Created by Ijeab – Freepik.com

These are trying times for many on-campus intensive English programs. Enrollment and revenue are down, and there is increased pressure from senior administration for many IEPs to demonstrate their continuing relevance and usefulness to the wider institution.

At the same time, many universities have enrolled international students who can benefit from language, cultural, and social support. IEPs have faculty and staff who are highly qualified to provide programming in these areas (and who may currently have less work to do), yet because IEPs are typically viewed as profit centers rather than service units, they are not called on to offer such support. This is short-sighted, as increased support for degree-seeking international students will improve their retention and completion rates – which is good for the students,  the university’s bottom line, and the institution’s reputation.

IEP directors can sell this idea to university administrators. Here are some activities the IEP can offer to improve the international student experience on campus:

Workshops for faculty: Offer strategies to encourage international students to participate in class discussions, or give advice on assessing written work of students using English as a second language.

Resource webpage for English language support:  Like this one at Hunter College. Include online dictionaries, grammar resources, and writing advice for international students across campus.

Tutoring: Many universities have a writing center, but few have a place specifically to help with second language issues. The IEP can provide this.

English language workshops: Students who have gained a high score on the TOEFL or IELTS may still be lacking essential English skills. Offer workshops in pronunciation, pragmatics, or giving presentations.

Career preparation workshops: Many international students may seek on-campus employment, co-op or internship positions, or CPT/OPT opportunities. Help them write an effective application and interview effectively.

Pre-arrival language preparation: Develop a short online course to give incoming international students confidence with English. Prepare them for the various situations they will encounter and provide strategies to continue working on their English once they arrive.

These ideas will likely require building relationships with other offices on campus, and IEP directors may run into territory issues. Getting buy-in from a senior administrator who can support these efforts may be essential. This person may also be needed in making the case that the costs incurred in these activities will be more than recouped in international student performance, retention, and completion.

On-campus IEPs are home to enormous expertise on international student success. It’s time to put that expertise to work across the campus.

 

Job number one for education managers and leaders

Startup Stock Photos

I used to see one former boss only when he stopped by my cubicle with a question or an urgent demand. Another gave me the shrill and unhelpful advice, “You should be panicking!” in response to enrollment declines. This is not what you would call effective supervision.

It’s a problem in higher education that individuals are promoted into positions in which they have oversight over others, without having undergone training to prepare them for that role. For university IEP directors, the problem can be compounded by the fact that they report to managers who have little or no knowledge of the workings of an IEP. While there are many good people working in higher education administration, these circumstances can lead to strained relationships, loss of motivation, and diminished performance.

The solution is for managers to understand their primary role, their job number one.  It isn’t revenue generation, test score improvements, or student retention, which are indicators of great performance but not activities in themselves. No, the manager’s first job is to support his or her people. They are the ones on the front line of providing service to faculty, other staff, or students, and who are best placed to deliver quality through their work. The manager’s most frequent question to those employees should be, “What can I do to support you?” This is followed by careful listening and the  acquisition and direction of resources to provide that support, so that employees can do the high quality work expected of them.

On the basis of a well-supported faculty and staff delivering quality, managers can have confidence in focusing on goals of increased enrollment and revenue, improved student outcomes, and program development. There are no short cuts to these goals. Don’t forget job number one: support your people.