Tag Archives: student learning outcomes

Other news

Getting Serious About Quality

“A high quality English learning experience.” “High quality English instruction.” “High quality courses.” “Quality English lessons.” “The highest quality English programs in the area.”  

Defining Quality
Whether explicitly stated in our materials or not, we all want to offer quality, but it’s an elusive concept. The dictionary gives two definitions:

Definition 1: degree of excellence; how good or bad something is

Most likely when programs make a claim for their ‘high quality,’ they have something like this in mind. Unfortunately, ‘excellence,’ ‘good,’ and ‘bad’ don’t bring us any closer to a definition of quality, since they in turn need to be defined.

Definition 2: the extent to which something is fit for purpose

In other words, it is good quality if it does what it is supposed to do. This is unsatisfying if your program wants to market itself as standing out from the crowd. Some combination of the two, such as ‘we are ideally set up to do what we do, and we do it excellently’ gets us closer, but is still just as easy a claim to make as ‘we are high quality.’

If your program wants to avoid using the word ‘quality’ as more than marketing copy, you ought to substantiate the claim. (In fact you should do this even if you don’t explicitly use the word quality.) You not only need to satisfy potential students and other stakeholders of your quality; you also need to define it for yourself.  You need to determine what constitutes quality at your program as a benchmark against which to measure your program’s performance.

Elements of Program Quality
There are four bases on which you might define and claim quality for your program.

  • Quality of inputs. Inputs include teachers, program staff, the curriculum, the textbooks, and educational technology. For example, you might emphasize the credentials, experience, or personalities of your teachers; your highly researched and trialed curriculum; or your state-of-the art classrooms and digital learning opportunities.
  • Quality of the experience. You might emphasize your modern, comfortable facility, your carefully vetted and monitored homestay families, or your exciting social and cultural activities.
  • Quality of outcomes. While the inputs and the experience may make the process of learning at your program positive and comfortable, students are there to make progress in their English. Quality of outcomes may include increased test scores, successful achievement of measurable outcomes based on reliable and valid assessments, and job placements or college acceptances that depend on English attainment.
  • External recognition of quality. If student completion of your program is accepted by universities as an indication of English proficiency for admission programs, you have an external acknowledgment of your program’s quality. Some programs mention their accredited status as a mark of quality, or collect and publish testimonials.

Evidence for any of these claims can support the ‘we do it excellently’ and the ‘fit for purpose’ definitions of quality, but a convincing picture of program quality will need to be based on a combination of all four. A program may have knowledgeable and stimulating teachers (input), but students will not be satisfied if they are sitting in a poorly lit, unventilated classroom (experience). Accreditation involves meeting a large number of program-wide standards (fit for purpose), but has limited ability to communicate excellence above and beyond other programs, hundreds of which are also accredited.

Measuring Quality
A program that is serious about quality and about substantiating its quality claims should have quality assurance measures in place. Quality assurance means not only systematically collecting information; it also means reviewing it, sharing it, and using it to make improvements. Many programs are effective at collecting information, but few seem have consistent procedures in place to make use of it to improve quality.

Most measures of quality address inputs. Only relatively recently have measures of outcomes become more prominent, as a result of calls for greater accountability in education. The last two items in the table below are outcome measures.

MeasureQuality TypeQuestion to Ask
Class observationsInputs/ExperienceAre there plentiful affordances for learning in the classroom, and are students making use of them?
Course syllabi  InputsAre syllabi consistent with the curricular goals and objectives?
Course or teacher evaluation forms   InputsAre students delighted with their classroom experience and learning?
Student surveys  Inputs/Experience/ OutcomesWhat common themes indicate what your program is doing well or poorly?
Student needs analysis   InputsHow closely aligned with students’ expressed needs are your courses, skills, and language knowledge?
ComplaintsInputs/Experience/ OutcomesWhat do students express dissatisfaction about?
Suggestion box Inputs/ExperienceWhat are students asking for that you are not currently providing?
Teacher and staff retention  InputsIs your workplace attractive enough to maintain continuity in a faculty and staff that have a stake in your program’s quality?
Teacher feedback  Inputs/ExperienceWhat can you learn from your teachers about the student experience?
Accreditation feedbackExternal recognitionHow many of the accreditation requirements are you meeting and not meeting, and in which areas?
Student achievement  Outcomes (direct measure)Are valid and reliable measures of student attainment of your program’s learning outcomes being used to determine the level of student success?
Job placement/College acceptances (if applicable)Outcomes (Indirect measure)What percentage of your students are meeting English language standards for their academic or career goals?

Using a mix of these measures, you can develop a quality dashboard to benchmark your current quality performance and set goals for improvement. Provided the measures yield positive information, you can use the information to substantiate your quality claims and strengthen your sales messaging.

Not a bad strategy in an always-competitive field.

For Schools

Education – the journey and the destination

Leave a comment

With a high school student about to enter 12th grade, we are finding colorful mailings from colleges in our mailbox every day. It’s a competitive market for students, and college marketing offices need to make their institutions attractive. 

As I was sifting through the pile, something struck me about how schools are trying to appeal to my rising senior. For over 20 years now, education at all levels has been driven by an accountability ethos. This is intended to ensure schools’ accountability for their quality by defining student learning outcomes (SLOs) and reporting students’ achievement of those outcomes. In an age of rising education costs and the demand for value for money (and for institutions that want to benefit from federal government financial support) this is considered to be good for students. And the definition and assessment of SLOs has become a centerpiece of accreditation standards. 

The funny thing is, I’ve never yet met a student who cares much about student learning outcomes. And if students did care about them, you would think the college marketers would have cottoned on to that fact by now. But look at some of the messages in the mailings we’ve been receiving…

“A college experience built around your definition of success.” 

This doesn’t sound like a college that is pushing its SLOs as a selling point. It is about the student finding meaning in the college experience. 

How about this student quote from another mailer…

“I’ve had professors who truly care about what I am doing and how I’m moving forward in my life.” 

I doubt you will find a requirement for ‘professors who truly care’ in your accreditor’s standards. 

“Best college town, extraordinary college experience.” 

Again with the ‘experience.’ Again, not a sales pitch based on outcomes. 

“Your (college name) story starts here.” 

College as a story – something filled with experiences and will create memories and meaning. 

“The question isn’t where you want to go. It’s how to get there” – an explicitly anti-outcome statement. 

And finally, my favorite: 

“You’ll never be bored in Buffalo.” 

Enough said? 

My point is that in the push for accountability and compliance, we can become too focused on outcomes that students may not be very interested in, and not pay attention to the quality of their experience. When we focus only on results, we can forget that what makes an education memorable is the location, the personalities, the interactions, the participation in the process, the experience of undergoing all that. A quality education should be rich in experiences, should encourage personal growth, should open us up to different ways of understanding the world. This is all very difficult to express as a measurable learning outcome. 

Does this matter? I think it does for several reasons. 

  • What does the grade for your course represent? Is it only the final result, the achievement of the outcome? Most teachers want to include assignments, projects, class participation and contribution, and quizzes – formative activities on the way to the goal – as part of the final grade. This is because teachers know that an education is not simply about the destination but also about engagement in the process. Students who fail to engage in the class but nonetheless achieve the learning outcome may have ‘succeeded’ in meeting an outcome, but may have failed to gain an education.


  • When can you consider that a class ‘worked?’ An education professor criticized teachers she observed who thought their class had ‘worked,’ because they failed to clearly define and assess an outcome for that class. But perhaps teachers know that ‘working’ can also mean having students engage in a process, in activities that enrich them in ways that are hard to measure – none of which is interesting to an outcomes-only oriented observer.

  • What is important when designing an educational program? Striving to comply with accreditation standards, many schools are focusing heavily on teaching to and assessing outcomes. An equal focus on how to get there – the journey too, not just the destination – should be taken seriously by schools and those who hold them accountable.

  • Is online learning just as good as in-person learning? From a purely outcomes point of view, maybe. Good online learning can of course be rich in experiences. But is it possible for good online learning to be as rich and engaging as good in-person learning? Personally I doubt it. 

And here comes the caveat, of course. I’m not arguing that establishing and assessing outcomes is not important. Clearly if education is going to consume students’ resources – their money, time, and effort – they want to know that they will gain knowledge and skills from it. I am concerned though that – as in many areas of life – the pendulum can swing too far in one direction. I’m arguing that it ought to swing back some way. College marketers, teachers, and students have understood this all along.

Other news

Inputs and outcomes – how we wound up with two systems for grading students in english language programs

row-students-doing-examAssigning final grades to students has been done in various ways over the years. In some contexts, everything rested on a final exam – this was the case with the O-level and A-level exams I took in a British high school ‘back in the day.’ Then ‘continuous assessment’ became popular, making the final grade a composite of grades for assignments completed during the course, either with our without a final exam.  This approach became popular in U.S. intensive English programs, where the final grade might be made up of homework assignments, projects, tests and quizzes, and the usually ill-defined ‘participation’ by the student. 

But English language programs, like all other schools in the U.S., became caught up in larger forces that had an enormous impact on how students were evaluated. Following the successful launch of the Russian satellite Sputnik in the late 1950s, there was much nail-biting over the quality of American education, culminating in the ‘A Nation at Risk’ report in 1983, which painted an anxiety-inducing picture of failing U.S. public schools. 

In the years following the publication of the report, the means of defining quality in education were questioned. The focus of quality tended to be on inputs – number of hours in class (this is where the ‘credit-hours’ system came from), teacher qualifications, teaching methods, and so on. Many schools in competitive environments still make such inputs the basis of their quality claims – “highly qualified teachers!” “innovative teaching methods!” 

But those raising the red flag about school quality were less concerned with inputs and more concerned with what the students came out of their education with – that is the outcomes, or as they have come to be known, student learning outcomes, or SLOs. No matter how great the inputs, if students were not learning useful knowledge and skills for the job market, the education they were receiving was not valuable. The solution was to turn the traditional curriculum planning process around and start at the end by first defining the desired outcomes, and having course design lead to student achievement of outcomes. 

This enabled education bureaucracies to hold schools and teachers accountable: school and teacher quality could be judged not by the quality of the teachers or the hours spent in class, but by the extent to which students were meeting the defined learning outcomes. In the public schools, those outcomes were assessed by standardized tests, and schools were judged and ranked by how well students scored on those tests. (The downside of all this was that quality aspects of school such as adequate breaks between classes and time for the arts, music and sports suffered as schools honed in on efforts to increase standardized test scores in math, science, and English.)

Back to grading. ESL teachers have for many years been used to giving final grades based on a combination of test and quizzes, homework assignments, projects, participation, and final exams. But with a shift toward accreditation of English language programs – mandatory in many cases, voluntary in others – teachers in those programs are now required to fall in with the requirement to define learning outcomes at the outset, and assess and evaluate students with sole reference to the students’ achievement of the outcomes. This has to be done at the school level, and it results in a greater standardization of curricula, syllabi, and assessments in schools. Schools are required to record and analyze the data arising from the assessment of SLO achievement. Decisions about whether a student may progress to the next level of study or complete the program successfully must be made solely on the basis of whether the student achieved the learning outcomes. 

The result is that schools have to take a mixed approach to grading students. Schools may still assign a traditional grade based on continuous assessment and participation, but they must also maintain a system that isolates achievement of the student learning outcomes and makes promotion and completion decisions based on that. What’s certain is that choosing one or the other of these two systems is not possible – both are needed. Yes, we can agree that it’s important for students and their sponsors to understand what the expected outcome of a course or program was and whether the student achieved it. This kind of accountability is needed when many are questioning the dollar value of their education. But as educators we also want to know whether students engaged with the educational process – collaborated with peers, challenged themselves on difficult projects or assignments, sought help and advice and gave them to others in turn. How the students got there is important to us, and still largely defines the benefit of studying at one school rather than another. 

And so we ended up with two types of student assessment and evaluation, one based on inputs into the process and ongoing or continuous assessment, the other on outcomes. Both systems are here to stay, and educators need to be familiar with the rationale and procedures for each of them. 

Background photo created by pressfoto – www.freepik.com

For Schools, Other news

Why you should be teaching to the test

Back in the day, if you were ‘teaching to the test,’ you weren’t really doing your job as a teacher. You isolated the pieces of knowledge and the skills that you knew would come up on the test and taught them to the exclusion of broader educational activities that might have enriched the students’ experience. You might have done this to ensure a high pass rate, which reflected well on you as a teacher if the higher-ups were judging you on your students’ test scores. But teaching to the test was frowned upon as a kind of shortcut for both teacher and students.

Since the advent of the accountability movement, teaching to the test is exactly what you should be doing. In the currently popular paradigm, schools and teachers are accountable for students’ achievement of defined learning outcomes, expressed in behavioral terms: “The student will be able to…” Examples in language programs are:

  • give a five-minute presentation on a topic of personal interest
  • write a five-paragraph narrative essay
  • summarize, in writing, a radio news story
  • re-tell orally the plot of a short story

If the learning outcomes are well-conceived, they should be a guide to what the test – let’s call it an assessment – should be. How do you assess students’ ability to give a five-minute presentation? Have them give a five-minute presentation at the end of the course. How do you assess their ability to write a five-paragraph essay? Have them write a five-paragraph essay. And so on. (The specifics of the assessment will need to be made clear, and rubrics provide a means of determining the students’ level of performance.)

And so what is the best way to prepare students to give a presentation? Teach to the test and have them practice giving presentations. To write an essay? Teach to the test and have them write essays. This is what I mean when I say ‘you should be teaching to the test.’

Some summative tests – including many of those provided in published textbooks – are not good tests to teach to. A grammar gap-fill isn’t much use in giving information about a student’s final level of achievement, unless your learning outcome is ‘be able to provide the appropriate grammatical forms in a gap-fill test.’ That’s not a very useful outcome to anyone, though this activity might help promote student learning along the way. A well-defined learning outcome is a behavior that you can describe to a future employer or school indicating the student’s ability to do useful things with language.

So let’s embrace teaching to the test – as long as you have good learning outcomes and a corresponding test that assesses them appropriately. (If you don’t, maybe it’s time for an overhaul.) And while we’re doing that, let’s not forget that games, songs, poetry, sharing experiences, and laughter create a positive, human environment that leads to unanticipated learning and ideal conditions for students to learn.

 

 

For Schools

The baby and the bathwater

If you read anything about curriculum design these days, or attend a presentation or workshop, you will learn only one model. Backward design starts at the end, defining student learning outcomes, then working backward through assessment, teaching and learning objectives, content and sequencing, and finally teaching and learning. This approach to curriculum design is so pervasive that anyone new to education might think there is no other way.

Thanks to the recently published second edition of Jack Richards’ book Curriculum Development in Language Teaching, we can learn about, or be reminded of, the alternative. Although it was never recognized with the name, forward design took the opposite approach: decide on your content and sequence it, teach, assess, and evaluate with a grade. This used to be the standard way to design  and teach a course in higher education. And the fact is, many teachers who used this approach continue to do so, or try to do so, in tension with an institutional backward design ethos promoted by accrediting organizations.

Backward design has the benefit of identifying (in theory anyway) student needs, developing measurable learning outcomes, and demonstrating program quality through analyzing and publishing student success rates against those outcomes. There is accountability for student success, which is important in an increasingly competitive environment in which customers (students and their sponsors) demand transparency in results. Yet as we are reminded in Richards’ book:

“The experience of language teachers today is often one of diminished classroom autonomy and of being managed by business-savvy administrators.” (Hadley, 2014, cited in Richards, 2017, p. 228)

This is because teachers are increasingly told to work with standardized outcomes and learning objectives, demonstrate that their assessments address the student learning outcomes, and use textbooks that deskill teachers by driving many of their instructional decisions (Richards, p. 247).

If backward design has introduced an obsessive focus on outcomes, or product, forward design was always much more about the educational process. A process curriculum:

“…is person-centered, considers users’ needs, identifies problems rather than rushing to solutions, and does not rely on top-down mechanistic models but is a process that works towards interaction between participants at all levels.” (Kennedy, 2013, cited in Richards, 2017, p. 227)

The emphasis is more on what teachers might call the art of teaching, making meaningful experiences for students, and letting the teaching and learning follow, to an extent, students’ needs and interests as they arise during the course. In a process approach, students and teachers form a learning community that explores together, often with an unclear destination. While backward design requires assessment of students’ achievement of learning outcomes exclusively, forward design grades participation, homework, and attendance, because these are indicators of students’ engagement in the process of learning.

While I don’t think we can or should return to a purely process-focused approach, we should consider what is lost when we throw out the process approach baby with the forward design bathwater, and embrace a product-based approach too strongly. Now that backward design is established as the accepted way to design curriculum, I hope that we can start to talk about how to maintain art in the process of teaching, one that recognizes teacher creativity, responds to the students in the class and the needs of the particular group, and provides a unique and unrepeatable experience for learners.

Thanks to Richards’ book, we may be able to start having that conversation.

Jack Richards’ book Curriculum Development in Language Teaching is published by Cambridge University Press.

For Schools

What makes a school?

We are used to talking a lot about quality in education. It used to be normal to describe quality in terms of inputs: faculty to student ratios, faculty degrees, school facilities, test scores of the incoming class, and so on. More recently, we have been pressured by government departments, funding agencies, and accreditors to prove our quality in terms of outcomes: can-do statements, demonstrable skills gained, behavioral changes in our students at the end of their course or program.

The input-outcome paradigm for determining quality is adequate enough if we are in a production mindset. In this mindset, education is analogous to the production of goods or services. “Our shoes are made of the finest Italian leather” is an input-based quality claim. “Kills 99.9% of bacteria” is an outcome-based quality claim. Similarly, “highly qualified and friendly teachers” is an input-based quality claim. “Our Academic English program will prepare you to succeed at a university” is an outcome-based quality claim.

I wonder if this paradigm tells the whole story about the quality of an education? This is important to consider for intensive English programs (IEPs), because they are increasingly competing against other models of English language education and training, such as in-country classes, online tutoring and courses, even apps on devices. The producers of these alternatives can point to their inputs and outcomes and on that basis apparently offer a viable alternative to an intensive English program.

But I want IEPs to revive the notion of the school, a concept that is far broader than mere production or educational delivery. If you think about your own educational experiences which were the most memorable? Which shaped you most as a human being? Were you most influenced by a program that had clearly defined student learning outcomes? Did you learn the most from the teacher who was the most highly qualified? Unlikely.

A school is a place where community is formed. Diverse (however you choose to define diverse) students, teachers, and staff, come together in the shared enterprise of teaching and learning. There is social interaction, friction, the challenging of dearly-held beliefs. There is laughter, disappointment, joy, and frustration. Students encounter teachers with idiosyncrasies that they will never forget. A story heard sticks in the mind forever. A kindness is extended and remembered.  All of this is the quality of an education that is not recognized by the production paradigm of educational quality. It cannot be measured. You cannot really put a price on it.

Think about this kind of quality when  you go to your school in the morning. You still have to hire qualified teachers and measure student learning. But you have the opportunity to create unique, precious, and lasting experiences for your students, staff, and faculty. This is what really makes a school.