I spent a pleasant and stimulating day at a conference on language teaching Sunday the 26th of May.

Why pleasant? There were only 70 or so participants, plenty of space between the desks and chairs in the classrooms, 15 minute breaks between sessions so you had time to have more than perfunctory conversations with people.

Why stimulating? Both participants and those who led workshops or gave plenary sessions each had a distinct message which they invited participants to experience.

But though I found the day pleasant and stimulating, I was troubled as I always am by the fact that each person on the program followed the usual rule that the goal is to convince participants that the activity they are introducing always works great.

There have been a spate of articles recently criticizing studies in the field of psychology for presenting only positive results. When the authors of articles were questioned they admitted that they jiggled the data to support their claims. They said that when they submitted negative results to journals their articles were rejected!

In the December 2012 issue of The Atlantic, I read a piece titled The Data Vigilante. One Uri Simonsohn, a research psychologist at the University of Pennsylvania’s Wharton School, has been examining studies to point out initially loose methodology. But he found flagrant dishonesty! (pages 32 and 33)

At the conference I just attended, I do not think any person on the program was dishonest. In fact, I cannot imagine any of them even knowing how to be dishonest—jiggling data to support their claims, for example. But I do think that their methodology was in Uri’s word “loose”.

I think it was loose for a number of reasons, none related directly to the intelligence or integrity of those who presented. One reason comes from pressure to show that we are effective from parents, principals, school boards. Another comes from sometimes unconscious competition with colleagues who do not think that what we are doing is as effective as what they are doing. As I implied above, a central reason is that if we try something new that others have claimed is successful we expect it to be successful.

The basic flaw in all claims that an activity works is that those who make the claims have not compared the results with related activities. Here are a few examples to illustrate this point.

Claim 1

You have to use extensive reading in your classes. When students read graded readers that are of interest to them, they increase their ability to recognize more and more words. I have tested my students by showing them flash cards of the words in the graded readers they have read and they can say the meaning in their first language very quickly.

Counter Claim 1

While students might recognize more words, we have to ask a few additional questions.

a. Though not all words they recognize have to be used, it is important to check how many they can use correctly.

b. When students are asked to read sentences aloud that they had read silently and then write what they said, if they write “How can I every change they go to feel.” For the sentence “How can I ever change things that I feel.” We see that the development and mastery of sentence patterns and structural words, as well as meaning, are not accomplished by extensive reading.

Claim 2

When students have a lot of rich, natural input at their level they will develop their speaking skills. My students say they like it when I speak naturally.

Counter Claim 2

One of the teachers recorded his “rich, natural input at his students’ level” and then asked the students to listen and write what he had said. They could replay the recording as many times as they wanted.

After you read what the students wrote can you write what he had said? Here is what the students wrote:

The hotel have dress paud

The hotel

The hotel has a dress cort

The Hotel has

The hotel is gre cool.

If none of the students were able to write what was said how can they be developing their language skills?

Claim 3

As long as the students get their meaning across I do not correct them. Mistakes are OK. The more they speak the more they will increase their confidence and therefore become better at speaking.

Counter Claim 3

A student who wanted to study in English in Canada was asked by her teacher why she wanted to study in Canada. Her response was

Canada beautiful nature.

The teacher said, “Oh, you think Canada has beautiful nature.”

The student said, “Yes.”

The teacher was so used to projecting what she thought her students were saying that she simply said what she thought was normal English.

It teachers accept whatever students say because they think they get the meaning, any confidence that the students might gain—and it is almost impossible to measure confidence—will be dashed by the first few conversations the student has with native speakers of English when they show bewilderment at what they say.

The lack of attention to accuracy—the dichotomy between fluency and accuracy that is part of the current jargon—makes no sense since fluency has to include accuracy or it is meaningless.

But my main point, as always, is that the only way we can make the claim that anything works great is to contrast the activity we are using with another one and compare the results.

I was on a panel at the end of the conference and one of the questions I was asked was “What do teachers need to know?” I did not plant the question! It was of genuine concern to the teacher who asked it.

My response was “You have to know what you are doing and the results of what you are doing. And the only way you can know the results of what you are doing is to compare it with a similar activity with slight variation and compare the results.”

I told the participants that I had just had a conversation with a teacher who corrects compositions using symbols. He gives the students the list of symbols with examples at the beginning of the semester and for ten years he has been using these symbols with great success. For example, if the student writes “Two years ago I go visit my grandmother.” he writes WVU for Wrong Verb Use above go so the student can correct his error. When I asked whether the student made the correction he said that he does not check because when he asks them what WVU stands for they all know it stands for “wrong very use.”

To do something for 10 years and never check to see how many students actually make the changes that you want them to make and not to compare over time whether making the changes with using the labels leads to a lower frequency of the error is a huge drag on not only making changes in our field but in improving outcomes.

I am attaching some suggestions for exploring what you do so that over time when you say, “This always works great!” you can support your claim.

I will be discussing my ideas about exploring our teaching in a live webinar on the 2nd of June 9 pm Tokyo time. If you go to this website you can get the details about connecting with the live seminar. In it I will also be introducing a live on line course I will be teaching for the 5 subsequent Sundays. Here is the site: