For Whom the Test Tolls

Why do (some) students cheat on tests? Because they want the test to show better results. Because poor test results will put them in a worse state than good test results. Because they don't want to be judged by poor test results. Because thy don't want to be damned by poor test results. The test is an enemy to be conquered or avoided... Now that's useful.

In the Internet age where it's very easy to create a poll, university professors and judges find themselves being evaluated by students and lawyers respectively. They hate it, and fight it with all the eloquence you would expect from those who make a living by talking and writing. But beneath the learned reasoning for not subjecting themselves to evaluation, what is the problem? Same as with the cheating students: They don't want to be judged and damned. They also don't want to be in the inferior position of reporting to anybody, though if they were guaranteed a glowing report, they may not mind too much.

A similar state of affairs exists wherever a guild manages to keep the power to define the mechanisms that govern it, but this post is concerned only with K12 students. Fixing the rest of the world will have to wait a few weeks.

The students are right: Regular exams at school are designed to judge them. The teacher grading the test may scrawl a few words of reprimand on the wrong answer, but the test as served its purpose as soon as a mark is assigned. Seldom will there be a follow up with a specific student on a specific error. Usually, no one expects to learn from a test. Testing is not part of the learning process. It's an overhead and, may I humbly suggest, a distraction.

I propose to start by deciding that all of what we do at school should directly promote learning - as much as possible. The great majority of tests, quizzes, evaluations etc. should be part of a feedback mechanism (see post) to improve teaching and learning. An exam examines the state of the teaching/learning; it doesn't evaluate if a student is good or evil. It would take an effort, but testing can be repositioned - reframed - as feedback:

The test's questions, even before they are answered, tell the students what the teacher expected them to have learned, and to be able to do. Just reading the test, the student can learn something new. Possible feedback messages from reading the test include: "Hey, you didn't teach that" (feedback to the teacher) or "I see I missed a whole aspect of what I studied" (feedback to the student) or "This test is too easy - we know more than that, so it doesn't provide enough relevant feedback" (feedback to the teacher).

Having answered the questions, or performed the tasks prescribed by the test, and having a teacher or peer review and comment on them, the student gets finer feedback, like: "I see I misunderstood part of the issue" or "The reviewer didn't understand my answer, because I didn't express myself clearly".

Reviewing the results for a whole class can provide extra feedback to the system: Good results mean teaching was done well, and there may be something useful peers can learn from the teacher. Slightly poor results mean teaching methods should be improved. One-time seriously bad results mean the curriculum should be temporarily adjusted to go over the material again, probably with a different method.

Reviewing the results from many classes, accounting for different teaching methods is called "research." It provides feedback for the whole education system. Comparing such results can enable improvement of teaching methodology over the whole system.

Reality keeps interfering: Judgement and damnation are not completely eliminated: Only in extreme cases, when the feedback doesn't work, will the tests ultimately will result in unpleasantness. For example, consistently poor results for a specific teacher's classes may mean the teacher should be removed. Consistently poor results for a student may ultimately mean having to move to a different class, with possible difficult social status consequences. The trick is to keep the main mass of testing as a feedback mechanism, for the benefit of the mainstream students. At the extreme bottom - we haven't solved the problem yet.

Grading and comparison: There is a view that says that every student must be treated strictly as an individual, and not be compared with others. But out-of-school reality doesn't work that way: When a company considering hiring an employee, or promoting an employee, it doesn't use individualized assessments: The contenders are graded and are being compared, and the one with the best weighted sum of qualifications - wins. Even closer to the students' lives, in sports - the fastest runner gets the gold; not the one who tried most. So, a side effect of testing can remain the grades. Competition over grades can be a powerful motivator, though care should be taken to avoid cutthroat competition that seeks to harm the competition instead of advancing oneself.

One aspect of potentially constructive grading and comparison is international tests, which provide feedback for the whole system, showing its standing in comparison to other national education systems. What can we learn from Finland? What can we learn from Singapore?

As always, careful research is required, but it seems that if tests are conceived primarily as useful feedback, they can become that.

The Main Course: Feedback

Borrowing from the wisdom of engineering: An education system is first of all just that - a system: A bunch of entities (students, teachers, schools, government, ...) which together form one entity that has some meaning for humans. The main expected output from the education system are the graduates, who have the skills to function in - and contribute to - society. Every education system is a complex system.

Let's consider a much simpler system - a washing machine: The machine needs to be filled with water in order to start washing. It could be done by opening the incoming water-pipe for a specified time, but then, if the water pressure is lower than we guessed, the machine won't be sufficiently filled; If the water pressure is higher than we guessed, the machine will overflow. We wouldn't think of using a guess. We would use a mechanism that will open the water pipe for just enough time to fill the water tank, but not enough time to cause a flood. We would use a float valve as a feedback mechanism to control the water flow. A system without feedback - an open-loop system - can only deal with expected and pre-analyzed circumstances. It would go haywire as soon as anything interesting happens in its environment.

The first half of the premise for this post is that education systems are by nature complex, and any complex system needs clever feedback loops in order to keep it running. In the case of an education system, part of the aforementioned "running" must be "adapting" and "improving", which requires even more sophisticated feedback loops. The second half of the premise is that most education systems function largely without feedback, or with the wrong feedback loops. Haywire is the steady state.

We can - and should - fill several books with specific discussions of the types of feedback necessary for a good education system, but for now I will just list a few domains for feedback mechanisms.

General:

• Measure what is deemed important, not what's easy to measure
• E.g., quizzes, research
• Get over the fear that being measured means being judged and damned
• Reward desirable behavior to build desirable habits
• Example desirable behavior: Success in personal assignments; Contribution to peers, to pupils and ward in general, to the system
• Example rewards: public recognition, inclusion in desirable work
• Not counter-productive rewards such as being allowed to play/vacation instead of study/work, which send the message that study/work is undesirable
• Penalize undesirable behavior to reverse undesirable habits
• E.g., private admonition, exclusion from desirable work
• Accountability - remove from a position where one causes harm
• Clearly separated from penalty
• The removal is public, not private and hushed
• Peer Review - less threatening, more economical and sends a positive message
• Percolation
• Mechanisms must be maintained for real-life information and insight to bubble upwards from students to teachers to principals to local governments to central education establishment
• Mechanisms must be maintained for research-based information and insight to seep downwards from central education establishment to principals to teachers to students
• Incentives determine behavior - Rewards and penalties must be in tune with the goals of the system

Students:

• Measurement
• E.g., skills more than crammed knowledge
• Reward
• E.g., for making a significant effort to learn
• E.g., for success in learning, as measured
• Penalty
• E.g., for doing slipshod work
• Accountability
• E.g., a student who disrupts the classes can be removed from a lesson, and even from the school
• Peer Review by other students
• Same age, older

Teachers:

• Measurement
• E.g., video-record classes, to allow later analysis
• Reward
• E.g., for students earning rewards
• Penalty
• E.g., for students receiving penalties
• Accountability
• E.g., a teacher who performs poorly must be removed from the position of harming the system and students
• Peer Review
• Teachers should talk more amongst themselves about problems, successes and failures. Engineers do that a lot

School Management:

• Measurement, Reward, Penalty, Accountability, Peer Review

Local government education officials:

• Measurement, Reward, Penalty, Accountability, Peer Review

Central government education officials:

• Measurement, Reward, Penalty, Accountability, Peer Review

Teacher Education establishment

• Measurement, Reward, Penalty, Accountability, Peer Review

Far-from-bottom line:

The mess of bullet-points above can be used as a minimal skeleton for looking into the issue of feedback. Existing research needs to be studied; existing thinking needs to be reviewed; more work is necessary to formulate a suggested structure and connections between the different stakeholders in the education establishment. The existing education systems need to be gradually improved towards the ideal. The ideal will often turn out (based on research) to have been wrong, and will have to be revised. Hopefully, with the right feedback loops, the system will tend to keep improving.

Just to make sure I don't sound too cheery: With the wrong feedback loops, any system will tend to fall apart.

The Ultimate Knowledge Work

The term Knowledge Worker was coined by Peter Drucker, and means a person whose main work "material" is knowledge. Using knowledge and developing knowledge as part of their work. A term that created to describe a budding trend in work-life, now describes more than 50% of employees in developed nations.

This term should emphasize teachers. It should have been created for teachers - for as long as there were teachers (thousands? hundreds of thousands of years?), they were the ones working with knowledge as their main material: Using knowledge, Developing knowledge for and in their students, developing their own knowledge. The term Knowledge Work usually doesn't include students of all ages. But it should. What is being a student if not Knowledge Work? But for this post I will concentrate on the teachers as the ultimate knowledge workers.

If the term Knowledge Work is relevant to the discussion of teaching, it is worth a short discussion of relevant characteristics of knowledge work.
According to Drucker, knowledge workers should be treated as volunteers: It must be clear to them that their work is important; that their work is either being appreciated right now, or it will be appreciated when society understands what it is about; Knowledge workers can't be bribed into doing a good job, but probably wouldn't do a great job if the salary were an insult. They also expect personal satisfaction from work. On the other hand, knowledge workers are often willing to go beyond the job description, which is not very clear anyway with their type of work.

In general, volunteers and knowledge workers have expectations from their organization - beyond the paycheck, and beyond the convenience of the job, and often they are happy to give more. Strangely, teachers are not like other knowledge workers in this respect, at least in the less successful education systems: Most teachers in public schools in these countries are disillusioned, and expect very little from the system. Many of them are there exactly because of the relative convenience of the job. Many of them are worn-out, and it's difficult to maintain a sense of the importance of your work in such a state of mind.

How come? My guess is that the problem starts with the behavior of the education organization, and continues with the behavior of the public: The education establishment in such countries treats teachers as low level officials with a specifically prescribed job - exactly what to teach, when and how. What the organization offers is not satisfaction, but job security - no matter how good or bad the teacher is. The public treats teachers the same way - as low level officials, with no authority. As extras in a play, that can be easily replaced with any other extra. It's difficult to maintain a sense of the importance of your work in such an environment.

Teachers' status is an often lamented issue. We are told that it used to be much better. I am usually suspicious towards nostalgia, so I was very happy to see an account regarding the situation in the beginning of the 20th century, in A.S. Neill's Summerhill School - A new View of Childhood:

"I discovered long ago where a teacher really stands
socially. When I was headmaster of a village school and some function took place the laird came first, then the minister, then the doctor. I was down the table with the head gardener."

So, teacher's status is not a new issue, but these days it is becoming more and more out of place: We do respect knowledge work in general. Almost everybody wants their kids to be as much as possible in the "knowledge business", and if menial work needs to be done - they prefer their neighbours kids to grow up to do that. Teaching is the ultimate knowledge work, and teachers are those who can help our kids become knowledge workers. Moreover, in the world we are now creating for ourselves and our children, more knowledge needs to be accumulated before a person can be said to be a proper graduate; More knowledge will have to be acquired after graduation, so it is imperative that students develop skills and habits necessary for ongoing knowledge acquisition. This makes the teacher's importance even more critical than ever before.

What we need to do is bring our attitude as a public into sync with the reality: The teachers' job is extremely important. Accordingly, we must change the attitude of the education establishment in the same direction - not just as lip-service and a bunch of ads saying how important teaching is, but as a change in the teachers' working environment: Authority with accountability, personal development, support from experts and opportunity to contribute to the system. It may be too much to expect the salary to match the importance of the job, but living wages would be nice.

Naturally, when the status of a given profession is low, many people who could contribute a lot to that profession, go and look for another calling. In contrast with the general civil society, some organizations, such as the military, have an established pattern of selecting the best for teaching: The top graduates of any training course are offered a position as professional trainers. We live in a knowledge-based civilization. We respect those who work with knowledge, and wish our kids to work with knowledge. Discussing enduring importance as opposed to temporary urgency: It would be reasonable and wise, to treat the ultimate knowledge work - teaching - as the single most important profession in modern society.

Learning to Err

A few encouraging anecdotes, that have the extra advantage of being true:

1. A science teacher stands in front of a fifth grade class, and discusses the use of different materials. One of the students suggests using tin for building airplanes, and insists the teacher said that in the previous lesson. After trying to convince the student for a few seconds that she didn't say it, the teacher changes her method and says: "Ok, so I made a mistake having said that tin is good for making airplanes, now please correct me."

2. As part of a one-upmanship campaign, a young engineer jabs a more experienced engineer with a reference to an mistake the senior engineer made. The experienced one acknowledges the error and goes ahead discussing the right way to do things, without even noticing the jab.

3. In a “lessons learned” session, there was a debate about whether some type of activity was done well. Some of the participants say the activity was fine. One of the participants stands up and explains, without any apparent self-consciousness, how he did things wrong and what damage it caused, in order to make sure everybody understands the way things were done was wrong, and to make sure the next time a different method will be used.

Now back to the norm:

I am told by educators that kids these days are very worried about making mistakes, and about being wrong. This is not a great surprise: In polite circles of society, “criticism” is synonymous with “attack.” This implies that making mistakes is unacceptable, and people should adopt a don’t-ask policy about it. Enlightened grown-ups try to shelter children from the possibility of seeing a mistake they made, for fear it will damage the child's self-esteem. It is now considered wrong even to point out a child's spelling mistakes, lest the child becomes self-conscious. I suspect the effect of this sheltering is the opposite of building up self-esteem: There is a message here that we can't deal with mistakes. The possibility of an error becomes more frightening, and the fear of making a mistake has a paralysing effect. If I don't do anything, I don't have a chance for errors, right? If I don't try, I won't fail. Fear of making mistakes is a major impediment to learning.

Another effect, in the cases we manage to disconnect the kids from a sense of having erred, is of diminishing the sense that we can do something well or poorly (What is the politically correct term for "poorly"? "differently well"?) We end up with kids, university students, and adults who are not in the habit of making an effort to tackle something difficult where they may fail, but are likely to improve themselves in the process - this is a complaint I hear from some university teachers. Some of them (by my own observation - not a proper research) have very little sense of quality. For example, they do write with spelling mistakes, but they don't know it.

The teacher in anecdote 1 above, by admitting a mistake, showed the students that it's ok to make mistakes; that they should examine critically information given to them; and probably a few more lessons about self assurance, integrity and such. Who cares she never really did say tin was a good material for airplanes.

Here's a semi-wild idea: It may be useful to actively create opportunities for such teaching. A teacher can occasionally make an intentional mistake, admit the mistake, and let the students correct it. Later, the teacher can simulate a mistake, and let the students realize it on their own.

While we are at it, here is a fully wild idea: It may be useful to build opportunities to make mistakes into the normal course of learning. By giving a class an assignment that has a catch in it, and letting everybody get it wrong. They can all feel secure in the fact that they all made a mistake, so nobody is a fool, and nobody can be the subject of the dreaded criticism. At a higher level, the teacher can give a few exercises, in each of which, about half the class will fail. This will give most of the students an opportunity to be on both sides of the divide. When students are part of the mistaking half, it can increase their ability to deal with their own failure, since it is more difficult to be in a smaller group that made a mistake, than to be part of a whole class that did. When students are on the half that did the right thing, the repeating exercise can reduce the tendency to gloat and to belittle others, since the same students made mistakes just a while ago, and are probably going to make mistakes pretty soon. More advanced erring exercises can let just a small group make the error, or even just an individual. This practice is reminiscent of the cultural engineering exercises in Skinner's Walden two, which is good company.

But even with the tamer methods, if we have more teachers like the one in anecdote 1 above, we may end up with more adults like those in anecdotes 2 and 3: People who think that errors are inevitable when doing anything, and that the natural thing to do with a mistake is not to hide it, but learn from it and let others learn from it.

One more thoughts:

At the extreme end of a Mistake, there is Failure; and about that Winston Churchill once said: "Success is the ability to go from one failure to another with no loss of enthusiasm."

Tests and Announcements

Nobody likes a pop-quiz. It is stressful and - in my case - often embarrassing. In large organizations, the use of pop-quiz is not very common. Instead, they use pre-announced tests. Education establishments are large organizations, and most of the testing in the education systems I am aware of, are pre-warned.

An announced test has a few main characteristics: Firstly, by definition, it allows much time for studying; Secondly, the test itself is short in comparison to the studying; Finally, all the studying becomes irrelevant as soon as the test is completed, and with many people, they indeed manage to forget everything soon after the test.

What does an announced test simulate? To what in real life is it similar? Activities which we know in advance we need to do, which are by nature quite short, where it is very difficult to regroup in the middle the activity, and where failure is very costly. A few examples are commando-type military operations, activities by astronauts, and building unusual structures in extreme conditions. These are not very common activities for most people. A bus driver needs to be prepared all the time for all likely events on the road; A bank teller needs to know what to do with the request of the next customer, without preparing for a day; A mechanic needs to solve problems as they pop up, and not according to a 6-month plan. All these examples are not similar to a pre-announced test, and they are very similar to a pop-quiz: "Here is a problem you should be able to deal with - solve it now."

The good news about the common pop-quiz of real life, is that usually we don't have to remember everything by heart. The bank teller can refer to a policy book; The mechanic can refer to blueprints; Many can go and find relevant help on the Internet. Many can call on an expert. The case of the bus driver is different - there is no time to ask questions when somebody steps in front of the wheels. I will soon get back to that type of issue. The parallel in testing of individual learning achievements, is an open-book pop-quiz where a range of supporting materials is available for the individual being tested: Books, notes, limited access to Internet, and maybe even limited access to experts.

Back to the bus driver; Indeed any driver. The driver's real-life pop-quiz is along the lines of "you are driving on a highway; a car in front of you just stopped; you remember seeing another car in the lane to your right; you don't remember seeing any car in the lane to your left; no time to look in the mirrors to see which lane is vacant; in a second you are going to be riding the steering wheel of the car ahead of you, having killed a few people; DO something NOW." An announced test will do very little to verify a driver's ability to deal with the situation. An open-book pop-quiz will do very little. Actually, this example is so extreme that even a closed-book pop-quiz will not be enough. We would need either a simulation, or a real-life observation test, which is what we do for testing driving, flying, etc.

A less extreme example is that of a spoken language proficiency test. In real life conversation, it is very inconvenient to keep referring to a dictionary. Communication should happen with what the speakers have right now. This is a good example where a pop-quiz with no available materials can do a good job.

Bottom Line:

There are several different levels of testing: Announced tests with open or closed books, pop-quizzes with open or closed books, simulations and real-life observations.

There are several different take-home benefits from school: Habits, skills and knowledge.

Knowledge in real life is generally used with little urgency - the usual modern bustling - and with time to look and check. Very similar to an open-book pop-quiz. Knowledge is also often quite specific - there is often just one right answer to any given question, which lends itself even to multiple-choice quizzes.

Skills in real life are generally used with more urgency, and with less time to check and ask. Mostly similar to a closed-book pop-quiz. In some cases, such as driving, we need to go further - to simulations and real-life observations. Skills are less specific, in the sense that there are often a few right answers to any given need.

Habits in real life don't lend themselves to a discussion about urgency or about opening a book. They are very non-specific, in the sense that there are many reasonable responses to any given situation. If we want to evaluate habits, a simulation is a minimal tool; and we will probably have to go with real-life observation.

The vague case for pre-announced tests: It may be that preparing for an announced test is a good practice for memorization and conscious building of knowledge and skills. It may even be a basis for the good habit of putting a concentrated effort into studying something. Another value for an announced test could be found in a graduation ritual: Study for a month; go over everything learned; get tested for several hours -- now you have changed from a student to a graduate. All these taken together can justify some announced testing, but certainly not as a major tool for evaluation.

Experiments? NIMBY!

NIMBY - Not In My Back Yard - is the standard human response to many situations and proposals for the betterment of society. A new prison? A must, but not in my district. Move Guantanamo Bay prisoners? Certainly, but not in my state. Conduct research on education, with it's accompanying risk of error? Can't do without it, but not in my kids' school.

Usually the NIMBY response is discussed with disdain, because there is something inconsistent about agreeing something is necessary, but not being willing to shoulder a part of the burden. But while disdain makes the disdainer feel morally superior to the disdainee, which let's admit is a great feeling, it doesn't solve the problem: Unless we find another type of solution, we still need to build a new prison, and it is going to be in somebody's back yard.

The reason we parents are unhappy with research in our kids' school, is that we are very protective of our kids, and want nothing bad to happen to them. That's good. We also tend to be quite conservative with our kids (as well as with our own survival.) The sense is that what didn't hurt us yesterday is not likely to hurt us today, so why change and run the risk that the new thing will hurt us. Completing this tendency is the fact that meaningful research is not just observation, but trying something and seeing if it works. And often it doesn't work. For example, trying a new way to learn Math may create, in a year or two, a group of children who are a year or two - or more - behind in their Math studies. Nobody wants that for their kids.

In some cases, the solution to the NIMBY dilemma lies completely in the mind, and is relatively easy. Such is the case with research in education. Reality check: Our kids are already subject to risky trial-and-error.
Example 1: A new teacher in school - is he going to be great, mediocre, or terrible? Test as we may - we will have to wait and see.
Example 2: Teaching kids about he Internet - it has never been tried before, so who knows if we are doing it right? Worse: The Internet we teach about today is going to be obsolete in just a few years. Will the way we teach it now serve our children or not?
Example 1+2: The teacher who is known to have been a pretty good teacher for 20 years. Now that same teacher needs to teach new kids in the context of new technology, new society, new values... Is the teaching going to serve the kids or hurt them?
Example 3: Tried and true methods of teaching are based on the assumption that the teacher knows everything necessary for the kids to know, and is responsible to convey that knowledge. This is no longer the case: Knowledge is available through the Internet - much more than what we can expect the teacher to know; The kids often know facts not known to the teacher; What used to be true, is now often called into question; Authority in general is now limited - uncertainty rules. There are no tried and true methods to deal with the current reality, so many school systems are changing the teaching methods of everything beyond the ancient subjects of language and basic math. New teaching methods mean more trial and error.

What stems from these examples is that trial and its accompanying error are already here. They have to be here in order to deal with both ongoing and changing reality. The question we need to ask is not "do we want trial and error?" but "how do we want our trial and error?" I propose the answer: "Scientific, and well managed."

There should be certain rules to experimentation in a context that can cause damage to something important

1. No experiments that can cause irreversible damage.
   That's not a real problem with education - the human (child) mind can heal very well
2. Controlled experiments on limited population.
   Unlike some education establishment nowadays, which tend to experiment on the whole population at once
3. Close-up tracking to identify unwanted effects, and quick re-evaluation in such cases.
   E.g, If a new method for teaching reading and writing doesn't seem to work well - stop quickly
4. Total commitment to correct any unwanted effects - even if the cost is high.
   In the context of education, this may mean bringing a super-teacher to help children catch up with lost materials, or to help them regain a positive attitude towards a subject
5. Continuous involvement with the population. In the 21^st century, it is not a real option to keep the people in the dark regarding experiments being done on them.
   In the context of education, it's the population's parents.
6. Scientific methodology in general: Double blind tests; Control groups; Published results for peer review; Repeatable experiments, etc.

This sounds a bit like a political statement, but I think it can withstand serious scrutiny: Keeping to those rules, and probably a few more, can help minimize damage and maximize gain from the inevitable experimentation that goes on in the education system.

Wild Ideas - Student Teachers

All I ever learned about education, I learned from my kids. That's an oversimplification, but a useful oversimplification.

The 3-year-old wounded his lip, so the 7-year-old showed him an animated movie from the "Once Upon a Time ... Life" series, discussing the platelets and how they help fix wounds. The elder explained a few things to the younger. The elder got to practice teaching; She understood the issue better herself for having had to spell it out: It is a familiar experience - that explaining something to someone else improves our own understanding. Of course she also enjoyed the high status of knowing something - of being a teacher. BTW, did you notice the issue of a low status for teachers and teaching is not inborn but acquired? Young kids look up to those who teach them anything, and would like to be like those teachers. The young kid enjoyed learning, improved his esteem of his older sister. In general, the kids' relationship improved, and no grown-up had to be very active.

Everything worked out nicely. This is good news. The great news is that this sort of interaction is not a one-time accident. It has a name - "peer mentoring"; it is a natural way to transmit skills and knowledge; and it is scalable, and (this is where we are getting wild) it can be used to tackle directly the problem of class size, and the related problem of teacher attention.

In the extreme, it may be claimed that kids can teach themselves, as shown by Sugata Mitra's amazing discussion of the "hole in the wall." But even if we remain just wild but not extreme, and we build peer mentoring into the core of the schooling methodology - and not just as a curiosity - we may achieve a lot: A sense of empowerment to the mentor; Improved depth of understanding for the mentor; Practice of teaching (in real life we teach each other all the time); Much more personal attention concentrated on the learner (1:1 instead of 1:40); Freeing up the professional teacher to perform higher level tasks, such as directing the learning and observing individual learners and mentors.

The word "may" above is highlighted, to emphasize the fact that this is an idea - a claim - and not a fact. To establish the claim as a fact, we need research. That's a hassle, but the rewards may be great.