Non-Goals of FIRE

When mapping the domain of FIRE, it is useful to consider and define what FIRE is going to do. It is also useful to spell out what activities and interests are explicitly out of FIRE's scope.

1. FIRE does not perform research.

This emphasizes the mission of FIRE to facilitate research rather than to perform research. Performing a research project costs much more. So for any finite amount of money available to FIRE, the number of research projects that can be performed and fully financed is much less than the number of research projects that can be merely supported. For example, consider the case of FIRE funding a full research project, contrasted with the case where FIRE only funds an extra follow-up phase on an already funded research: If FIRE obtains $1,000,000 in funding, and if an average smallish research costs $100,000 and the cost of a small follow-up phase is $10,000 - performing full research projects would mean contributing to humanity 10 projects. Supporting specific aspects - follow-up in this case - would mean contributing 100 projects. 100 is greater than 10.

Another set of reasons for not performing research, even if funding is practically unlimited has to do with academic capability and integrity: A single research organization doesn't have access to all the knowledge and ideas. If we had just one university, we wouldn't have as much research - and presumably knowledge - as we do now. More research and thinking capability stem directly from diversity. Also, any single organization concentrating on research is in danger of developing conceptions about the world and an agenda that may dominate the types of research it does. Integrity can be found in diversity.

2. FIRE does not dictate research topics or domains.

Even without full financing of research, as ruled out in 1. above, an organization may have too much power in determining agendas. Avoiding having excessive influence in this way, would take ongoing attention. This is because FIRE does intend to be active in the vicinity of this non-goal, without being active in the non-goal itself. One such adjacent activity is maintaining a pool of possible research questions that lend themselves to actionable research. It is important for FIRE to succeed in maintaining a list of optional research questions while not letting it degenerate into a list of recommended research questions or even further into an exclusive list of admissible research questions.

3. FIRE does not promote any specific theory.

If FIRE is not careful, and becomes partial towards one theory or another, it would lose the trust of those not supporting that theory. And since it doesn't seem that a single theory of education is going to emerge any time soon, there are going to be different people who support different education theories - or should I say education faiths? Anyway, I never met a serious educational researcher who claims to have the one TRUTH.

4. FIRE does not promote any specific terminology.

"Language Creates Thought." The terminology used by researchers at the very least affects their thinking, and definitely defines the meaning of any research. For example, consider the issue of scholastic aptitude: If a researcher looks at scholastic aptitude level, this may lead to making tests and measurements of an individual regarding a predetermined set of knowledge and ability domains, and coming up with a number - a level. If another researcher of the supposedly same issue by looking at scholastic aptitude domain, it may lead to looking at different knowledge and ability domains, and checking where an individual has high aptitude. The researcher interested in "level" may come to a conclusion that the individual's level is low, while the researcher interested in "domain" may come to a conclusion that the same individual is a genius painter.

The above example may be oversimplified, but there are many sets of differences in terminology, which affect the researcher's actions and interpretations. Just one example from the academic world, and we will go back to human language: Trying to model the relations between interest and knowledge, K. A. Renninger looks at a matrix of different levels of Prior Knowledge and Value of Activity, while S. Tobias looks at Prior Knowledge and Personal Interest. The subtle difference between "Value of Activity" and "Personal Interest" can have an effect on what the researchers look for, and how they interpret what they find.

It would be very tempting for anyone trying to build a body of knowledge to base it on a single set of concepts. But what is the "right" set of concepts? What even is a "good" set of concepts? This is an issue for academic discussion, and not for an organization like FIRE, which is not concerned with generating new theoretical ideas.

5. FIRE does not provide scholarships.

To succeed, FIRE needs to be able to focus its attention on a cohesive set of activities. Other organizations provide scholarships and have the relevant know-how. It seems that FIRE wouldn't have anything significant and new to add in this realm.

=============

As is the case with many facets of FIRE - and of any other initiative - there is no end for the possible details regarding what FIRE could but would not because it should not do. The above sampling shows the direction.

Financially Challenged Research

Most of us need more money. At least we tend to think that. Researchers have a special name for money - they call it "research funding". Theoretically speaking, if grant money was freely available, we could expect more and better research. But it's not freely available, which causes - beyond the trivial difficulty in funding any research - a host of specific limitations on research that does get funded. Some of these limitations make it more difficult to perform actionable research, which is the domain of discussion in this blog.

Often (always?) a researcher faces a situation where the available funding can provide for a limited research and does not allow for everything the researcher would have liked to do. Then the researcher needs to decide whether to drop the particular research, or do what they can while being aware of the limited validity of the results. Neither decision is a happy one.

A few examples of money-related limitations on research, and ways FIRE can improve the situation:

1. Small sample size

When studying a large population - E.g., the millions of first-grade-age kids at any given year in Africa - it is not practical to directly study all of them. We usually study a smaller group - a sample of the population. Performing a research on a small sample is sensitive to statistical and methodical problems. It is error-prone, so the results can't be completely relied upon as a basis for action. All researchers know that, but when a researcher obtains a fixed amount of money, this will limit the sample size the research can handle. FIRE can complement the funds of a grant specifically in order to enlarge the sample.

2. Short attention span

Often a study can benefit from being extended in time. This extension can have the form of a short follow-up of the research a while after the main research work was completed. It can have the form of continuous study of the same group of people - a longitudinal study. It can have the form of replicating a study or of drilling-down and performing an extra phase of study to get a better understanding of what the initial stage of the study suggested. A specific example of a drill-down, which is of special interest to me, could be the performance of a significant quantitative research to check ideas generated by an initial qualitative study. FIRE can complement the funds of a grant specifically in order to perform such time-extensions.

3. Limited geographical span

Trying to study the population of a large country means working with participants and research collaborators over a large area. Travel is costly. Communication technology is expensive: line-phones, cell-phones, satellite-phones, together with their respective audio and video-conference facilities, etc. This can cause the research to be limited in geographical scope, and therefore limited in applicability. FIRE can provide funding for travel, equipment and access to equipment.

4. Limited use of recordings

The relative advantages of different recording techniques in an interview is discussed in a previous entry. FIRE can provide funding, equipment, access to equipment, transliteration, automatic analysis facilities and services, etc.

5. Suboptimal experiment environment

Some research should be done in a lab settings, where the environment is as much as possible in the control of the researcher. Any aspect of the environment - any variable - that is outside the control of the researcher, may damage the validity of the research, and the applicability of its results. To conduct a serious research, one sometimes needs a well equipped lab, capable of hosting many participants over a period of time. FIRE can maintain such labs for the benefit of researchers, and provide funding for the use of other labs.

6. Other trivial limitations

Laptops, off the shelf software, custom developed software, photocopying, graphic design, research assistants, training, public relations (to ensure high rate of participation), legal fees (dealing with ethics and liabilities), database access fees, translation costs, etc. FIRE can provide assistance in addressing these and many more, to make researchers' life easier.

A general note about FIRE's role in the context of research finances: FIRE does not perform research, and doesn't fund complete research. If it did, there would have been a very limited amount of research FIRE would be involved in - limited by the amounts of money FIRE would be able to command.

FIRE is a research infrastructure provider, concerned with facilitating actionable research. This means that support and funding provided are meant to encourage researchers and would-be researchers to perform applicable research. This may sometimes mean providing enough support to make it possible for an already funded research to be upgraded to the next level of potential usability. This way, FIRE aims at having a positive effect on the largest number of educational research projects with the widest range of scientific agendas.

Boring Tasks in Research

Thomas Edison said that genius is one percent inspiration and ninety-nine percent perspiration. He used the same concept for invention. It seems that a similar formula is right for research, which is what may lead to invention, which in turn may lead to the reputation of genius. To make things even less glamorous, much of the perspiration doesn't relate directly to the area of expertise in the research. A well-known example is Edison's (again) experiments with the incandescent light bulb. Different sources claim he performed between 6,000 to 10,000 experiments before he found a reasonable combination. Not fun.

Educational Research is no exception. There are many tasks that must be performed, taking up much perspiration and time, which do not have much to do with expertise in education. I am told it often happens that the prospect of having to do such tasks often deters Masters and Doctoral students from performing certain types of research. Society loses many opportunities to learn about Education. The good news is that some of these tasks may be outsourced: They can be done by someone else with no loss of integrity to the research. This is part of the mission of FIRE - to take as much of the overhead tasks off the shoulders of the researchers who wish to concentrate on their subject - education.

A few examples for such boring tasks that can be taken up by FIRE:

1. Validating questionnaires and other research tools

Designing a questionnaire is a professional task, and the researcher needs to make many decisions regarding what to ask, how, in what order, etc. But after the initial questionnaire design and before it is possible to use it with confidence, there is a need to validate the questionnaire: To make sure that the questionnaire indeed tests what it is meant to test, and that it doesn't test anything else, that it suits the intended audience, and much more. This requires considerable work, together with financial and human resources. To make it even more daunting, to be done well, this process may need to be repeated several times while changing the questionnaire until it can be considered valid. This is a potential deterrent for someone who just wanted to do research in education. FIRE can shoulder much of the validation work, though none of the design work.

2. Re-validating questionnaires to maintain their validity over time

Because the world is changing quickly, a questionnaire validated 20 years ago can’t be considered still valid. The group for which it was validated doesn't necessarily exist anymore. For example, if the questionnaire was validated perfectly for average-grade middle-income suburban Caucasian 6^th graders in 1990 (having started the perfect validation work in the 1980s), the group disappeared, because average-grade middle-income suburban Caucasian 6^th graders in 2010 watch different TV, roam the Internet being exposed to very different types of information and content than their 1990 ancestors; Their parents have very different work patterns and patterns of relating to the children; the climate at school is different, etc. The questionnaire needs to be re-validated for the new group whose description appears to be the same as the original one, but the Best-Use-By date is significantly different. The revalidating process may sometimes require changes in the questionnaire, for example, to update the language used according to the transient fashions of spoken language. FIRE can perform such re-validation, therefore maintaining the usefulness of the questionnaire.

3. Replicating new research to make sure it is… well… replicable

In strict scientific communities, a single research supporting or refuting a claim is not quite enough. The scientific community feels much better if the same research or a very similar one can be performed, which gives identical - or at least similar - results. This doesn't seem to happen very often in educational research. Maybe it's an indication that the educational research community doesn't take itself very seriously as a science. Whatever the reason, replicating research is not considered an interesting enough topic to get the funding and commitment from researchers. FIRE can encourage such replicating research by providing part of the funding.

4. Replicating old research to check if it is still valid

After a physics theory was confirmed by research to the satisfaction of the scientific community, the laws of physics tend to stay quite stable. There is no need to repeat the research again. In education this is not the case. Society changes constantly, and with it the attitudes, behavior, expectations and even abilities of individuals. For example, in 1939, Lewin, Lippitt & White published a study about “Patterns of aggressive behavior in experimentally created social climates.” This study is still affecting the thinking about teacher leadership styles. The study is over 70 years old. Shouldn't we suspect that our aggressive behavior patterns and responses to aggression have changed a bit since before WWII, with added television, ubiquitous news/drama, multiculturalism etc? We need to explicitly revalidate the results of the study by performing it again - in whole or in part. FIRE can encourage such replicating research by providing part of the funding.

5. Performing follow-up research

It is often useful to revisit an experiment or other research after a while - even years - and perform an extra step. For example, when checking the effect of a new teaching methods, the basic research may check students knowledge acquisition right after a lesson. It is interesting to check how much of that knowledge is retained after a certain period of time - say a year. So one would expect the researcher to perform a follow up step a year after the original research... But the original research is completed, the paper was already published, and the student who performed the research has graduated and is gone. There is no glory in performing the remaining step, and probably no funding. Too often, the follow up just won't happen. FIRE can encourage such follow up research by providing some of the funding or by other means.

6. Obtaining existing data from known sources

Much research can be done based on existing historical and background data, such as student grades, student socioeconomic background etc. Obtaining this data is not always easy. Problems range from legal issues, to privacy concerns, to garden variety bureaucracy. Fighting city hall is an unwelcome prospect to a would-be researcher - they didn't want to have a fight, just to do research in education. FIRE can provide "data extraction" services, with a complete umbrella of human relations, legal abilities, trust relationship and technical solutions to privacy issues (anonymization), and moral and political clout to encourage cooperation from those in control of the databases.

7. Obtaining existing data from unknown sources

Often, even if the data is assumed to exist, it is difficult to know where reliable data can be found. For example, looking at populations of immigrants within a larger established population may prove to be tricky. Which government or city office has the info? Does the info account for illegal immigrants? Does it account for the now-established children of immigrants? Does it account for less recent immigrants? Obtaining this data can deteriorate to the field of private investigation. Not part of academic education concerns. This, too can deter a researcher from tackling a research question that might otherwise be very interesting and useful. FIRE can provide data-location and evaluation services.

8. Translation. E.g., from Broken English to English

Sometimes sources for the research are written in a language not easily accessible to the researchers. Other times the desired language for publication is not easily accessible. FIRE can provide translation and proof-reading services.

... You get the picture. FIRE's Roadblock Clearing Catalog will contain many more ways to improve the life and work of the researchers.

The Book

This entry is written as an introduction to the book being edited out of this blog's entries in 2009:

Less than a year ago, I set out to systematically study the issues surrounding mass education. I felt the current state of affairs is very wrong, but I didn't know many details. Not knowing enough about the problem, I of course couldn't know much about possible solutions. It took me a few months of study to understand the problem a bit better. Then it took a few more months to shape my opinion about what kind of education system would work. Then it took a few more months to get a sense as to what might be a useful first step to get from where we are now to the ultimate goal - a viable education system for the 21^st and 22^nd centuries. And here we are.

This is a rather lofty goal. Who am I to design an education system, or even to think about it? It would take a much better qualified person to do it. To be really worthy of planning an education system, it would take a lot of knowledge and experience:

• Knowledge and experience as a developmental psychologist
• Experience in teaching different ages
• Preschool
• Primary school
• Secondary school
• High school
• Experience as a school principal
• Experience in various related state-government roles
• Experience in various related local-government roles
• Knowledge and experience in regulation
• Academic depth in education, and specifically, public education
• Experience and academic depth in alternative education
• Academic depth in public education policy
• Know humanity’s accumulated knowledge and experience in the field of education
• Through time
• Past 200 years
• Throughout history
• What about Prehistory?
• Across the world
• What happened in each country?
• Why it worked?
• Why it didn't work?
• Be aware of all research ever done, with a critical eye to discern the strengths and weaknesses of each such research
• Education
• Relevant fields of psychology
• Relevant fields of public policy

It's a tall order. No human being matches all of the above requirements, and I certainly don't match any of them. Instead of total expertise, maybe something can be done with common sense, willingness to learn, and cooperation with experts.

This book intentionally sticks to everyday layperson’s language. It avoids specific educational/psychological/political jargon that would make it less accessible and less useful to the public in general. It is not a scientific book written for scientists. This is one reason I allow myself to use references to Internet locations such as wikipedia, which while not acceptable as a source for academic papers, is surprisingly accurate - more than enough for a layman's use.

Another aspect of this book is that it is written mostly from the point of view of an outsider to the education system and to the adjacent political systems. Being an outsider doesn't mean I can be disconnected and talk as a pure theoretician - that would not be likely to generate a book that has much to do with reality. To have relevant ideas, I have to be involved - get in the field and see how things work. Of course it is never “enough.” But one needs to find a balance: To see and know about reality, but not get so involved with that reality that one becomes an insider, committed to certain world views and agendas. Becoming an insider, I would risk becoming used to think as a status quo teacher, academic, researcher, government etc. On the other hand, there is a chance an outsider would see everybody’s thinking with fresh eyes. .An outsider is more likely to be understood by laypeople. No jargon. No reliance on specialized previous knowledge... No promises, though.

A few thoughts about getting involved: How does one get involved enough to gain understanding, without losing too much objectivity?

• Be in a class as a pupil - We all did that for many years
• Be in a class as a teacher, but don’t take on the stance of a teacher
• Work with academics in thinking and research, but don’t become a pure academic
• Learn from academic sources about psychology, motivation, education, public policy, etc. But don't take their world-views as your own
• Look closely from within at government decision-making, but for heaven's sake don't become a politician
• Work with NGOs concerned with education, while remembering most of them have different aims than you
• ... You get the picture

I tried to do much of that, and still doing it to improve my understanding, trying to keep from buying into any specific agenda.

One last thought before we go: The book was originally written as a blog, as part of a process of learning and exploring the issues around mass education. It documents much thinking and rethinking, so many thoughts and details do not lead directly to the main conclusions. Rather, they provide some context, depth and hopefully a better basis for understanding. The main ideas that constitute the backbone of this book are: Market economy for education, minimal but strict Regulation and Knowledge-Building through Research. The book should be read in light of these concepts.

Great Expectations

This entry is about what people can and should expect from themselves and from each other as an accepted standard of behavior, in the context of education. It is not about the technical motivation/psychological construct of expectancy.

Generally speaking, it appears that many people don't expect much of the education system. Many parents I speak to think of school as a glorified baby-sitter. Many pupils just try to survive the system for the years they have to. Many teachers just try to survive the day. This is so wrong that it's weired. There is no reason anybody should live as if they are constantly enduring some punishment. Also, this low level of expectations perpetuates the current situation which lives down to the low expectations. Higher expectations could help drive better results. Here are a few things we should expect from ourselves and others:

Parents should expect the education system to make excellent education available for their kids. They should also expect a wide variety of available education that will match the parents’ wishes and world-view, children’s wishes and children’s tendencies. Parents should expect their children to learn. The parents may allow the children to choose what to learn, but then the kids should learn.

Children should expect the system to “see” them as individuals: To notice the kids' strengths so they can build on them, and to help the kids overcome weaknesses and temporary difficulties. Children should also expect the system to teach them how to learn, to teach them how to deal with information. In general they should expect the system to help them pursue the general skills and habits that will make their lives better as adults in the 21^st and 22^nd centuries. The key words here are “help” and “pursue”. The children should not expect anybody to do the hard work of learning for them. Everybody must expect the students themselves to take the ultimate responsibility for learning facts, skills and habits.

The teacher and school should expect the parents to take responsibility for children’s learning in the parents' dealing with the kids. If a pupil doesn't learn, or doesn't learn well enough, or is experiencing any difficulty not immediately addressed by the system, blaming the system (or anyone else) doesn't work. The closest adult who knows the child and is in a position to notice there is a need for action - is the parent. So if there is a need, the parent needs to initiate action to fulfil that need. The school should expect the parents to take responsibility for children’s learning with the parents' dealing with the education system, too. When the parents identify an unmet need or have any criticism, it is their responsibility for expressing that criticism and suggesting concrete ways to improve. There are many wrong ways to do anything, and much fewer right ways. Therefore it is much easier to spot something that is done wrong, than to find a right way to do it. Teachers, schools and the education system in general have a complex task, and it’s very natural to perform it imperfectly. The rightfully critical parents should be expected is to bundle their criticism with an intensive effort to find a better way of doing thing. If these parents don’t manage to find a better way, at least they will have a more humble and constructive attitude in their criticism.

Everybody should expect everybody to answer every single question, to adapt, to be open to suggestion and to learn. For example, if a parent questions the way material is taught, they should get an honest answer. The answer doesn't have to convince the asker that the right thing is being done, but it needs to convince the asker that what’s being done is being done mindfully. Ignoring the question doesn't work, and note that "Your request is being processed" or "it was referred to a committee" count as ignoring. "This is the way we have always done things" doesn't work: The system must consider the changing environment and expectations, and adapt if necessary. Pulling rank - “We are the pedagogic professionals” works only if immediately followed by an explanation that can be understood by the parents. If the representatives of the system can’t make the parents understand, they are not such great pedagogues, are they? (This brings to mind a quote from Albert Einstein: "If you can't explain it to a six year old, you don't understand it yourself."

Pedagogic school staff in general should expect researchers to provide useful proven theories to drive improvement in teaching methods. Similarly, parents and students should expect researchers to provide such proven theories to drive improvement in learning methods and in ways to make the most of the students' abilities. Society should expect researchers to be a concrete (not ivory) part of a system that creates school graduates able to become contributors to society.

There are many more specific expectations that it would be healthy for us to develop. I won't try to capture them all.

Bottom line 1: As in everything else in life, each one of us who are either part of the education system or a customer of the system, should expect himself or herself to improve anything they see as needing improvement. "It's beyond my control" is just copping out.

Bottom line 2: As in everything else in life, everybody active in the education system, working in it, influencing it or being influenced by it, should expect this activity to be constructive and satisfying. It should be fun.

Data Collection - Focus Groups

Focus groups have all the advantages of a brainstorming session. A focus group may explicitly consider an issue such as the advantages and disadvantages of a proposed product design. Such a group consists of several minds working together to look at the issue, see what's understood about it and what is missing, find suspected patterns, get a sense of what it means and feels like, and in general - generate ideas. When the group member currently speaking runs out of ideas, another member may have something to say. Each member's speech or behavior may trigger ideas from other members. The group's ability to generate visibility into the issue and ideas about the issue is substantially more than the sum of the individual members' abilities (6 >> 1+1+1+1+1+1).

A focus group may implicitly react to an issue without the knowledge of the members. For example, the design of the chairs the group members are sitting on. The ruse for the focus group may be anything - like the product design mentioned above - but the moderator can stir the conversation towards how comfortable or uncomfortable the chairs are. This kind of focus group has significant elements of an observation-in-the-lab experiment, discussed later. Still, the same brainstorming-like advantages can be realized in the implicit focus group as in the explicit focus group.

As long as the goal of the current phase in the research is to generate ideas and hypotheses, to bring up creative directions of thinking, to identify problems, to prepare for a more accurate research phase - focus groups are an excellent option. But focus groups are normally rather small (around 10 people) and the members are selected according to specific considerations and not randomly. This means that by design a focus group does not correctly represent a large general population. We can’t generalize or extrapolate safely from what happens in a focus group. So if we think of the focus group as an information source capable of determining a decision or an action outside the realm of research - then there is trouble. Focus groups provide information upon which we can base a more focused research, not an action. For example, a quick Internet search brings up texts like "The focus group said there would be a market for ..." Such a statement indicates that someone is using the focus group as an clear indication about the outside world. A less worrying statement would be "The focus group indicated there might be a market for ...", which looks like a direction for a more focused market research.

Like in an interview, the focus group moderator must have good interpersonal skills, acting skills etc. Not very common traits. Even more than in an interview, the group moderator must be able to notice non-verbal communication, write it down, and respond mindfully. Furthermore, the focus group moderator needs to hear, see and respond to several people at the same time, whereas the superhero interviewer "only" needs to consider a single person at a time.

More than in an interview, the video recording should cover all the participants all the time, not only when they speak, to capture non-verbal responses to what the participant currently speaking says or does. Also, there shouldn't be a video-man moving the camera from one participant to the next, since this attracts attention and definitely influences people’s behavior. At best, there should be several stationary video cameras capturing all the participants all the time. Not very easy to do for every focus group in every research.

These demands regarding the moderator and the recording would be difficult to meet, but luckily, they are not quite as critical as for interviews. We define the purpose of the focus group as generating ideas that are the input for the next phase of a research. We don't use the focus group as the ultimate output of the research. Any imperfect results of the focus group phase can be caught in subsequent phases. The worst that can happen, if the moderator is not a superhero, is that fewer ideas will be generated or that the ideas will not be quite as well developed as they could have been. This may cause subsequent phases of the research, designed to confirm or reject the ideas, to be less conclusive. Imperfect interview data, used as the basis for the whole research results, may cause the research as a whole to generate false "knowledge." Imperfect focus-group data, used as a trigger for a more specific research phase can at most mean the research as a whole doesn't generate useful knowledge, which isn't fun, but not nearly as bad as generating fallacies.

Bottom line: A great tool for poking, prodding and generating creative ideas. As in every tool of every trade - use only for the advertised purpose.