Knowledgebase Access Methods

It is not enough to collect a lot of data and organize it nicely. We need to be able to look at the data in a useful way. Security issues issues were discussed earlier, so now we are free to concentrate on the happier aspects of a database - useful access methods.

The database must be easily searchable. A good starting point of comparison for organized knowledge is an Internet search engine such as Google, but it's not enough for complex and statistical searches. We need the database to be easily computable. An initial anchor for comparison to computable knowledge is Wolfram Alpha. The Semantic Web is not there yet, and the tools we have so far are not enough. Internet search often fails to find and identify meaningful information. Computation inquiries often fail to be computed themselves. But we can dream, and this type of dreams does tend to come true soon enough. In the mean time, we can get ready.

Visualization tools for query results must be available. From simple interactive tools such as TinkerPlots, to complex tools such as Matlab. Since the former is specializes in schools, and the latter is a specialized tool for scientists and engineers, there is a lot of room for more tools suitable for the uninitiated public. Graphs, tables, statistical analysis - all interactive and intuitive, oh my. Gapminder has some interesting data nicely visualized with Google Motion Chart. It makes one optimistic about the near future visualization capabilities.

Another requirement of the knowlwdgebase is that it will be amendable by the public. Anyone with an insight will be able to add it - wiki-style. This means that notes, comments and tags can be added to any item in the database, and immediately become a searchable and computable part of the knowledge base. Naturally, an anonymous tagging will have to be marked as less dependable - part of the Knowledgebase Data. There can also be room for opinions to be entered into the knowledgebase. Some practices may be designated as suggested "Best Practices". As long as these are clearly marked as opinions, they can sit there without interfering with the consensus.

In addition to comments and tags, any ad-hoc analysis made by a knowledgebase user (the public) may be fed back into the knowledgebase. For example, if a lay researcher (any member of the public) discovers a meaningful rise in the test scores of a certain age group in a certain school district, this fact may be recorded in the knowledgebase, so it can become a building block for subsequent queries by others - like looking into the possibility that the school district superintendent might be doing something right.

With a powerful tool of any kind, comes the danger of abuse or misuse. In the case of a tool for managing information, this danger translates into disinformation, misinformation and misinterpretation. Disinformation and misinformation are quite successfully minimized in wikipedia, so there is room for optimism. Misinterpretation is a much tougher problem, since it leaves no mark on the knowledgebase, and therefore can't be corrected by those who know better. In a computable knowledgebase, this problem is even more severe than in a merely searchable knowledgebase, such as wikipedia. The only thing the knowledgebase can do to minimize misinterpretation of information , is to discourage simplistic analysis.

The most complete way to discourage simplistic analysis is to educate the masses to such a great extent, that everybody knows very well how to perform valid analysis. Forget it. Not even when the new and improved education system is in place. The human drive towards misunderstanding and simplistic views is too great, and this isn't going to change any time soon.

A less complete way is to nurture a healthy sense of scientific doubt: To encourage people to look at any result with a critical eye, bearing in mind the fallibility of data and of opinions that can be derived from it. This should be easier, since the late 20^th century and early 21^st century did a good job shaking our belief in any authority. Some see this as part of post-modernism. There is no need - or easy way - to educate the masses in general towards this tendency. The access venues to the knowledgebase can keep reminding the user of the limitation of the analysis. The access venues can also use some of their own background analysis to emphasize the doubt when appropriate: When much of the data at the basis of the analysis is tagged as not very dependable; when the analysis is based on a small amount of data; when there is extrapolation involved; when the system has detected similar analyses producing dissimilar results; when the system detects an iterative activity that may be an effort to fine-tune the analysis results according to a preconceived target, etc.

As usual, the problems and solutions described here are just a part of what reality is. It's the part that can be seen from the outside of the Research-Based Education project, and much more will become apparent the more we go into that project.

What was unthinkable a few decades ago - freely available statistics about huge piles of information - is now a mundane fact, with computers and the Internet being widely available. We need just a bit more than that in order to achieve real research-based education.

Knowledgebase Data

The Education Research Knowledgebase is the almost-physical embodiment research-based education system. It is one of the main benefits a school gets from “the system” in a semi-anarchic education system. The knowledgebase has two main attributes: Data organized and accumulated in a database, discussed here. And methods to access the data, discussed elsewhere.

The more detailed information exists in a database, the more information can be analyzed by the public, and the more informed the public can be. The Internet is a fine example of that. Or going to be some day. On the other hand, giving information to irresponsible people, is harmful to the system, to the schools and teachers, and ultimately to the very people who get that information. A trivial example is that of giving a violent parent the name and address of the teacher who failed his kid at a test. This problem is wider than the violent fringes: Giving raw information to the ignorant, gives the unscrupulous a good opportunity to shape that information in any way useful to them. See any hatemonger for examples. Worse yet, “the ignorant” is a pretty large group. It may even include people we meet in the mirror. For example, in the past few months we got a lot of raw information about Swine Flu. Bottom line: Should I worry about it? Is it dangerous for me and my loved ones? It is not all that clear to me. I am too ignorant.

The Education Research Knowledgebase should be widely accessible, in order to maximize its usefulness. The more people work with the knowledgebase, the more diverse ideas and thinking can go into it and come out of it. How wide? All qualified researchers and educators? All researchers and educators? All? Let’s go with All, as a working assumption, and we’ll see if we need to scale it back a bit. If we manage to stay with “All”, it will be a good step towards taking education seriously as everybody’s business.

In the current thinking about privacy, we must minimize the public’s access to the identity of individual students. On the other hand, the less details are available, the less information can be analyzed. A database open for everybody needs to be carefully anonymized.
Let’s consider a few types of details:

Individual student names must and can be kept out of reach.

Individual teacher names: In the current climate, they will have to be unreachable, though a culture can be created where the teachers don’t mind, and parents considering a school demand the information.

School-names: This item suffers a lot from the information-to-the-ignorant problem. One needs to have very keen critical thinking, coupled with considerable familiarity with statistics to avoid pitfalls in comparing between schools. For example, suppose School A has a 90% success rate in SAT, and school B has a 70% success rate. Most people will conclude (and nearly all people will feel) that School A must be better than school B. But maybe School A just pre-tests students, and prevents the less successful from even trying the SAT? Maybe School B takes those students dumped by School A, and manages to get an amazing success rate of 70%? Simplistic analysis is a danger, but instead of giving up and keeping information away from “All”, we can try to meet this danger head on, and remove it - See discussion of access methods. The school name is also a problematic piece of information because it is partially identifying: It can be used to identify individual students, especially in small schools. But this would take an effort of a data-savvy person, and such people are known to find all kinds of information, such as medical information, credit information, etc. even without a publicly accessible database. Let’s go ahead with “All” a few more steps and see how far we get.

Researcher details: Name, current activities, past activities, etc. Freely accessible. If a researcher wants to remain anonymous, it may be possible, but then the research has to be marked as suspicious…

Individual parameters: Date of Birth; Ethnicity; Address of residence; Medical history; Family circumstances, etc. These suffer from being partially identifying, and should be partially anonymized one way or another. For example, the date of birth can be rounded to the Month of Birth, and the Address can be replaced with a few measurements along lines such as affluence, altitude, air quality, noise level, etc. Alternatively, the types of analysis can be limited so that such partially identifying information will not be visible for groups smaller than 100 people.

Test scores: Individual (anonymized); Association with school; Association with teacher (anonymized for now); Association with individual parameters etc. All can be freely accessible.

Individual choices, such as courses taken, associated with dates and correlated with any other information in the database. These are also partially identifying and can be partially anonymized, and then be made accessible.

Research dependability quotient: Calculated from details such as the number of times the research was repeated by different teams, the number of pupils participating in the research, scores of formal reviews, dependability of researchers, etc. Freely accessible.

Formal research details: Ah, there's the rub. Formalized, quantifiable details are essential. But the more formalized the data is, the more it loses the “juice” - those details that were not expected by, the designers of the research or the designers of the database. And this is where most of the meaning of the research may be. I have been dancing around this issue for a while now, and there are no answers yet. Generally, one can guess that there will be semantic markings on such details, creating a semantic web. Some people are working on such projects.

Raw research details: Until we got the Formal Research Details engine right, and I don’t expect it in the 21st century, we will need to keep the raw details available. This again brings up real privacy issues, discussed earlier in the blog. It may be that here access will be restricted and recorded: Restriction can be to generally qualified personnel, further restricted to preapproved people, or discussed on a per-request basis, to minimize the possibility of recognition of specific students by anyone accessing the raw data. Also, there could be different levels of rawness of the data: For example, a video footage with blurred-out faces may be more freely accessible than the same video in the raw.

These are examples of items in the database. The full database design would take more time than that required to write a blog entry.

Research in Context

So, we have some excellent research. We checked it twice, and we know something. Maybe we know that a certain teaching method, works better than another for a certain age group. Or maybe that a certain teacher-training method produces more teachers who produce more students with better retention of … something. Now what?

The fact that a researcher found some truth, doesn’t easily translate to a real world result. We need a bidirectional effect here. One direction is field-to-research: The effect of the lessons from the field on the research. Formal research should be made about what is interesting and necessary for the teachers in the trenches, and not about purely theoretical issues. The other direction is research-to-field: Beyond the need to perform research that is applicable in its nature, there is a need to create the framework that will encourage the application of that research - that will make the research actionable. Without these two bridges, we remain with an unbridged gap between research and teaching. For both bridge-directions to be open, the teachers should be aware of the research activity and potential, and the researchers should be aware of the teaching activity and considerations. Much of it is about communication, and we should create the mechanisms and culture to support this communication. Much of it is about discipline, and again there can be mechanisms and culture to support such a discipline.

Some successful commercial companies have in place a set of such mechanisms, policies and cultural dynamics to support such cooperation between research and the field. The field (Marketing, Sales, Engineering, …) informs the researchers (Economics, Psychology, Physics, …) what is necessary. The researchers produce results and inform the field what can be done. In the best organizations, the communication between researchers and field-people is so good, that when research turns out an unexpected opportunity for the field, the field people become aware of it and can act on it. This strong bridge is not easy to achieve and maintain, since the researchers and the field-people are two separate groups, with separate background, jargon and even values. It is quite a feat to bring them together to a common ground that will enable cooperation - both in terms of leadership and in terms of administration.

In the world of education, the first tier of field people includes teachers, principals, school administrators, pupils, parents, ... Not necessarily in that order. There is also a second tier, but I don’t want to start talking about religion, philosophy and politics at the moment. The first tier of researchers includes psychologists, sociologists, anthropologists, organizational experts, economists, designers, engineers, … A very wide range of skills can be used here.

Enough theory. Now for some more practical thinking. And I will start with technical mechanisms and policies, since they are more straightforward than a discussion of culture and leadership.

Complete financing of research: This issue was visited in a previous blog entry. Research funding needs to take into account the need for additional investment to make the research useful. A few activities that need to be financed, and need to be budgeted in advance:

1a. Plan

• Both the research and its scheduled follow-up

• Initiation led by the field

• Details of research led by researchers

1b. Do the research - experiments and all

• Led by researchers

1c. Review results

• Led by the field: Look for opportunities in the field that may have been unearthed by the current research, and suggest action plans

• Led by the field: Look for potential spin-off research that may be suggested by the current research

• Led by the field: Decide whether to perform the scheduled follow-up

1d. Assimilate into the Education Research Knowledgebase

• Not a trivial task. Will be discussed in a future post

2a. Revisit the plan for scheduled follow-up research

• The decision to perform the follow-up has already been made. This stage may only make some modifications to the research plan

• Led by researchers

2b. Do the scheduled follow-up research

2c. Review results

2d. Assimilate into the Education Research Knowledgebase

Notes:

• All activities should be multidisciplinary: Have involvement of different types of researchers, as well as field people

• The “led by ...” notes are not to be interpreted as “forcefully controlled by ...”. In general, the field people are the customers, and they should lead the “what to do” part. The researchers are the experts in their specific fields, and should lead the “how to do” part

• Then there can be any number of iterations of: Re-plan --> Follow-Up Research --> Review --> Assimilate. Since these future iterations may not be scheduled in advance, budgeting doesn’t need to be done in advance. The first 4 steps - the initial research - must always be budgeted and planned for, even if the experiment was a complete failure. The next 4 steps - the scheduled follow-up - must be budgeted and planned for, though in case of a total failure of the initial research, the follow-up may be dropped

• The research itself may need to be very formal. All other activities are more open for unexpected activities and consequences – innovation

Wrapping up with the theoretical thinking above: The field-to-research bridge is taken care of by the fact that field people participate in all research-related activity, and are the leaders of the initiation and review (or evaluation) of the results. The research-to-field bridge direction is hiding behind the “action plan” part of the results review. This heavy challenge of causing change in behavior as a result of the research will be discussed in a future post, if I manage to wrap my mind around it.

Price of Freedom

Freedom is good. The freedom to create schools with very different goals, and very different curricula is one of the necessary foundations for an adaptable education system. But freedom has its price.

One general problem with freedom is that it clearly puts the weight of responsibility on the free person. In this case, freedom to choose means that we - the individual parents - have the responsibility for our children’s education. We can’t sooth ourselves by saying that the powers that be know what’s right. We can’t blame the government for bad choices. This is the type of price we already pay in higher education, where we choose much of what we do. In many countries there is also such freedom of choice with medicine, associated with the same side-effect of shouldering the responsibility. Barry Schwartz wrote a whole book about the problem - The Paradox of Choice: Why More Is Less and gave a great TED talk about the matter.

What can we do about it? First of all, control expectations: Even with 100 choices for a school, the best one will not be perfect for the discerning parent. It will “only” be much better than the no-choice school assigned to a kid because of the address where they live. Secondly, learn to forgive ourselves for errors: We cannot correctly compute all the differences between those 100 schools (or 10,000 if we are willing to relocate), especially when we don’t know what the world will look like in 10-15 years, after the kids graduate. We could also give up on the project, and settle for the 18^th-19^th century mode of education. This is a real option and not a mockery, since it’s the familiar way of doing things, and that has its comforting effect. The effect is so strong, that I expect that creating the 21^st-22^nd century mode of education will be an uphill struggle. Anyway, keeping things as they are is not the scope of this blog, so let’s move along.

But there is also a technical problem that needs to be addressed. What happens if a kids enters a technology oriented school, and after 5 years decides that an art oriented school is a better choice? Or - just to emphasize that there are more than just these two choices – if the kid wants to move from a social-responsibility oriented school to a science-oriented school? The social-responsibility curriculum possibly emphasizes sociology, psychology, politics, etc., but probably very little science-related material. How would the kid pick up 5 years of calculus and theoretical physics? This problem must be addressed effectively to allow freedom of choice without lifelong commitment at the age of 6 (or even 3).

One answer is to allow for catch-up time. On the extreme end, we can allow 5 full years to catch up. This will enable the kid to catch up completely, but a “penalty” of 5 years is more than what most pupils will consider. On the other extreme, the kid can just move from 6th grade of school A to 6th grade of school B, but the student will just be lost in a world of unfamiliar concepts and missing skills. Even if the school tries to help the new student by attaching tutors, the gap will be too large to work well. A possible middle-ground is one year of intensive catching-up with the main issues taught in the new school. We have some good experience with children moving from a relaxed democratic school to a stricter school, and picking up missing material and skills during the summer vacation. We also have good experience with preparatory courses to university, enabling students who took a relaxed approach to school, to achieve university-entry capabilities. Some choice may be offered here, too, regarding the length and intensity of the preparatory phase - maybe allocating Two years instead of one year will enable deeper catching-up. More choice may be given to parents in employing special catching-up teachers - paid either by the parents or by the state.

This catching-up time is a straightforward solution, but it may encounter some opposition. One source of such opposition is the state: The state likes people to start being economically active (read: paying taxes) sooner than later. This opposition should be dealt with on the political level. Many states also like to conscript young people at 18 or so, but most of them also allow delays in the military service for extended studies, so the framework is already there. Another possible source of opposition is parents who are unhappy about delaying their child’s entry into real life. But this concern will have to be weighed in the light of the importance of the change in the child’s interests: Another year at school, vs. continuing in the wrong direction for years. Something like “throwing good time after bad”. Also, if we do a good job, being at school will be considered as a pretty good way to use time.

Grand Plan vs. Specific School Ideas

Over the past 6 months, I have been writing down thoughts about education and schooling. Since there was no outline prepared before writing, the blog entries took their own direction. Or, so it seems - two distinct directions: One group of entries has to do with specific ideas of how to teach a kid, or a class, or how to run a school. The other group has to do with a grad plan for an education/schooling system, that should be useful for any type of school. There is a bit of a contradiction between discussing the Grand Plan and discussing Specific School Ideas, since one of the premises of the grand plan is that there are many sets of specific school ideas, which are equally valid. From the point of view of the grand plan, specific school ideas have very limited value. Still, they are my ideas, so like them.

I will now put some order into the previous entries by categorizing them as either Grand-Plan-related or Specific-School-Idea… Or somewhere in between.

1. April 11, 2009 - About K12 Education - Grand Plan introduction
2. April 14, 2009 - Bare Necessities: Goals - Grand Plan basics
3. April 15, 2009 - Scope: Education for whom? - Grand Plan scope
4. April 18, 2009 - Bare Necessities: Measurement - Grand Plan basics
5. April 19, 2009 - Bare Necessities?? Research - Grand Plan basics
6. April 20, 2009 - Bare Necessities - A summary - Grand Plan basics
7. April 20, 2009 - The State Of The Art of Education - 2009 - Both Grand Plan and Specific School Ideas - context
8. April 22, 2009 - The roots of the problem - Grand Plan context
9. April 23, 2009 - "It can't be helped" - Grand Plan context
10. April 25, 2009 - I don't have a plan - personal
11. April 26, 2009 - Some old ideas - Specific School Ideas - context
12. April 27, 2009 - Goals - Foundation Layer - Grand Plan foundation
13. April 28, 2009 - Measurement - In Practice - Grand Plan practicalities
14. April 30, 2009 - Finland - How to form a reform - Specific School Ideas - context
15. May 1, 2009 - Primum non nocere - First, do no harm - Grand Plan basics
16. May 5, 2009 - Learning and Meaning - Grand Plan basics
17. May 7, 2009 - Critical Mass of Learning - Specific School Ideas - basics
18. May 8, 2009 - Pedagogic Urgency - Grand Plan basics
19. May 9, 2009 - Teaching and Salesmanship - Grand Plan basics
20. May 10, 2009 - What we can take home from school - Grand Plan basics
21. May 14, 2009 - Hear, See, Do - Grand Plan basics
22. May 15, 2009 - Education and The State - Grand Plan basics
23. May 16, 2009 - Emotional Involvement in Learning - Grand Plan basics
24. May 17, 2009 - Explaining to a six-year-old - Specific School Ideas
25. May 18, 2009 - Customers - Grand Plan practicalities
26. May 21, 2009 - Long Term Results - Grand Plan practicalities
27. May 24, 2009 - Wild Ideas - Student Teachers - Specific School Ideas
28. May 25, 2009 - Experiments? NIMBY! - Grand Plan practicalities
29. May 26, 2009 - Tests and Announcements - Specific School Ideas
30. May 27, 2009 - Learning to Err - Specific School Ideas
31. May 28, 2009 - The Ultimate Knowledge Work - Grand Plan philosophizing
32. May 30, 2009 - The Main Course: Feedback - Grand Plan practicalities
33. May 31, 2009 - For Whom the Test Tolls - Specific School Ideas
34. June 1, 2009 - Wild Ideas - Anarchy - Grand Plan wilderness
35. June 2, 2009 - Mind the Gap - Grand Plan practicalities
36. June 6, 2009 - If We Build It, Will They Come? - Specific School Ideas
37. June 7, 2009 - Research Based - Grand Plan practicalities
38. June 9, 2009 - Teaching Teachers - Grand Plan details
39. June 10, 2009 - Paid Volunteers - Grand Plan practicalities
40. June 17, 2009 - Defining a system - Both Grand Plan and Specific School Ideas - practicalities
41. June 18, 2009 - Ask Around - Grand Plan practicalities
42. June 20, 2009 - Dumbing Down - Grand Plan context
43. June 24, 2009 - Let Kids be Kids - Specific School Ideas
44. July 2, 2009 - Administration Remystified - Grand Plan practicalities
45. July 3, 2009 - Communicate^3 - Grand Plan practicalities
46. July 4, 2009 - Wild Ideas - Distributed Establishment - Grand Plan practicalities
47. July 4, 2009 - Education Market Analysis - Grand Plan practicalities
48. July 5, 2009 - What... Is Your Mission? - Grand Plan mission
49. July 8, 2009 - Mission - The Easy Part - Grand Plan mission
50. July 8, 2009 - Mission - Complicating Factors - Specific School mission
51. July 9, 2009 - Regulation - Grand Plan practicalities
52. July 11, 2009 - KIPP, SIPP and HIPP - Both Grand Plan and Specific School Ideas
53. July 11, 2009 - Example: Technology Oriented Schooling - Grand Plan specific examples
54. July 14, 2009 - Interim Summary - Both Grand Plan and Specific School Ideas
55. July 15, 2009 - The Essence of Time - Grand Plan philosophizing
56. July 18, 2009 - Still No Plan - Both Grand Plan and Specific School Ideas practicalities
57. July 19, 2009 - Example: Art Oriented Schooling - Grand Plan specific examples
58. July 20, 2009 - Centuries 21 and 22 - Grand Plan context
59. July 21, 2009 - Australia - Specific School Ideas - context
60. July 23, 2009 - The Graduate - Grand Plan basics
61. July 24, 2009 - Education System Architecture - Grand Plan basics
62. July 25, 2009 - Information Exposure - Grand Plan context
63. July 27, 2009 - Data Crunching - Specific School Ideas - practicalities
64. July 29, 2009 - Limitations of Data - Specific School Ideas - practicalities
65. July 30, 2009 - Evolution vs. Revolution - Grand Plan basics
66. August 22, 2009 - Creativity - Specific School Ideas
67. September 20, 2009 - Too busy - Grand Plan practical philosophizing
68. September 23, 2009 - Unspecial Education - Grand Plan basics
69. September 23, 2009 - Quotes - Both Grand Plan and Specific School Ideas philosophizing
70. October 7, 2009 - Research and Student Privacy - Grand Plan practicalities
71. October 8, 2009 - Research - Technical Issues - Grand Plan practicalities

The classifications above - Grand Plan or Specific School Ideas - are open to debate. Often the grand ideas stray away from the consensus, and should be demoted to specific-school-ideas. Sometimes specific ideas are actually general enough to qualify as part of the grand plan. Still, the above list gives a good idea of the content of the posts.

Research - Technical Issues

Below are some preliminary thoughts, before diving into the ocean of details of research problems, solutions and opportunities.
The problems and perceived problems should be thought of in the context of the broad issue of ongoing research as the tool for ongoing improving and adaptation of education to the constantly changing society - Looking ahead.

At the heels of the thought of research, comes the thought of experimentation, and the natural negative response to experimentation has already been discussed - indeed not exhaustively, but one has to start somewhere.

Another issue is that of research interference with the optimal course of learning, when the needs of the research seem to dictate an action that would not have been done otherwise. The most trivial example is that of asking students to provide their opinion, which will be part of the input for analysis in the research. The time spent by the students answering questionnaires could have been used for other ways of learning. Note, though that in some education systems, this is already the case. Countries are concerned about their position in the international tests (PISA, etc), and they put a lot of effort into intermediate testing aimed at preparing the kids to the international test. Also, since the learning disabilities are a hot issue, in some education systems tests and assessments take considerable amount of time in the lower classes of primary schools.

Looking ahead again: The problem of non-study-related activities required of the students can be mitigated by being aware of it and trying to minimize the disruptions. We must make sure that any action required or requested of the students, is truly necessary. Part of that, is the use of non-intrusive techniques for feedback and assessment, such as analyzing students’ behavior as it is reflected on normally accumulated records. Such records may include:

• Assessments the students would have taken anyway, right after the event the research is focusing on

• Assessments the students would have taken anyway, a while after the event the research is focusing on - as a sort of follow-up / retention feedback

• Assessments the students would have taken anyway, in fields other than the field the research is focusing on

• Normally scheduled satisfaction assessments of students, parents, teachers, tutors, etc

• Records of students choices of next-course following the event the research is focusing on

• etc.

The usefulness of video recordings was mentioned before, together with its accompanying need for anonymization and Chinese walls. But there is also the technical and logistic issue of analyzing the recorded video - Hours of it? Thousands of hours? Ok, so we can have many researchers analyzing different parts of the video. Such analysis requires a clear methodology, to ensure that different people analyzing different stretches of video, will come up with compatible partial results, so these results can be aggregated into a coherent overall result. If we have a very strict methodology, we will be able to create such a coherent picture, and even experiments that are repeatable - that can be repeated by different research groups, with consistent results. This would make great science, but we are likely to limit the types of results we may get, and therefore miss some of the picture. For example, Counting the number of times a teacher asks a question and waits for an answer from the class is a strict and objective way to analyze a video, but if the researcher is limited to this measurement, we may miss other interesting patterns, such as the length of the median answer from the class, or even the more subtle indications of the median amount of interest the students display. So, even with a good methodology, we are going to find ourselves in an imperfect point between producing consistent results with limited field-of-vision on one end, and producing deeply meaningful results with no repeatability (and hence verifiability) on the other end. I am sure this has been discussed academically in social studies, but I suspect that discussion has been more relaxed than what is needed for a systematic research based education system.

There are also some easier questions regarding analysis of video materials: How to search for an event? How to correlate actions by the same person in different parts of the video? Or maybe different days? Or maybe different research projects? Or their test records? How to block-out the faces for anonymity, where facial expressions are not important? How to count spoken words? How to measure the tone of voice? How to measure stress? How to measure level of interest? How to identify idle time? …
Some of these - the more the better - can be facilitated by technology. My guess - as a technologist - is that they are easier than meets the eye.

Back to more difficult issues - maybe the crux of the education research problem in an education system that allows different ideas and methods: How to make research in one school relevant in another? If the school system is highly standardized, there is no problem. But in a distributed school-system, where different school may have different emphases and goals, this is not trivial. How do we correlate the results in a school that emphasizes academic achievement with the results in a school that emphasizes cooperation between students? One option is to have different research projects and methods, managed by the schools themselves. But this way we will lose the system’s ability to understand the schools, and find ourselves nearing educational anarchy, which is more than what most of us will be comfortable with. The “system”, the state, the regulator have to have some understanding of the schools.

One direction for thinking about conducting a homogeneous study of a heterogeneous population, is to do "qualitative research". This method can generate intuitive understanding in the individuals involved in the research, and it can give rise to some questions to be researched further. But I can't bring myself to thinking about it as producing answers. It is too much of a personal experience by the researchers, and therefore not repeatable, and doesn't lend itself to neither scrutiny nor follow-up. There is a need to come up with quantitative research methods for the "soft" field of education. Tricky.

Another direction would be to break up the research subjects into small components that are measurable and can be compared across different schools, even when the goals of the different schools may have a different nature, such as individual achievement vs. social responsibility. Such component-subjects may include "individual achievement in basic arithmetic learned by rote". This component-subject may be more important in one school, and less important in another. In some schools, it may even be undesirable. But the quality of the teaching/learning of that particular component-subject can be measured, analyzed and compared across many types of schools.

The above suggestions are incomplete and a bit naive. To get better answers, it is necessary to do more study, thinking and mostly - more research about research. Quantitatively.

Research and Student Privacy

Talking about research in school starts easy, but gets in trouble very quickly.

A large class of problems has to do with the privacy of the students.

This is a general problem, but it can be illuminated best by considering the research method of recording classes and analyzing the records later. This brings up the problem of legal issues of student privacy, which was already there as soon as we started conducting any assessment, but seems to be aggravated when recording - especially in video - of class activity. Parents must agree to video recording; video recording may be totally illegal regardless of parents’ attitude; people’s images may be their own property and can’t be used by others without explicit permission; video may be allowed while sound-recording illegal; etc. But the efficacy of analyzing video is so great, that it is worth our time to look into ways to overcome this problem. In the long run, the legal issues can be resolved by passing the right laws.

But beneath the legalities, there are real concerns of parents regarding the privacy and safety of children. There are many aspects of that concern: Parents are worried that kids will be caught copying in a test, which they could otherwise get away with. This may sound a bit comical, but talking to a parent - who happens to be a primary-school teacher - I heard a deep reason for that concern: The stated rules of society are not absolute. They are part of a dialog/conversation between society, subcultures within the society, and the individual. Breaking the rules occasionally, is part of that conversation, which allows the individual to map the real limits of behavior, and is part of building the personality. Introducing what can be used as full surveillance interferes with that healthy dynamic. The rules of society may be acceptable and reasonable, but imposing them with too much success may be unacceptable. This is a nice opening for a philosophical discussion which is way out of scope here, but the concern is there, and therefore it must be acknowledged and taken into consideration. Suggested guidelines for a solution will be outlined below.

Parents are worried about the classic issues of privacy, applied to children: The danger of ridicule (e.g., when a student is “caught” on camera picking her nose) is always real, and is amplified by the ease of uploading such material to YouTube and the like to remain there for all eternity. On a smaller scale, there is the danger of abuse of information, like exposing a secret whispered between friends. There is even the worry that the system will know the weaknesses of the child. The same guidelines below can be a starting point for dealing with these worries.

Finally, parents are afraid of pedophiles watching their kids. Whether or not this fear is well-based or not, it must be answered convincingly.

Some thoughts towards resolving the concerns over student privacy:

First, it is worthwhile to remember the current situation, to which we are used, and which we manage to live with: Students don’t have much privacy. They live in a dense society where their secrets are revealed, their weaknesses - real or imagined - exposed, and their smallest quirks ridiculed. They are watched by the system through testing, in-class observation, snitching, and teachers’ “eyes-in-their-back”. Finally, the pedophiles don’t wait for massive video-based research to catch on - they have their ways. All this is just a reality-check, and it is true that careless implementation of recording-based research can aggravate all of the above. But the last sentence leads to a better way - that of careful implementation of recording-based research.

One guideline for such careful research come from the worlds of information technology and medical research: Anonymization - The removal of all identifying details not needed for the analysis at hand. As much as possible, the identity of the students should be concealed. Starting from names, which can be withheld without any damage to the research. Also, the name of the school can be safely withheld, and the researchers need not know anything about the community where the recordings were made. Same with the identity of the teachers. In general, every detail that is not necessary for the research can be eliminated. A minimal effort can eliminate the soundtrack of a video when body-language will do, and vice versa. A higher effort can automatically block-out faces, specific people, identifying location-related signals, such as pictures hanging on walls, etc. In general, the amount of effort being put into that anonymization can determine the level of privacy. There are complicating factors, such as the need to cross-reference observed student behavior with recorded test results of the same student, while leaving out the student’s identifying details. These factors can be dealt with, but this is out of the scope of this already long post. I intentionally left out the option of sub-optimal research, where details relevant to the research are hidden from the researchers. This is not part of the plan.

Another guideline for careful research come from the world of service-providers dealing with customers’ information: Chinese Walls. This is more tricky than mere anonymization, since while anonymization requires mostly working automatic mechanisms, Chinese Walls require people’s willing cooperation. In case a research is even mildly relevant to issues of privacy, a “Chinese Wall” needs to be erected between the students participating in the research on one hand, and the researchers on the other hand. Complete separation. This means, for example, that none of the teachers can be exposed to the recorded materials, and of course no student can have such access. This will hurt the research since it will remove the chance of unexpected insights by people who know the participants. But unlike the simple case in anonymization, the more complicated case of Chinese Walls pushes us towards suboptimal solutions.
… At least until we relax a bit about privacy issues.