Within minutes this week, two articles crossed my Twitter feed, both telling me how difficult it is to study some very important things.

One was on diet (Why Diet Research Is So Spectacularly Thin, by David S. Ludwig and Steven B. Heymsfield) and the other was on teaching writing (Scientific Evidence on How to Teach Writing Is Slim, by Jill Barshay).

The similarities beyond the headlines ("Thin"/"Slim") are striking. Both articles focus on the lack of high-quality research in their respective areas.

Conducting research on the effectiveness of diets is apparently quite difficult. While we may think that theres an easy metric against which were measuring (weight loss), the confounding variables make it very difficult to attribute any single outcome to a change. As the authors say, High quality trials are hard to do because diets, and the behavior of humans who consume them, are so complicated.

Diet interventions that may work in the short term may do long-term harm. Contestants on the reality show The Biggest Loser lost hundreds of pounds in a matter of months, but many of them quickly gained the weight back, sometimes surpassing their previous levels.

The extreme amounts of exercise and highly restricted diets are not sustainable. Theyre probably not healthy, either. Conflating weight loss with increasing health is probably a category-error mistake to begin with.

The authors close with a call for a Manhattan Project to find definitive answers to epidemics of diet-related disease. They want the research to have the same quality and rigor as pharmaceutical research that is meant to treat disease, rather than prevent it, as good diet can.

Im not in medicine, but I am a researcher. I wonder about that last bit, but lets table it while we look at the article on the research on teaching writing.

Jill Barshay quotes Robert Slavin of the Center for Research and Reform of Education at the Johns Hopkins School of Education, saying, Theres remarkably very little high-quality evidence of what works in writing.

The research problems in measuring writing are similar to dieting. It is difficult to find a true control group. And unlike diets, where we at least have weight loss (as problematic as that may be as our criteria), evaluating writing is inherently subjective.

Tested methodologies for writing show mixed and/or inconclusive results. What works in one group may not in another.

The commonality that Slavin did find is that Motivation seems to be the key: If students love to write, because their peers as well as their teachers are eager to see what they have to say, then they will write with energy and pleasure.

The research shows that the atmosphere in which students are learning makes a difference. What they are doing and who they are doing it for goes a long way to helping students write better because theyre more engaged to write more.

As to the lack of high-quality research, Im wondering if this is truly the problem we should be tackling or rather if we should expand our notion of what high-quality research looks like in these sorts of complicated human endeavors.

Isnt it possible, even likely, that in realms where human variability is at play, we are unlikely to find a single common approach that works best for all, or even most? As anyone who has tried diet and/or exercise has experienced, the chief problem is not necessarily whether or not the diet works -- the principle of taking in fewer calories than your body burns is pretty rock solid -- but whether or not the person can maintain the program itself.

The limiting factor on the success of a diet is not the quality of the diet, but the attitudes and experiences of the person.

The same is true, in my view, of writing. The key to improving as a writer is persistence. Good writers simply keep writing, and anything that keeps one writing is good. Trying to design experiments around these complicated things that meet these "high-quality" standards often involves moving the participants further and further away from the genuine, organic behaviors that attach to these activities in the real world. The diet or writing method that seems to work in the controlled lab experiment may not translate to the wider world. This is the exact problem with the highly prescriptive practice surrounding the use of the five-paragraph essay. Training students to pass the assessment that has become privileged has made them less capable as writers in general, while killing their spirits to boot.

Now that my own approach to teaching writing is out in the world, as embodied in Why They Cant Write and The Writers Practice, I am confronted with questions about how I know if my approach works.

I mean, I know it works. Ive refined it over years of working with students through a continuous process of qualitative research. Because it is not generalizable, qualitative research is not considered high quality, but this does not mean it is inherently low quality. When were looking at these complicated things where solutions are unlikely to be wholly generalizable, it is, in fact, invaluable.

One of the ways I measure the effectiveness of my approach is to ask students whether or not they think theyre learning. I find this to be meaningful data.

Another method I use is to ask students how they would approach an unfamiliar writing task. Here I am assessing the development of the writing practices, the skills, attitudes, knowledge and habits of mind of writers. If they can articulate an approach to a new writing problem, I know that eventually, through practice, the written artifact itself will become better and better.

I want to know how students feel about their writing abilities, whether or not they perceive an increase in their writing power. If I were a nutritionist, I would also want to know how my patients feel when on my program of diet and exercise. If they feel like crap and the experience is miserable, how could I ever expect them to persist?

A generalizable, quantifiable measurement simply doesnt apply here. It is a mismatch between desired information and methodology. The problem were studying is too complex, and what happens when it comes to writing and developing as a writer is a little different inside everyone.

I suspect this is why the available research finds that the writing atmosphere is important seem to be the most promising. Inside a good atmosphere, different students can travel different paths toward similar (yet still different in important ways) destinations.

As to the evidence I look for to see if The Writers Practice is working as I hoped, Im feeling pretty good about this.

Read more:
When "High-Quality" Evidence Maybe Shouldn't Be the Goal | Just Visiting - Inside Higher Ed