What’s A Good Question?

When considering a research angle, this is a really important question. Probably not a good one, but important nonetheless.

I see a lot of reductionist questions being asked in agroecological literature, like a recent article that came out comparing carbon sequestration and soil health metrics between conventional and ‘transitioning to organic’ dairy systems. They found minimal differences, and concluded that maybe the challenges of organic transition weren’t worth the cost and hassle.

There are plenty of other examples; I won’t detail them here (for now).

But one thing that does really interest me is in finding out the depth to which agroecologists, soil biogeochemists, and others studying ‘regenerative’, ‘organic’, ‘diversified’ (and so on) agriculture have plumbed the universe of Possible Statistical Methods. I ask because nearly every conversation I have about setting up a good experiment, testing some hypothesis, and so on, has at some point got to land on: “So what’s the control?”

So to get some good ideas from others, I start searching: “When is it not necessary to use a control in an experiment?” (Okay, to be honest, I used ‘experiment without a control’ but same drift). And to my delight, I actually found that there are many cases in which a control isn’t necessary. I’m interested in exploring how these might be applied to agroecological research, to get beyond reductionist, single-effect thinking and simplified conclusions:D

  1. Prospective cohort studies are one, in which a group of individuals are tracked longitudinally.
  2. Another is observational studies wherein the study does not have an experimental manipulation, but rather participants are followed in their naturalistic setting, or alternatively, all participants undergo the same experimental protocol. Pilot studies are commonly set up this way. It’s a good way to collect preliminary data without too much expense. Typically, in these kinds of studies analyses look at repeated measures, within-subject effects (i.e., before vs after).
  3. If there are substantially more parameters than you can investigate in a reasonable size of experimental array, it is better to use the available resources in this array, as opposed to taking some of it out for a control group.

I also found this book chapter by Jack McKillip that I’d like to get ahold of to learn more about experiments without using controls. This is a new area of exploration for me but I think it’s an important one: I remember from my undergraduate research long discussions and a lot of papers on the weaknesses of hypothesis testing and p-value reliance, which led us to use information theoretic approaches, e.g. Akaike Information Criteria (AIC), to analyze our data — which, like many agroecological studies, were numerous, complex, and it wasn’t always straightforward to determine which we should be looking at as most important. (Which brings up the conversation of spurious significance from over-testing, but that’s for another night).

There’s another whole discussion of this here in reference to quasi-experimentation, chiefly referencing the development of experiments that allow for descriptive causal inference but lack a control as in a traditional or more conventional experimental set-up. An example is in the history of PKU treatment for preventing phenylketonuria in infants, in which 47 infants with PKU received treatment; 44 did not develop symptoms, and of the 3 that eventually did, 2 were minor enough to be missed during the screening tests.

Analogous scenarios could be considered for agroecological research; the primary need would be a large, and possibly diverse, enough number of ‘test subjects’ to which one might apply a treatment – say, compost application, or a particular grazing methodology.

Overall it seems to me that something like the prospective cohort studies approach would be most appropriate: a ‘cohort’ of farms that follow the same type of treatment, and are analyzed over time in relation to each other and to their own past states. One could still argue that a control is needed, though. The general idea is that some kind of time-series analysis becomes necessary, rather than a simple factorial analysis.

There’s much more to explore here…

Credit to…

  1. David Eddie, ResearchGate.
  2. ibid.
  3. Karl Siebertz, ResearchGate.

Leave a Reply

Your email address will not be published. Required fields are marked *