I’ve been thinking lately about simplified accounts of scientific method; simplified philosophy of science, I guess. Many people—often considering themselves ‘pro-science’—adopt a kind of dogmatism about how science works, or what it takes to be scientific. That science must involve experiments, or repeatability, for instance, or that you need a large data set and statistics to infer anything. There’s some reason to promote this kind of dogmatism: public trust in science’s power is reinforced by relatively easily packaged, easily understood, models of method. They’re reassuring.
But ultimately I think this simplification can lead to disaster. This is because it opens the door to what I’ll call the Dogmatist’s Debunker, an argument that looks something like this:
1. A particular method is necessary for scientific legitimacy;
2. A particular investigation doesn’t use that method;
3. Therefore, that investigation isn’t scientifically legitimate.
The Debunker rears its head in two contexts. First, a scientist from one domain wanders into another, noticing the different standards, different methods, and different rules. The scientist concludes that this other domain must be junk. Second, the public (or, more often, someone manipulating the public) notices/points out that some science (think climate science, evolutionary theory, medicine) isn’t really following the scientific method—and therefore lacks authority.
If you adopt a simple, dogmatist story about scientific method, when you peer behind the curtain and see how science really works, you’re in for a shock. A shock because science is a very human, idiosyncratic, messy business (yes, philosophers, historians and sociologists of science, this shouldn’t be news!). However, science often isn’t successful despite the mess – but because of it. The Dogmatist Debunker is unjustified because scientific investigation ranges across a variety of contexts: different kinds of questions are asked of different kinds of targets and probed using varying tools. If you want to criticize scientific practice you must navigate a range of contextual factors with care.
My aim today is to illustrate the failure of Dogmatic Debunking by looking at fossil preparation. I’ll first discuss an epistemic virtue: independence. Independence is roughly the idea that the theory being tested should be kept separate from the theories underlying the test. I’ll then turn to the division of labor between scientists and lab technicians and sketch how this promotes said virtue. With that in place, I’ll follow Caitlin Wylie’s work in contrasting labtechs with fossil preparators. She argues that fossil preparation is creative and idiosyncratic – not a virtue we typically associate with methodical lab work. I’ll then—speculatively!—argue that in the context of vertebrate fossil preparation, often idiosyncrasy and creativity promotes, not undermines, the production of trustworthy, legitimate scientific knowledge. I take this as an illustration of how thinking too simply about scientific method can blind us to the subtleties necessary for understanding why it works.
Before kicking off, I should point out that much of the work I’ll be drawing on—particularly Steven Shapin’s and Caitlin Wylie’s work—is naturalist in a way that mine is not. They’re interested in explaining scientific practice in terms that don’t presuppose its success or rationality. Caitlin, for instance, is interested in understanding the dynamics between researchers. She argues—I think persuasively—that fossil preparators’ claims to creativity serve to distinguish them from ‘mere’ labtechs and to provide ownership—power—over their work. Where the labtech/scientist relationship maps onto employee/boss; the preparator/paleontologist relationship is more collaborative. Both bring different expertise to the table. By contrast, I’m interested in seeing if we can give a vindicatory story about scientific practice—I’m asking whether the role preparators play is a role we’d like them to play. We could put the question like this:
Under what circumstances would a set-up like the one Wylie describes be optimal, epistemically speaking?
I don’t see any reason to think that naturalist and vindicatory (or critical!) explanations of scientific practice must conflict. It can be true both that fossil preparators emphasize their creativity to establish power relations in the lab and that this should reassure us about the epistemic credentials of fossil preparation.
Whew, let’s get going.
Independence.
When I want to test some hypothesis or theory, I need some evidence. Evidence doesn’t just come out of the blue—I need some background theory which links my observations to the theory. For example, using a fossil to reconstruct an extinct critter relies on understanding the processes of fossilization. Now, in doing so, I’d better keep the hypothesis I’m testing separate from my background theory. For instance,
Suppose Adrian is interested in the hypothesis that popcorn is the best snack. To test this, he needs a metric measuring ‘snack-bestness’. To prepare said metric, he gathers facts about popcorn. For instance, that pre-popped popcorn shells are made of cellulose (these are called ‘pericarp’), that the popping occurs due to water contained by the pericarp turning to steam and creating pressure, that the Aztec used it for Jewelry, that 100 grams of popcorn is about 375 calories, and so on (as a good scientist, Adrian has researched popcorn a lot). He then creates a metric based on these properties—the closer any snack gets to those properties, the better it is qua snack. Based on comparing a long list of snacks, Adrian concludes that, yes, popcorn is indeed the best snack.
Unsurprisingly, Adrian’s study is rejected from the prestigious Royal Society journal Interface – not because it’s about popcorn (because check this out), but because the hypothesis being tested was not independent of his background theory—you can’t determine that popcorn is the best snack using a metric which is constructed from an analysis of popcorn!
Independence, then, holds between (1) the theory which grants observations evidential relevance, and (2) the hypothesis on which that evidence bears (Alison Wylie has great work on this notion in archaeology). If independence fails, your study ends up being problematically theory-laden. Theory-ladeness in itself isn’t a problem—you need some theory in order for your observations to count as evidence. But it is problematic if that theory is too close to what you’re trying to test. Independence protects from both cognitive and theoretical versions of confirmation bias.
Okay, so independence is a thing we want. How do we ensure it in the lab?
Lab Technicians & Divisions of Labor
In the public imagination scientific studies are things that scientists do—or at least scientists are often thought of as the labcoated specialists playing with test tubes. Often, however, the day-to-day hands-on stuff is carried out by ‘lab technicians’. Labtech work is not creative, they are supposed to follow the rules—the protocols—of the studies they are carrying out. They are also ‘invisible’, very rarely mentioned or credited in publications. There is, then, a division of labor between the skilled but uncreative labtechs and the scientists.
Steven Shapin has argued that if we’re going to understand science, labtechs musn’t remain invisible. Ignoring them, by his lights, has reinforced “… predominant biases in the Western academic world [which] have traditionally portrayed science as a formal and wholly rational enterprise carried out by reflective individual thinkers” which “… block naturalistic understanding of scientific activity in favor of a set of idealizations” (Shapin 1989). As I’ve said, my interests aren’t purely ‘naturalistic’ in Shapin’s sense, but I certainly think that if we’re to understand or critique science, we’d better get how it works right.
The scientist/technician distinction appears to promote independence. Given the nefarious powers of confirmation bias, it seems like a good idea for the people preparing your evidence to be largely ignorant of the hypotheses that evidence will be used to test. Labtechs methodically follow procedure which is designed on the basis of background theories about the generation of the relevant evidence. Scientists then utilize this evidence in testing hypotheses. On this kind of picture, this division of labor has an epistemic purpose: guarding against failures of independence (NB: this is itself an idealization: many labtechs have sophisticated knowledge of the science they are involved in, but this doesn’t necessarily block the benefits of the division of labor…).
Fossil Preparation
In the world of paleontology, fossil preparators play the role of labtechs. Fossils are funny-shaped lumps of rock encased in more normal lumps of rock. People need to split the good (fossil) rock from the bad (just rock) rock. And this is super labor-intensive. Below are some shots of a truly remarkable nodosaur (a kind of ankylosaur) specimen which Mark Mitchell has been preparing for the Tyrell Museum (getting to go bug paleontologists and preparators at their place of work is one of the occasional perks of doing philosophy of paleontology!). Here’s a nice article about the find. Mark’s been working on the piece for over four years now, and has at least another year to go. One of the things which was fascinating about talking to him was the combination of a deep knowledge of anatomy and geology, fine-tuned manual skills, and a kind of idiosyncratic, partly tacit, approach to carving fossil from rock which such work required.
Left, Mark Mitchell being very patient with myself and some philosophy students from the University of Calgary. Right, the Nodosaur specimen, mid-prep.
Caitlin Wylie, who wrote a wonderful post for us recently, has interviewed fossil preparators and paleontologists across the US, developing a picture of the preparator which contrasts starkly with Shapin’s picture of the labtech. She emphasizes the creativity of fossil preparation and emphasizes the connections preparators identify between themselves and artists. First, both artists and preparators need similar skills: an eye for detail, a steady hand, and so forth. Second, what counts as a good prepared fossil turns in part on apparently tacit aesthetic criteria. These serve to help decide which fossils to prepare, who will prepare them and—most interestingly for our purposes here—what success looks like. Third, problem-solving is ‘creative’: there are no general standards or techniques for how one goes about preparing a fossil. That is, there is no analogue of the protocols which labtechs follow. Fossil preparation is idiosyncratic, involving specialized skills.
On the face of it, idiosyncrasy and the use of aesthetic criteria is not something we want in labtechs. If there is no standardized method for data generation, amalgamation becomes difficult, and it’s easy for biases to slip in. Moreover, surely aesthetics doesn’t promote truth or reliability? However, I think a vindicatory story can be told, and I’ll sketch it now.
There are two important features of many vertebrate paleontological fossils which I think justify preparators’ use of aesthetics to guide decisions, and their idiosyncratic methods. The first feature is that there is underdetermination between the original fossil and the prepared specimen (see Caitlin’s post). The second is that often paleontological reasoning is based on one shot studies. These are ‘one-shot’ in two senses. First, each vertebrate fossil has unique features—and fossil preparation is a largely one-directional process. You can’t go back and redo things. Second, sometimes the ‘smoking guns’ of historical investigation are single (or a few) finds.
We know an awful lot about how fossils form, and this allows us to infer from the morphology of a fossil to many things about the past. However, this is often not all that helpful for getting from a raw to a prepared specimen. In fossil preparation, there are many ‘choice points’ along the way which theory doesn’t determine, and it’s difficult to know whether ‘mistakes’ have been made or not. This is to say, for any one raw fossil, there are many possible, apparently ‘equally correct’ completed specimens. Philosophers have quite a lot to say about situations like these. They are often highlighted as circumstances where ‘non-epistemic’ values might make a difference to scientific practice. If your evidence doesn’t tell you which way to jump, and you need to make a decision to keep the investigation going, well, just find some way of making a decision. What that ‘some way’ is going to be will often depend on context. If, for instance, you’re testing the safety of some new drug, you might just want to play it on the safe side—although you need to balance the cost of a false positive (saying its safe when it’s not) against a false negative (sayings it’s not safe when it is).
Now, typically in paleontology the stakes aren’t so high. But regardless, there are some decision points during fossil preparation for which there are insufficient empirical constraints. So, it might seem reasonable to base our decisions on largely aesthetic grounds. Such a move, at the very least, meets one really important epistemic principle—one we’ve just met—it maintains independence. The aesthetic judgments of fossil preparators are likely to be independent of the kinds of things which paleontologists want to use the fossils to theorize about. That is to say, although involving aesthetics doesn’t seem to provide positive reasons to think we’re promoting truth, it does guard against failures of independence. Better to have fossil preparators using fine-tuned aesthetic judgements than for them to be thinking about the hypotheses which the fossils might be used to test! (it’s worth noting that Wylie (2015) suggests that the aesthetic language masks background theory – and so the decisions might not be properly ‘aesthetic’ after all – my point is that even if they are aesthetic, they can still maintain independence).
But why do we want the preparators to be idiosyncratic? Here’s something we don’t know about judgement made on these aesthetic bases: whether they have a negative impact on the fossil’s epistemic value. Especially considering the one-shot nature of fossil preparation, it’s hard to compare different methods. Presumably, some set of methods damages or biases fossils, while another set doesn’t. Insofar as we can’t tell which methods fall into which set, it would be wise to spread out our options. Roughly, we shouldn’t put all our methodological eggs in a single basket.
In sum, aesthetic decision-making guards against failures of independence, and idiosyncrasy guards against biased method. Critically, this vindicatory story relies on context. If our technologies and background theories progressed such that there was less underdetermination between prepared and unprepared fossil, or if we’re not in a one-shot situation (and in many invertebrate paleontology, and some vertebrate, contexts we’re not), then this justification for creativity would dissolve.
Which brings me to
A lesson:
The right way to generate evidence depends crucially on the kind of evidence you’re trying to generate, how much relevant background knowledge you have, what kinds of technological resources you have at your disposal, what question you’re trying to ask, and so on. The applicability of epistemic principles is context-sensitive. And this means that the Dogmatist’s Debunker is not justified. You can’t critique an investigative method simply by comparing it to another. You need to get into the details of the method’s employment: the action is local. (Naturally, this isn’t a new idea: indeed, it underlies much of STS and HPS, but I think it’s an idea which deserves more attention both in public and in general philosophy).
This, I think, has troubling consequences for the kinds of pictures of science disseminated in public. On the one hand, simple (dogmatist) accounts of science are reassuring, and serve to reinforce its (often attacked) authority. Moreover, thinking about the subtleties of context-sensitive scientific practice is hard and not everyone’s cup of tea. On the other hand, encouraging dogmatism leaves science open to being debunked in the way I’ve described. I’d love to say that I think we should encourage a richer understanding of philosophy of science—and I do think this—but I worry as well that such an approach doesn’t lend itself to the kind of bumper-sticker rhetoric which actually convinces people. I hope I’m being too pessimistic on that point.
Further Reading
For Caitlin Wylie’s views on fossil preparation, see:
Wylie, C. D. (2014). ‘The artist’s piece is already in the stone’: Constructing creativity in paleontology laboratories. Social studies of science, 0306312714549794.
A good starting place for Shapin’s work is:
Shapin, S. (1989). The invisible technician. American scientist, 77(6), 554-563.
Alison Wylie has a lovely discussion of independence (she calls it ‘vertical independence’) in:
Wylie, Alison (2011). Critical distance: stabilising evidential claims in archaeology. In Philip Dawid, William Twining & Mimi Vasilaki (eds.), Evidence, Inference and Enquiry. Oup/British Academy.
For an entertaining (although perhaps inaccurate…) examination of failures of independence, particularly due to cognitive confirmation bias, see:
Gould, S. J. (1996). The mismeasure of man. WW Norton & Company.
A good starting place for discussion of the role of values in science is:
Douglas, H. (2009). Science, policy, and the value-free ideal. University of Pittsburgh Press.
I'd also like to thank Caitlin Wylie for comments and suggestions!