Your inner hunter–gatherer is stopping you solving problems
Statistics has a people problem. And those people are scientists. Despite all the success that statistical design of experiments (DOE) has had solving problems in industry, many scientists and engineers still aren’t using it and instead waste their time on inefficient and ineffective methods. It is incredibly frustrating. But innovations such as automated experimentation and Bayesian optimisation might be about to change things.
The reason why scientists don’t use DOE is partly to do with familiarity, but it’s also because it requires a way of thinking that can be at odds with a scientist’s instincts. The most effective innovations in DOE have therefore come from those who understood this tension and were able to find a balance. Looking back at these achievements helps us see where the next innovations will come from.
Thinking outside the box
One of the 20th century’s most important innovators in data-driven problem solving was George Box, and he was no fan of theory for theory’s sake. He felt that a generally useful tool was preferable to a locally optimal solution – likening this approach to the way the human hand can carry out many different tasks.
These views were informed by Box’s own background in experimental science. He started out as a chemist, and discovered textbook statistical design principles while working at the Chemical Defence Experimental Station at Porton Down in the second world war. At ICI Dyestuffs in the 1950s he developed the response surface method that would go on to be widely used for optimisation in the process industries generally. His hands-on approach was successful because it focused on helping real chemists solve real problems: ‘I had to know details of the processes […] climbing up and down ladders, talking and arguing every day with technical staff and process workers, and teaching them a little bit about statistical design and analysis,’ Box explained.
Experimenters are uncomfortable with statistically designed experiments
By the 1990s, statistical approaches had become more sophisticated. Algorithms for ‘optimal design of experiments’ – still used in today’s DOE software – were developed to enable experimenters to create a customised experimental plan, rather than trying to force their problem into a ‘textbook’ design.
Yet, despite being demonstrably useful and (at least initially) addressing real needs in industry, these designs were not widely adopted. By contrast, a concept called definitive screening designs (DSDs), which researchers happened upon while exploring optimal design algorithms, has become much more successful.
The reason for this is that optimal design didn’t address the key problem that Box had identified working at ICI: experimenters are uncomfortable with statistically designed experiments. Optimal designs only amplify this discomfort with complex technical jargon (D-optimality seeks to maximise the determinant of the information matrix!) and an array of choices that are off-putting for most scientists and engineers.
Behavioural barriers to adopting DOE are deeply rooted
A DSD on the other hand is a more general solution that offers scientists and engineers simplicity and immediacy. I saw the value of these designs myself when I used them to improve a membrane filtration process. With five variables and limited time on the pilot plant we couldn’t find a good solution with existing design approaches. Using a DSD we were able to understand the important behaviours in just 15 runs and double the productivity!
Scientific hunter–gatherers
These behavioural barriers to adopting DOE have deep roots. In his ‘Apes in Lab Coats’ blog, Dennis Lendrum relates a study of 69 scientists attending a recent summer school in DOE. The researchers were given a simulation of a polymerase chain reaction and asked to maximise its yield by manipulating 12 different variables. Predictably, most scientists tried to simplify the problem by fixing variables to reduce the number of dimensions. But what is particularly fascinating is the unsystematic way they then explored the possibility space – tending to focus on areas of high reward and then making ‘excursions’ of increasing length to other regions as the returns decline. As Lendrum notes, this looks a lot like the ‘foraging strategies of apes and other animals … [that] evolved to exploit patchy resources.’
The best strategies for using automation’s offer of huge, high-dimensional experiments are yet to be devised
Clearly there is a big gap between this ineffective but instinctive foraging approach and the more efficient but less intuitive strategy of DOE. Bayesian optimisation (BO) might just be the compromise that helps to bridge this divide. Like other DOE methods, this is a data-driven strategy for exploring multidimensional systems. The key difference is that it offers rapid feedback, because the model is refined after every experiment and the algorithm then proposes where to look next to move closer to the goal.
The big opportunity for BO is therefore as a data-driven ‘recommender’, supporting scientists who carry out manual experiments. This partnership allows instinctive scientific foraging in which the experimenter still has agency, but with statistical guidance to enable efficient exploration of multifactor systems.
The future of DOE
However, there is already a scenario emerging in which none of these DOE methods will make sense. When automation enables parallel execution of dozens or hundreds of runs, approaches that focus on maximising information obtained from small numbers of runs no longer apply. The best strategies for using automation’s offer of huge, high-dimensional experiments are yet to be devised, and they will surely come from close collaboration between the experimenters and the researchers devising these methods. Ideally, we need generalists with a foot in each camp.
The history of DOE tells us we should take Box’s lead and listen with intent to the most pressing needs of industry to bring about innovations that can have the real impact.
Join us on 14 October to hear more from Dennis Lendrum about how we can control our scientific foraging instincts, and use tools for smarter experimental design.
No comments yet