research

Getting physical

I had meant to move on from  this issue about the complexity of biomechanics and the quality of the research questions we ask … but then last night, in her comment, Quin drew my attention to a series of articles entitled “Biomedical research – increasing value, reducing waste” that were published in the Lancet in June (these are free to download but you need to register first – also free). They make fascinating reading. If you think I was a bit grumpy and cynical in what I wrote the week before last then you should have a look at what these guys are saying! (The articles are a bit heavy going and an easier and more entertaining alternative is Ben Goldacre‘s book Bad Science, it’s been around long enough that you can pick it up for 1p on Amazon and just pay the postage).

The issue marks the twentieth anniversary of an article, The scandal of poor medical research, written by the statistician Doug Altman in the BMJ which was perhaps the first public recognition of the poor quality of much clinical research with the tag line, “we need less research, better research and research done for the right reasons“. In another commemorative piece, Medical research still a scandal,  Richard Smith, who was editor of the BMJ at the time, laments on how little has changed, despite, perhaps, a wider awareness of the problems.

One of the responses to Bland’s original article which appealed to me has been given the title Theory must drive experiment. In it the author (a JA Morris from the Royal Lancaster Infirmary) attributes the problem of poorly formulated research questions to a failure of clinical scientists to develop an underlying theoretical basis for their experimental observations. This has always puzzled me as well and, over the years, I’ve come to the conclusion that, as someone who trained originally as a physicist, I’ve got a markedly different view of the world to many of my colleagues who trained in medicine or health sciences.

Arthur Eddington, “… do not put too much confidence in experimental results until they have been confirmed by theory”

As a physicist I expect to understand the results of my experiments and to be able to align them with an underlying theory. An understanding of that underlying theory then develops new research questions. My  knowledge continues to develop by the continued construction and refinement of the underlying theory (I’m sure there is a posh name for this in the philosophy of science). Taken to an extreme this results in Eddington‘s warning to the physicist  not to “put too much confidence in experimental results until they have been confirmed by theory“. Whilst at face value this sounds like an injunction against publishing experimental data it is actually a plea for careful consideration of the results in the light of the underlying theoretical framework and a refinement of that framework if  necessary.

Ernest Rutherford, “When it comes to science there is physics and there is stamp collecting”

I wouldn’t claim this as a unique skill of the physicist. Whilst over a hundred years ago Rutherford could quite reasonably(?) claim that “all science is either physics or stamp collecting” , the world has moved on. The rise of anatomy, physiology and particularly biochemical biology since that time mean that there are now underlying theoretical frameworks that we can use to explain the results of clinical and health sciences research.

We very rarely do though and I think this is partly attributable to education of doctors and allied health professionals being rooted in an earlier era. It wasn’t so very long ago that most results of clinical research where, effectively, beyond explanation. There was no point trying to fit those results into any underlying theoretical framework because the basic principles of that framework had not been established.  Knowledge in the clinical domain was essentially phenomenological – a knowledge of what happens rather than why it happens. Education then becomes a matter of teaching the facts rather than the underlying principles that link those facts. Of course if you don’t have an underlying theory to work from then you are going to find it much more difficult to generate sensible questions to drive the next generation of research. This is, of course, exactly the point that Morris was making and links to my post from last week.

As we move out of that era though we’ve got to put a much heavier emphasis on developing the underlying theoretical basis of our subject and using this to drive our research questions. Which leads me to my highlight of the ESMAC-SIAMOC conference which was Adam Shortland’s key-note talk, “The neuromuscular prerequistes of normal walking and the early loss of ambulation in cerebral palsy“. In it he reviewed what is now known about neurophysiological development and laid out a conceptual framework that explains much of what we observe in cerebral palsy and also provides a platform to generate new research questions … but then if you look at his CV you’ll see that he trained first as a physicist as well!

 

Advertisements

Training our PhD students to ask questions

This afternoon I give an annual lecture about what a PhD is. It is designed as part of the orientation for new PhD students and as a time for reflection to those already well established in their studies. It’s interesting for me to think about this talk after my post last week about how few of the papers I read or hear presented advance my understanding of a particular area.  Another way of looking at this is to comment on the number of papers and particularly conference presentations I see which don’t, to me, appear to contain a clearly formulated or insightful research question. Clearly if your research isn’t driven by a clear question then it is extremely unlikely that it is going to deliver a clear answer.

As research progresses, the fields within which we all function get more specialised and complex. Most of the obvious, simple questions have already been asked. This leads to a large choice of less obvious, derivative questions. Not only does productive research need to ask a relevant and interesting question but it needs to ask one that has the potential to be answered by the techniques that are available. As science progresses there are more and more techniques to choose from and more skill required in selecting the correct one (as in Kat Steels’ prize winning paper at ESMAC). In summary it would be quite reasonable to state that the principle skill that a research leader needs in the modern environment is the ability to think up well-formulated, interesting and answerable research questions

A PhD is many things but amongst these it serves as an entry level qualification for our future research leaders. The process  has changed markedly since I obtained mine (awarded almost exactly twenty years ago). I was chatting about this the other day with a colleague who was remembering completing hers with an absolute minimum of supervision. She commented on how this equipped her with a degree of self-sufficiency and independence that the current system doesn’t always deliver. I shared that experience, and on the one hand agree that it nurtured (inflicted?) independence. On the other, it meant that there are whole areas that I now consider essential for a PhD student to master that I didn’t even touch. No-one ever suggested I should have a systematic approach to searching the literature (at that time there wasn’t a great deal of literature to search). I don’t remember any consideration of the ethics of my research despite the fact that I was modifying external fixation frames that were being applied to patients in the local fracture clinic. There is no statistical analysis anywhere in my thesis. None of it ever got published. Perhaps most importantly of all, given the focus of this post, there is no formal statement of a specific research question. I’m glad that the 30 year old me didn’t have the 50 year old me as a PhD examiner, because I would have failed.

Given the complexity of research in the modern world, I don’t think we have any choice but to have highly structured PhD programmes. There simply isn’t time to allow students to do their own thing anymore. But there is a danger that, in cramming a PhD syllabus full of all the detail of how to systematically review, apply for ethics, master complex measurement techniques, perform rigorous statistical analyses and submit papers, we lose focus on the core research skill of asking a relevant,  insightful and answerable question.

This can be exacerbated by many PhD studentships these days being offered on the basis of funding to conduct a particular project. In these the research question may have been decided before the student even arrives. Sometimes only a general area has been identified, but even then supervisors will often have very strong views about the direction that research should take. I must admit to being guilty of this myself. If I’ve worked hard to get money to support a PhD studentship or am investing my time in supervision, I want to be certain that the resulting research answers the questions that I’m interested in rather than dreaming up their own.

And then of course there are pressures from the system. Universities in the UK are now judged, in part, by how many PhD students complete within the equivalent of 4 years full time study. To ensure this my own University asks the student to submit a Learning Agreement within 3 months, an Interim Assessment within one year and an Internal Evaluation including substantial drafts of thesis chapters before the end of the second year. The pressure is for them to hit the deck running. By far the easiest way to ensure this is to tell students what their research question is and let them to get on with it.

The one thing that is perhaps helping is the drift away from the classic PhD, which reported one large study, to the modern standard which reports several smaller ones. On this model the student gets the opportunity to design a number of studies. Each one will give practice in formulating a relevant, insightful and answerable research question. I wish students who are facing this challenge well and look forward to hearing them present their answers at the future conferences I attend.

Producer pays publishing

In UK Higher Education research quality is assessed every four or five years through the Research Excellence Framework (REF). We have  just finished our submission to REF 2014 and it won’t be until late next year that we receive our results. Already, however,  the government’s plans for the next REF are starting to emerge. The latest communication was an indication that only journal articles made available for open access are likely to be accepted.

The drive is now well and truly on for open access publishing. If public bodies or charities have funded research then surely the results should be open to everyone. I feel a bit like Canute in arguing against the tide, but then again I feel like Canute quite a lot of the time.

The most obvious point to make is that most journal articles are open to everyone already. If you log on to any of the big publisher’s web-sites you can buy a copy of any journal article you like (cost is typically about £30).  The issue isn’t whether articles are open or not, it is what they cost. Most universities and academics are happy to supply copies of individual articles free if you just e-mail a specific request. “Green” publishing effectively formalises this process.

The argument that the research has been funded by governments or charities is still relevant but it is not clear cut. There are many instances where governments invest money but expect the user to contribute at point of use. I’m sitting on a train at the moment that benefits from considerable public subsidy but I still expect to buy a ticket for each individual journey. Even where freedom of information legislation applies there is generally a charge for that information being made available.

The research paper is a product. It’s a product that, if it is useful, someone should be prepared to buy. The general rule for pricing is that the value is determined by what the consumer is prepared to pay.  Open access publishing, at least in the “Gold” form where a fee is charged to the publishing institution, is moving to a model where the producer is required to pay. I can’t think of any other product whose cost is charged to the producer – it runs counter to the logic of the market economy.

The problem with shifting to a producer pays model is that the economics of the system becomes driven by the need to produce rather than any consideration of whether the product is of any value. Most of us in academia are not challenged by the scarcity of data out there – we are overwhelmed by the quantity of it, particularly that which is of low quality.  There is a real risk that by removing demand side drivers and increasing supply side drivers the quality sieve will become coarser, not more refined and there are indications that we are beginning to see this already.

Of course the elephant in the room, and it is a very big elephant, is the academic publishing industry. Its initial reluctance to embrace open access publishing now appears only to have represented the thinking space required to work out a business model to exploit it most productively. Having established this, the big publishers are wading in with “open” arms. It’s a reflection of their inertia that they’ve taken so long to realise that this is a licence to print money. They can now charge the consumer (academic institutions are continuing to pay large fees to have ready access to content) and the producer. Over time the balance will shift from the former to the latter but it is becoming increasingly apparent that publishing a product whose value is set by the producer is a much more attractive proposition than one whose value is set by the consumer.

Publishing one paper to point out faults in another

This post is prompted by a discussion we had internally about a paper co-authored by one of my colleagues at the University (Dall et al. 2013). This was written as a response to an earlier paper (Tudor-Locke et al. 2011) based on data from the National Health and Nutrition Examination Survey (NHANES) for 2005-6 showing how many steps people took in each minute epoch as measured by an activity monitor. They assumed this was a measure of cadence and came up with the conclusion that:

Self-selected walking at 100+ steps/min was a rare phenomenon in this large free-living sample of the U.S. population, but study participants did accumulate 30 min/day at cadences of 60+ steps/min.

This is simply wrong. Whether the number of steps taken during any minute represents cadence or not will depend on whether the patient has been walking for a full minute or not. Take a person who is recorded as taking ten steps in one minute. This could come from someone who has walking difficulties and walked continuously for a minute but took only ten steps at a true cadence of 10 steps per minute. In this case steps per minute epoch is equal to cadence. Equally it could come from someone who had no difficulty walking and who walked ten steps at a cadence of 120 steps per minute but only for five seconds (ten steps) within the minute. In this case, which will be far more common than the first, cadence and steps per minute epoch are quite different. Recordings of 100 steps per whole minute is not rare because people walk with slow cadence but because it is actually very rare that we walk continuously for a whole minute (Orendurff et al. 2008). If you want to define a threshold value for cadence as was the original intention of Tudor-Locke et al. then you actually have to find some way of recording true cadence and not the number of steps per whole minute.

I think the issues are clear cut so far but then what should our response be? Malcolm and his colleagues had access to data collected with their activPAL device that would allow both true cadence and total number of steps per minute (step accumulation as they call it) to be calculated and demonstrated convincingly, but rather unsurprisingly , that the two are quite different. The published paper (Dall et al. 2013) makes a very interesting read – but should we have to go to this effort? Are there more effective ways of just telling people they are wrong!

Writing a letter to a Journal’s editor is one option but it always feels to me as if there is a time window on this – that the letter should really be submitted fairly soon after an article has been published. I’m not very good at keeping up with the current literature but when I’m working on a particular topic I will often read the relevant articles, both recent and not so recent, quite critically. Working like this it is often some time after publication that I read things that concern me.  A combination of my own inertia and the feeling that I am too late prevent me from doing any more about it.

Maybe I’m wrong in this – maybe we should feel free to use this route at any time that a mistake becomes apparent. Certainly this route ensures that the corrective letter is recorded in the same journal and under the same title as the original article and modern databases are becoming better at flagging this. A disadvantage of the approach of Dall et al. is that the new article is in a different journal published under a completely different title. In this case it has been published in a more technical journal (Medicine and Science in Sports and Engineering) which is unlikely to be read (or even searched) by readers of the original article (in the journal Preventive Medicine).

This wouldn’t be a problem if this were an isolated incident but biomechanics is a complex subject and I suspect that there are many more published mistakes and misconceptions than anyone in the field would want to acknowledge. In the worst case (again more common than we’d want to admit) published mistakes and misconceptions are adopted uncritically by other teams and before you know it what started off as an erroneous paper becomes first a series of erroneous papers and then a tried and trusted method (I’d see the use of CMC  (Kadaba et al. 1989) as a useful measure of repeatability of gait data as an example. Buy my book and read the appendix if you want to know more!).

The situation is exacerbated by the number of people who are involved in biomechanics as a secondary discipline. Some readers (and occasionally authors!) are not in a position to judge whether a method is valid or not. Does this increase the onus on those of us within the community who are aware of problems with specific papers to be more proactive in drawing people’s attention to them?

.

Dall, P. M., McCrorie, P. R., Granat, M. H., & Stansfield, B. W. (2013). Step Accumulation per Minute Epoch Is Not the Same as Cadence for Free-Living Adults. Med Sci Sports Exerc.

Kadaba, M. P., Ramakrishnan, H. K., Wootten, M. E., Gainey, J., Gorton, G., & Cochran, G. V. (1989). Repeatability of kinematic, kinetic, and electromyographic data in normal adult gait. J Orthop Res, 7(6), 849-860.

Orendurff, M. S., Schoen, J. A., Bernatz, G. C., Segal, A. D., & Klute, G. K. (2008). How humans walk: bout duration, steps per bout, and rest duration. J Rehabil Res Dev, 45(7), 1077-1089.

Tudor-Locke C, Camhi SM, Leonardi C, Johnson WD, Katzmarzyk PT, Earnest CP, Church TS. Patterns of adult stepping cadence in the 2005-2006 NHANES. Prev Med 2011;53:178-81.