I have just had the good fortune to attend this two-day London conference. There is really too much to report in just one blog post, so I'll start by focusing on one issue. More posts may follow.
Methodological and Measurement challenges in coaching research
How might you respond when you see a research article that claims that:'75% of coached respondents reported that thier relationships with others in their team had improved by 50%. At first sight this looks like a 'good result' for coaching, and adds further data to just how great coaching is. But does it?
Here are a number of questions we should ask before accepting this study at face value.
1 What was the quality of the inputs (e.g time, quality of the coaches)
2 What exactly was improved?
3 Did the respondents merely move from being 'Very ineffective' to just 'Ineffective'?
4 Who reported the changes - were these self -reports, with all the inherent biases of this approach.
One of the challenges faced by coaching researchers is defining what we actually mean by coaching - the precise nature of the intervetion. We also need proof that coaches actually do what they say they do or think they do. Furthermore, one of the most common research designs that we use, pre-post is inherently weak as a design; most of the measures that we use are not sensitive to detecting change. The impact of coaching may even work in a reverse direction (even though the intervention is effective!) For example, take a manager who is highly unreflective and rates herself quite highly on a trait; the effect of coaching might be to improve her reflection, with the result that she now rates herself lower on the trait because she has more self-understanding!
A better design than pre-post is the use of repeat measures both pre and post. Hence: AAABBB
In this design a series of repeat measures on the same person are taken (AAA); then repeat post measures are taken on this person (BBB). This allows for some stability of measurement.
