Bruce's Modelling Gripes, No. 6: Over-hyping significance of a simulation to Funders, Policy Makers and the Public

When talking to other simulation modellers, a certain latitude is permissible in terms of describing the potential impact of our models. For example if we say "This simulation could be used to evaluate policy options concerning ...", the audience probably knows that, although this is theoretically possible, the many difficulties in doing this. They make allowance for the (understandable) enthusiasm of the model's creator, for they know such pronouncements will be taken with 'a pinch of salt'.

However, it is a very different situation when the importance, or impact or possible use of models is exaggerated to an audience of non-modellers, who are likely to take their pronouncements at face value. This includes promises in grant applications, journal publications, public lectures and discussion with policy actors/advisers. They will not be in a position to properly evaluate the claims made and have to take the results on trust (or ignore them along with the advice of other 'experts' and 'boffins').

The danger is that the reputation of the field will suffer when people rely on models for purposes that they are not established for. The refrain could become "Lies, damned lies, statistics and simulations". This is especially important in this era where scientists are being questioned and sometimes ignored.

Some of the reasons for such hype lies in the issues discussed in previous posts and some seem to lie elsewhere.

• Confusions about purpose, thinking that establishing a simulation for one purpose is enough to suggest a different purpose
• Insufficient validation for the use or importance claimed
• Being deceived by the "theoretical spectacles" effect [note 1] -- when one has worked with a model for a while that we tend to see it through the "lens" of that model. Thus we confuse a way of understanding the world for the truth about it.
• Sheer fraud: we want a grant, or to get published, or to justify a grant, so we bend the truth about our models somewhat. For example promising far more in a grant proposal than we know we will able to deliver.

In a context of other modellers, we can be easily found out and understood. With others we can get away with it for a time, but it will catch up with us in terms of an eventual loss of reputation. We really do not want to be like the economists!

Note 1: "theoretical spectacles" was a phrase introduced by Thomas Kuhn to describe the effect of only noticing evidence that is consistent with the theory one believes.