« Hierarchical group means | Main | Stratify despite no interaction? »

MAGNITUDE, direction and statistical significance

From a response to a question:

In what I took as Epi 2 we were drilled on always reporting the direction, statistical significance and MAGNITUDE of associations. There are two ways to get the magnitude:

If you run a simple correlation it is the correlation coefficient. SAS output gives that, the sample size and the p-value. The r and the n determine the p, so I suppose you can use any two to approximate the third. If you report a "significant" finding it could be

A) r=0.9 n=10 p=0.05 or
B) r=0.1, n=10,000 p=0.04

In either case they are "associated", but in the first case the association "explains" 1 percent and in the second it "explains" 81 percent of the variation.

This is worse when findings are described as 'not significant'. Association A could be 'not STATISTICALLY significant' if the p-value is 0.0501, but I would certainly say it is CAUSALLY significant. (Meaning that there is an association, confounded or not.)

Another expression of magnitude is the regression coefficient. If changing caloric intake explains 100% of weight loss that's great, but it may or may not be useful in obesity prevention. If you have to cut 5,000 kcal per day to lose 1 pound a year then it is a worthless public health strategy.