This is not the signal-to-noise paradox, this is just a tribute.
By: Dr. Leo Saffin
The signal-to-noise paradox is a recently discovered phenomenon in forecasts on seasonal and longer timescales. The signal-to-noise paradox is when a model has good predictions despite a low signal-to-noise ratio which cannot be explained by unrealistic variability. This has important implications for long-timescale forecasts and potentially also predictions of responses to climate change. That one-line definition of the signal-to-noise paradox can seem quite confusing, but I think with the benefit of insights from more recent research, the signal-to-noise paradox is not confusing as it first seemed. I thought I would use this blog post to try to give a more intuitive understanding of the signal-to-noise paradox, and how it might arise, using a (cat) toy model.
Seasonal forecasting is a lot like watching a cat try to grab a toy. Have a watch of this video of a cat. In the video we see someone shaking around a Nimble Amusing Object (NAO) and a cat, which we will assume is a male Spanish kitten and call him El Niño for short. El Niño tries to grab the Nimble Amusing Object and occasionally succeeds and holds it in position for a short amount of time.
Without El Niño the cat, the Nimble Amusing Object moves about fairly randomly*, so that its average position over a window of time follows a fairly normal distribution.
Now suppose we want to predict the average (horizontal) position of the Nimble Amusing Object in a following video. This is analogous to seasonal forecasting where we have no skill. The best we can do in this case is to say that the average position of the Nimble Amusing Object will be taken from this probability distribution (its climatology).
This is in contrast to more typical shorter range forecasting where some knowledge of the initial conditions, e.g. the position and movement of the Nimble Amusing Object, might allow us to predict the position a short time into the future. Here, we are looking further forward, so the initial conditions of the Nimble Amusing Object gives us little to no idea what will happen.
So, how do we get any predictability in seasonal forecasting? Let’s bring back El Niño. We know that El Niño the cat likes to grab the Nimble Amusing Object, putting its average position more often to the left. This would then affect the probability distribution.
Now we have a source of skill in our seasonal forecasts. If we were to know ahead of time whether El Niño will be present in the next video or not, we have some knowledge about which average positions are more likely. Note that the probabilities still cover the same range. El Niño can pull or hold the Nimble Amusing Object to the left but can’t take it further than it would normally go. Similarly, El Niño might just not grab the Nimble Amusing Object meaning that the average position could still be to the right, it’s just less likely.
To complete the analogy, let’s assume there is also a female Spanish kitten, La Niña, and she likes to grab the Nimble Amusing Object from the opposite side, putting its average position more often to the right. Also, when La Niña turns up, she scares away El Niño, so there is at most one cat present for any video. We can call this phenomenon El Niño Scared Off (ENSO).
For the sake of the analogy, we will assume that La Niña has an equal and opposite impact on the position of the Nimble Amusing Object (to the limits of my drawing skills).
Now, let’s imagine what some observations would look like. I’ve randomly generated average positions by drawing from three different probability distributions (similar to the schematics). One for El Niño, one for La Niña, and one for neither. For the sake of not taking up the whole screen, I have only shown a small number of points, but I have more points not shown to get robust statistics. Each circle is an observation of average position coloured to emphasise if El Niño or La Niña is present.
Average Position
As expected, when El Niño is present the average position tends to be to the left and when La Niña is present the average position tends to be to the right. Now, let’s visualise it would look like if we tried to predict the position.
Average Position
Here, the small black dots are ensemble forecasts and the larger dot shows the ensemble mean for each prediction. Here, the forecasts are drawn from the same distributions as the observations, so this essentially shows us the situation if we had a perfect model. Notice that there is still a large spread in the predictions showing us that there is a large uncertainty in the average position, even with a perfect model.
The spread of the ensemble members shows the uncertainty. The ensemble mean shows the predictable signal: it shows that the distributions shift left for El Niño, right for La Niña, and are centred when no cat is present, although this isn’t perfect due to the finite number of ensemble members.
The model signal-to-noise ratio is the variability of the predictable signal (the standard deviation of the ensemble mean) divided by uncertainty (given by the average standard deviation of the ensemble members). The model skill is measured as the correlation between the ensemble mean (predictable signal) and observations. In this perfect model example, the model skill is equal to the model signal to noise ratio (with enough observations**).
The signal-to-noise paradox is when the model has good predictions despite a low signal-to-noise ratio which cannot be explained by unrealistic variability. So how do we get a situation where the model skill (correlation between ensemble members and observations) is better than the expected predictability (the model signal-to-noise ratio***). Let’s introduce some model error. Suppose we have a Nimble Amusing Object, but it is too smooth and difficult for the cats to grab.
This too-smooth Nimble Amusing Object means that El Niño and La Niña have a weaker impact on its average position in our model.
Importantly, there is still some impact, but too weak, and we still know ahead of time whether El Niño or La Niña will be there. Repeating our forecasts using our model with a smooth Nimble Amusing Object gives the following picture.
Average Position
What has changed is that the ensemble distribution shifts less strongly to the left and right for El Niño and La Niña resulting in less variability in the ensemble mean. However, the ensemble mean of each prediction is still shifting in the correct direction which means the correlation between the ensemble mean and the observations is still the same****. The total variability of the ensemble members also hasn’t changed, so the model signal-to-noise ratio has reduced because the only thing that has changed is the reduction in the variability of the ensemble mean.
The second part of the signal-to-noise paradox is that this low model signal-to-noise ratio cannot be explained by unrealistic variability. We could have lowered the model signal-to-noise ratio by increasing the ensemble spread, but we would have noticed unrealistic variability in the model, which is not seen in the signal-to-noise paradox. For the example shown here, the variability of the ensemble members is equal to the variability of the observations.
So there you have it. A signal-to-noise paradox, a model with good predictions despite a low signal-to-noise ratio which cannot be explained by unrealistic variability, in a fairly simple setting. This does bear some resemblance to the real signal-to-noise paradox. The signal-to-noise paradox was first seen from identifying skill in long-range forecasts of the North Atlantic Oscillation which is a measure of large-scale variability in weather patterns over the North Atlantic. It has also been shown that the El Niño Southern Oscillation, a pattern of variability in tropical sea-surface temperatures, has an impact of the North Atlantic Oscillation that is too weak in models. However, there are many other important processes that have been linked to the signal-to-noise paradox.
This model is very idealised. The impacts of the two cats were opposite but also in a very specific way that the overall impact of the cats did not affect the climatological probabilities*****. This is very idealised and not true of reality or even the schematics I have drawn. From the schematics I have drawn you can imagine that the net effect of the cats is to broaden the probability distribution so it is more likely to have an average position further from zero and that the weak model does not broaden this distribution enough.
In this situation we should see that the model distribution and the observed distribution are different, but this is not the case for the signal-to-noise paradox. There are a few possible reasons this would still be consistent.
- Model tuning – We noticed that our NAO was not moving around enough so put it on a longer string to compensate
- Limited data – The changes are subtle and we need to spend more time watching cats to see a significant difference
- Complexity – In reality there are lots of cats that like to grab the Nimble Amusing Object in various different ways. These cats also interact with each other
To summarise, I would say the important components from this cat-toy model to having a signal-to-noise paradox are that:
- There is some “external” source of predictability – the cats
- This source of predictability modifies the thing we want to predict (the Nimble Amusing Object) in a way that does not dramatically alter its climatology
- Our model captures this interaction, but only weakly (the overly-smooth Nimble Amusing Object)
Footnotes:
*assuming the human would just shake around this toy in the absence of a cat
**In the situation shown, when extended to 30 observations, the signal-noise-ratio (0.46) is actually slightly larger than the correlation between the ensemble mean and the observations (0.40) because the limited number of ensemble members leads to an overestimation in the variability of the ensemble mean, and therefore an overestimation of the signal-to-noise ratio.
***The ratio of these two quantities is known as the “Ratio of Predictable Components” (RPC) (Eade et al., 2014) and an RPC > 1 is often seen as the starting point in identifying the signal-to-noise paradox.
****The correlation is actually larger (0.45) for the sample I ran, but that’s just due to random chance.
*****I used skewed Gaussian distributions to generate the observations and model predictions. The average of the two skewed Gaussian distributions results in the original unskewed Gaussian distribution.
