An Example in R

The code for this simulation is provided below for those following along at home. If you don’t care about code that’s fine. Just skip those parts and focus on the figures and discussion.

library(ggplot2)#plotting library
library(reshape2)#reshaping data
library(scales)#using percentages
library(TTR) # for the simple moving averages functions

set.seed(42) # set the random seed to replicate results

# Set Up Small Sample
# Randomly determine # leaving each month
s1 <- rbinom(n = 48, size = 20, prob = .05) #48 draws (months), 20 people, .05 prob of leaving
sp <- s1/20 # calculate the percentage leaving each month
mean(sp)

## [1] 0.06875

# Set Up Big Sample
# Randomly determine number leaving each month
b1 <- rbinom(n = 48, size = 100, prob = .05) #48 draws (months), 100 people, .05 prob of leaving
bp <- b1/100 # calculate the percentage leaving each month
mean(bp)

## [1] 0.04604167

Figure 1 shows simulated turnover rate (y axis) for this group calculated every month for 48 straight months assuming a 5% chance of leaving their job.

The rate skips all over the place relative to the 5% baseline (black line), shooting up to 15% or 20% in some months and then plummetting to 0% in others. Moreover, we also see a few instances of high rates in consecutive observations (see time 30-32).

The mean turnover rate for the small Accounting group over these 48 months is 6.8%, kinda sorta close to the 5% value that we specified in our simulation process.

### Structure data and get the figure
all <- data.frame(id = seq_along(bp), small = sp, big = bp) # setting up the data frame
all <- melt(all, id.vars = "id") # reshaping for use with ggplot2

### Plot Small Sample Only
ggplot(data = all[all$variable =="small",], aes(x = id, y = value, group = variable, color = variable)) + 
    geom_line(size = 1.2) + scale_colour_brewer(palette="Dark2", name = "Group Size") + 
    geom_hline(yintercept = .05) + ylim(0,.25) + scale_y_continuous(labels=percent, name = "Turnover Rate")

What happens to our turnover numbers when the group is larger? Figure 2 compares the smaller Accounting group results with the larger Sales group turnover.

We certainly see some ups and downs for Sales, but generally the rates are consistently closer to the true 5% rate in the simulations (actually, it’s 4.6%). Gone are the huge swings all the way up to 15% or 20%.

### Big and Small Sample Together
ggplot(data = all, aes(x = id, y = value, group = variable, color = variable)) + 
    geom_line(size = 1.2) + scale_colour_brewer(palette="Dark2", name = "Group Size") + 
    geom_hline(yintercept = .05) + ylim(0,.25) + scale_y_continuous(labels=percent, name = "Turnover Rate")

Remember that the ONLY difference between these two groups is the size of group, not the quitting process. Every employee within these simulations is operating under 5% probability of quitting each month. That’s it.

What’s the lesson?

Changes in the group size (or sample size) can have a dramatic impact on the variability of a measure even when the underlying the processes are exactly the same.

If you are using small groups, don’t jump to conclusions when the rates go sky high or drop through the floor.

Suggestion #1: Reduce Reporting Frequency

People want up-to-the-minute data but the Law of Small Numbers suggests that much of the fluctuations that we see are due to random variation and sampling error, not meaningful change.

One obvious solution is to report data less frequently, say, on a quarterly rather than a monthly basis.

If that sounds unrealistic, consider the following three benefits.

1. Reduced time spent creating reports
2. Reduced leader time trying to understand and explain random noise (v. true change)
3. Reduced likelihood of making decisions based on imaginary patterns.

There’s an argument to be made for simply reporting less often.

Suggestion #2: Use a Simple Moving Average

If you can’t reduce reporting frequency, you should consider using a moving average. This also sometimes referred to as a “running average” or “rolling average”.

A simple moving average is averaging some value over multiple time periods. In the example below, we average the current value with the preceding two values. We are a using a “bucket size” of three but the choice is up to you.

If this is the month of March, for example, then we would average the turnover data from March with the two months immediately preceding it (January and February). Similarly in April we would average the April data with that from February and March and so on.

To see how a simple moving average works, compare the first observations from the small sample data and the simple moving average of that same data.

Notice that the first two spots in the moving average are empty. That is because we don’t have two preceding observations to help us calculate a moving average.

Looking at the raw numbers, we can that the average of 2,3, and 0 is 1.67, giving us the first simple moving average value.

s1_sma <- SMA(s1, 3) #making a simple moving average, averaging over 3 items
head(s1, 10) #first 10 raw observations

##  [1] 2 3 0 2 1 1 2 0 1 1

head(s1_sma, 10) # first 10 moving average

##  [1]        NA        NA 1.6666667 1.6666667 1.0000000 1.3333333 1.3333333
##  [8] 1.0000000 1.0000000 0.6666667

Moving averages help smooth out the data. This makes it easier to spot trends and, in this current case, ignore fluctuating noisy data.

Figure 3 shows you the orginal 20-person Accounting turnover rate data together with the moving average data. Note how the moving average eliminates the huge peaks and troughs.

In practical terms this means eliminating the compulsion to explain striking but nonetheless random rises and falls.

s1_sma_perc <- s1_sma/20 # converting from raw number of to a rate
plot(sp, col = "red", type = "l", lwd = 2.5, main = "Monthly v. Simple Moving Average", 
     xlab = "Observation Month", ylab = "Turnover")
lines(s1_sma_perc, col = "blue", lwd = 2.5)

legend(x = 30,y = .18, legend = c("Monthly","Moving Average"), lty=c(1,1),
lwd=c(2.5,2.5),col=c("red","blue")) # gives the legend lines the correct color and width

Suggestion #3:

Find Other Measures Less Impacted

A final option is finding measures of “operational health” that might be less impacted by the Law of Small Numbers. For instance, instead of looking at completed sales numbers which might vary substantially from month to month (especially with just a few sales reps), look instead at more persistent, process-based measures like contacts initiated or outgoing current client calls.

Changes in sales numbers might be subject to a variety of external factors in any given month. Changes in the effort that sales reps put forward to secure those sales likely are not.