I recently received an email from my employer, with a link and a request to fill out a short survey evaluating a new service available to our patients. I had a brief gap between patients, so I clicked and did the six questions. As I hit submit, it occurred to me that the survey seemed familiar. Hmmm. Looking back in my older emails, I found that I had received a similar request from the same group about 2 weeks earlier, with the same link. I contacted the person who had sent the second email.
Here is the ensuing conversation, edited for anonymity, clarity and grammar, not comedy. (This is a true story. You can't make this stuff up.)
Me: I did the survey this morning, but in retrospect, I think I may have done it two weeks ago. Was it the same one and you're doing it again?
Him: No, we're not doing it again. It's the same survey. We didn't get enough responses so we sent out the request again.
Me: Why would you get more responses the second time?
Him: Hang on. I'll look. (pause) We've already gotten a bunch more this morning.
Me: (Beginning to suspect where this is going, but reluctant to believe it.) So, it’s not a new survey, just another invitation to the old survey?
Him: Right.
Me: But how will you control for the fact that some people might fill it out twice?
Him: What do you mean?
Me: The same people likely to fill it out the first time will probably fill it out again, so many of the new responses are probably duplicates.
Him: So?
Me: But I thought you wanted a better response to the survey?
Him: Yes, and we already have a larger sample size.
Me: No, you don't. Or at least, you don't know if you do.
Him: Yes, we do. We have more responses.
Me: That's not the point. Sample size is the number of people who answer, not the number of answers.
Him: What's the difference?
Me: Well, let me explain it this way. You know that some people think it's a problem that only about 55% of Americans vote in a national election, right?
Him: Right.
Me: Should we let some people vote twice so we have more votes? Or would that screw up what the results mean?
Him: Oh. I hadn't thought of it that way.
About five minutes later there was an email asking people not to fill it out if they already had. Too late. The barn door is closed and the horse is nowhere to be seen.