David Roberts on Scaling Survey Responses

My name is David Roberts and I’m an independent consultant in evaluation and market research working out of Canberra, Australia. We recently had a discussion on AEA’s LinkedIn group focusing on using scales in surveys. While I am not an expert in analyzing scales here are some things I have found useful:

Rad Resource – Research from Jon Krosnick: Scroll down on the bio page from this Stanford professor to review summaries of his research on scales and to access study reports.

Hot Tip – Scaling Approach 1: One option for scaling responses is to analyze each individual’s responses and then score each response against the range of that individual’s responses. The simplest way to do so, is to treat each individual’s normal responses as varying around 0 and score accordingly. So if one person consistently rates between 4 and 5, a 4 is re-scored as -1 and a 5 is scored +1. Other responses are re-scored in terms of their distance from that individual’s median score. You can then analyze the scores for each question rather than the raw responses. It works better if you use at least a 7 point scale (Krosnik’s work suggests you should do that anyway). You can also use more sophisticated scoring methods based on range and standard deviation of the individual’s responses, but the utility of such an analysis is marginal for most applications.