measuring attitudes in Other People's Money
How can we measure the effectiveness of Other People’s Money?
Research Question: How do “bridging moments” affect the reflective play experience — where “bridging moments” are moments that touch on (or even directly spell out) the reflective goals of the game. Do these moments serve as an anchor aiding in reflection, or do they irritate or distract players from reflection they would have done otherwise? This question is too broad to answer in one study, but this study will provide evidence in a specific context.
Ideal transformations (reflective goals):
- Disposition: Players more positively favor reasonably expensive large projects, possibly via taxation.
- Knowledge: Players better understand what an order of magnitude is, and how large projects that all seem “expensive” can still be separated by orders of magnitude.
- Behavior: Players will be more likely to logically reason about the expense of a project, rather than quickly assuming it is expensive.
For qualitative research, I added a playtest question for each ideal transformation, with the thought that we can do a qualitative analysis between groups that play with bridging moments, and those that play without.
Playtest questions:
- Standard Questions
- Think aloud during the playing of the game
- How would you describe this game?
- Were there any confusions with the controls?
- What could have been explained better or earlier when I was teaching the game?
- What kind of emotions or feelings did you feel / notice as you played?
- Reflective Questions
- Is there anything for which your attitude changed?
- Is there something new you feel like you learned?
- Are there actions you took previously, that you’d change after the playing of this game.
- Comparative Questions
- Was the text annoying?
- How would you feel about the other version of this game?
For quantitative research:
For surveys towards wealth inequality and government spending, there’s a good bunch out there. The World Values Survey - Inequality and International Social Survey Program have widely used and well validated questionnaires. They are a bit long, which is why I’m also interested in this Attitudes to Inequality Paper, since it has a shortened questionnaire.
If we want to get more creative, we can take inspiration from Inequality Aversion And Risk Aversion-Carlsson, which asked “What kind of society would you like your grandchild to grow up in? One that is inequal, but has a higher average income, or one that is more equal, but has a lower average income.”
My favorite though, was Measuring Attitudes Towards Inequality-Amiel. In it, they asked a bunch of variations on this question: “Person R is 10x wealthier than person P. A tax takes $1,000 from person R. Some of that tax is given to person P, while the rest is lost in administrative fees. What amount needs to make it to person P for the tax to be worthwhile?”
I don’t think we should take that question directly, but the general idea is interesting. We could do something similar with “attitudes towards government spending.” For example we could ask participants: “You are in charge of a government agency, and are debating whether to implement a policy that would reduce air pollution. A research team comes back with a report indicating that the policy would have 200 Billion dollars in economic benefits, and would save 23,000 lives each year. It’s possible that the researchers missed something, but their report seems thorough. In your opinion, what is the maximum cost that would still make this policy to be reasonable?”
One of the intended transformations is “Behavior: Players will be more likely to logically reason about the expense of a project, rather than quickly assuming it is expensive.”
This transformation seems tougher to measure without doing some sort of delayed test. I’m not too familiar with these kinds of tests, but I would be open. I was also thinking of some sort of activity where participants are given a list of possible projects and are asked to sort them. We could see how the participants sorted the projects, how long it took them… but I’m not convinced it would give us the data we needed.
