measuring attitudes in Other People's Money
How can we measure the effectiveness of Other People’s Money?
Research Question: How do “bridging moments” affect the reflective play experience — where “bridging moments” are moments that touch on (or even directly spell out) the reflective goals of the game. Do these moments serve as an anchor aiding in reflection, or do they irritate or distract players from reflection they would have done otherwise? This question is too broad to answer in one study, but this study will provide evidence in a specific context.
Ideal transformations (reflective goals):
- Disposition: Players more positively favor reasonably expensive large projects, possibly via taxation.
- Knowledge: Players better understand what an order of magnitude is, and how large projects that all seem “expensive” can still be separated by orders of magnitude.
- Behavior: Players will be more likely to logically reason about the expense of a project, rather than quickly assuming it is expensive.
For qualitative research, I added a playtest question for each ideal transformation, with the thought that we can do a qualitative analysis between groups that play with bridging moments, and those that play without.
Playtest questions:
- Standard Questions (For pilot testing only, remove for main study)
- Think aloud during the playing of the game
- How would you describe this game?
- Were there any confusions with the controls?
- What could have been explained better or earlier when I was teaching the game?
- General Reflective Questions
- Think aloud during the playing of the game
- What were you thinking about as you played the game?
- What kind of emotions or feelings did you feel / notice as you played?
- Specific Reflective Questions
- Is there anything for which your attitude changed?
- Is there something new you feel like you learned?
- Are there actions you took previously, that you’d change after the playing of this game.
- Questions for the “with-text” version of the game
- Let’s talk about the text. Do you have any specific thoughts about it?
- Do you feel like the text could get annoying, or did you not mind it?
- Questions for the “without-text” version of the game
- (No specific questions, just watch out for mentions of the game feeling too abstract, or if the player felt like they wanted more structure / focus / discussion of the topics at hand. Follow up on these mentions, maybe even ask if there’s something that would have made the game feel better.)
- Comparative questions
- How would you feel about the other version of this game? (Modify this question based on their version and previous responses)
For quantitative research, we can do a pre and post test of players, either through hired playtesters, or people who stumble on the game naturally. I’m thinking this would be in the form of an attitudes survey.
For surveys towards wealth inequality and government spending, there’s a good bunch out there. The World Values Survey - Inequality and International Social Survey Program have widely used and well validated questionnaires. They are a bit long, which is why I’m also interested in this Attitudes to Inequality Paper, since it has a shortened questionnaire.
If we want to get more creative, we can take inspiration from Inequality Aversion And Risk Aversion-Carlsson, which asked “What kind of society would you like your grandchild to grow up in? One that is inequal, but has a higher average income, or one that is more equal, but has a lower average income.” (paraphrased)
My favorite though, was Measuring Attitudes Towards Inequality-Amiel. In it, they asked a bunch of variations on this question: “Person R is 10x wealthier than person P. A tax takes $1,000 from person R. Some of that tax is given to person P, while the rest is lost in administrative fees. What amount needs to make it to person P for the tax to be worthwhile?”
I don’t think we should take that question directly, but the general idea is interesting. We could do something similar with “attitudes towards government spending.” For example we could ask participants: “You are in charge of a government agency, and are debating whether to implement a policy that would reduce air pollution. A research team comes back with a report indicating that the policy would have 200 Billion dollars in economic benefits, and would save 23,000 lives each year. It’s possible that the researchers missed something, but their report seems thorough. In your opinion, what is the maximum cost that would still make this policy to be reasonable?”
One of the intended transformations is “Behavior: Players will be more likely to logically reason about the expense of a project, rather than quickly assuming it is expensive.”
This transformation seems tougher to measure without doing some sort of delayed test. I’m not too familiar with these kinds of tests, but I would be open. I was also thinking of some sort of activity where participants are given a list of possible projects and are asked to sort them. We could see how the participants sorted the projects, how long it took them… but I’m not convinced it would give us the data we needed.
