Did you consider asking people a lot of frequency type questions, a kind of semi-objective measure of personality? E.g. how many times the last 2 weeks did you: cry, drink alcohol with drinks, play a board game with drinks, play an electronic game on computer/console/phone, browser Twitter, eat candy, watch TV in bed/sofa.
No, but it might be neat to do. I've been thinking that once I get enough data, I can remove some of the worse-performing items and replace them with new ones.
Thank you very much for posting this where we could read it! It's great to see this kind of thing on substack, and seeing this post convinced me to put up some of my own research, which I'd previously decided against - you can see it up here.
My personal take on your own work here (which you probably won't agree with) is that you've done a good job showing that the usual lexical pathway psychology has been using is actually on the correct track. When you say things like "Creativity (appears to conflate Creative problem-solving and Artistic creativity)," this is a feature, not a bug, of dimensional models. No one says that things like A) creative problem solving, and B) artistic creativity are literally the same; there's a large body of research on the difference between scientific and artistic success out there.* Yet there's obviously going to be some similarity between these two things that allows them to be positioned nearby in a space of personality traits.
More useful to you would probably be this bit of advice: The impact of personality on behavior is strongest when options are open.
Twelve different people wandering in the desert for three days are all going to be fighting over a bottle of water they come upon lying in the crevace between a rock. "If you were thirsty in the desert, would you want water?" is more a question designed to flush Lizardmen than an attempt to measure personality.
In other words, when you design concrete questions, a good strategy would be to focus on choices over duties. Having a stressful job, spending a lot of effort on parenting, and many other concrete details often depend on other situations or people around you much less than the effects of innate personality. I know that just over the past six months, my answers to those two questions would have changed dramatically.
* For example, I liked: Kaufman, S. B., Quilty, L. C., Grazioplene, R. G., Hirsh, J. B., Gray, J. R., Peterson, J. B., & DeYoung, C. G. (2016). Openness to experience and intellect differentially predict creative achievement in the arts and sciences. Journal of personality, 84(2), 248-258.
Thank you for your comment! Sorry it took a very long time for me to get around to responding.
"My personal take on your own work here (which you probably won't agree with) is that you've done a good job showing that the usual lexical pathway psychology has been using is actually on the correct track."
I'm not sure whether I agree or disagree. 😅
I think my post raises questions about certain specific traits (e.g. Conformity and Creativity), while it gives us traits great praise, and is sort of ambivalent about a lot of traits.
To me the biggest interesting thing is the comparatively strong effects for ideology compared to other traits. It makes me think ideology is more important than I used to consider it. (See also: https://twitter.com/tailcalled/status/1711843272455278678)
Another thing is that now I've done research with this large-scale approach, I start feeling like it would also be interesting to go the opposite way - zooming into the details of how people act in common situations, and why they act that way.
"When you say things like "Creativity (appears to conflate Creative problem-solving and Artistic creativity)," this is a feature, not a bug, of dimensional models. No one says that things like A) creative problem solving, and B) artistic creativity are literally the same; there's a large body of research on the difference between scientific and artistic success out there.* Yet there's obviously going to be some similarity between these two things that allows them to be positioned nearby in a space of personality traits."
This is what's not so obvious to me. If they are uncorrelated then that suggests that they are not actually positioned nearby in the space of personality traits - though I will have to increase the scale of my factor analysis to test it more carefully.
"More useful to you would probably be this bit of advice: The impact of personality on behavior is strongest when options are open."
I've heard of this theory before. It seems plausible but sometimes it was pretty hard to write nice items from people's responses in the qualitative survey.
One problem with concrete items is that they mean different things in different places. Take engaging in politics, for example. I have lived in Utah, Boston, and Mexico. It is very common to engage in politics in Boston, less so in Utah, and even less so in Mexico. So if you ask about politics to infer if someone is easy-going, Boston will come out looking extraordinarily high strung. Concrete items may be better because they require less interpretation, but even then people are quite bad at remembering even basic things. For example, you can ask large samples of heterosexual men and women their number of partners and the math just doesn't add up; men over-report (comparatively). Not to say we shouldn't use concrete items, just not obvious to me that they are better.
In my view there is a personality space which, thankfully, we can reduce to low dimensions. That's where any method starts. We can find that space via factor analysis of survey items or adjectives. I don't get so hung up on specific factors, per say. Any direction in personality space is real. Though perhaps a word like "creative" is interpreted more contextually than "agreeable". In personality space, I would interpret this as it referring to several directions, kind of a diffuse factor. I wouldn't exactly call that real, it's the descriptor's fault, not a lack of veracity of individual (and intra-individual) variation along those axes. One of my problems with the Big Five is that the interstitial space then gets short shrift; instruments are designed only to measure personality at each pole. But any other rotation in that space is just as real!
Much of this extends for other data reduction techniques as well such as network analysis, though the interpretation is slightly different than "factor".
It seems that to explore personality space we need data generated from humans, usually self report (though I have also used word vectors). With adjectives these are filtered through millions of people who evolved to gossip, ie find important individual differences and articulate them. I put a lot of stock in that filter, but one can get pretty similar results from items (Revelle and Condon found the Big Five as well as the little 27: https://www.sciencedirect.com/science/article/pii/S0191886920300945).
One of the things I hope with this test is that I can collect enough data to do a culture-level factor analysis. I ask about the country people live in as well as what their religion is, which gives a crude measure of culture. Maybe I will discover that by controlling for culture, I can get better results.
Have I mentioned that it would be cool to see your analysis of Aella's dataset? She also is experimenting on the best types of items and it would be interesting to see if her intuitions are buttressed by quantitative analysis
Average across enough items however, and different items will have this sort of bias in different directions, which at the test level, is much preferable to the global reference group effects that bias normal items.
Throw enough crap at the wall until it sticks, and you may even be able to find normal items which are minimally-biased and which have minimal reference group effects.
Yep. My prediction is that 1. Tailcalled will do this, and that 2. the end result will just be a rediscovery of the HEXACO model, maybe plus schizotypy/psychoticism.
Did you consider asking people a lot of frequency type questions, a kind of semi-objective measure of personality? E.g. how many times the last 2 weeks did you: cry, drink alcohol with drinks, play a board game with drinks, play an electronic game on computer/console/phone, browser Twitter, eat candy, watch TV in bed/sofa.
No, but it might be neat to do. I've been thinking that once I get enough data, I can remove some of the worse-performing items and replace them with new ones.
Thank you very much for posting this where we could read it! It's great to see this kind of thing on substack, and seeing this post convinced me to put up some of my own research, which I'd previously decided against - you can see it up here.
My personal take on your own work here (which you probably won't agree with) is that you've done a good job showing that the usual lexical pathway psychology has been using is actually on the correct track. When you say things like "Creativity (appears to conflate Creative problem-solving and Artistic creativity)," this is a feature, not a bug, of dimensional models. No one says that things like A) creative problem solving, and B) artistic creativity are literally the same; there's a large body of research on the difference between scientific and artistic success out there.* Yet there's obviously going to be some similarity between these two things that allows them to be positioned nearby in a space of personality traits.
More useful to you would probably be this bit of advice: The impact of personality on behavior is strongest when options are open.
Twelve different people wandering in the desert for three days are all going to be fighting over a bottle of water they come upon lying in the crevace between a rock. "If you were thirsty in the desert, would you want water?" is more a question designed to flush Lizardmen than an attempt to measure personality.
In other words, when you design concrete questions, a good strategy would be to focus on choices over duties. Having a stressful job, spending a lot of effort on parenting, and many other concrete details often depend on other situations or people around you much less than the effects of innate personality. I know that just over the past six months, my answers to those two questions would have changed dramatically.
* For example, I liked: Kaufman, S. B., Quilty, L. C., Grazioplene, R. G., Hirsh, J. B., Gray, J. R., Peterson, J. B., & DeYoung, C. G. (2016). Openness to experience and intellect differentially predict creative achievement in the arts and sciences. Journal of personality, 84(2), 248-258.
Thank you for your comment! Sorry it took a very long time for me to get around to responding.
"My personal take on your own work here (which you probably won't agree with) is that you've done a good job showing that the usual lexical pathway psychology has been using is actually on the correct track."
I'm not sure whether I agree or disagree. 😅
I think my post raises questions about certain specific traits (e.g. Conformity and Creativity), while it gives us traits great praise, and is sort of ambivalent about a lot of traits.
To me the biggest interesting thing is the comparatively strong effects for ideology compared to other traits. It makes me think ideology is more important than I used to consider it. (See also: https://twitter.com/tailcalled/status/1711843272455278678)
Another thing is that now I've done research with this large-scale approach, I start feeling like it would also be interesting to go the opposite way - zooming into the details of how people act in common situations, and why they act that way.
"When you say things like "Creativity (appears to conflate Creative problem-solving and Artistic creativity)," this is a feature, not a bug, of dimensional models. No one says that things like A) creative problem solving, and B) artistic creativity are literally the same; there's a large body of research on the difference between scientific and artistic success out there.* Yet there's obviously going to be some similarity between these two things that allows them to be positioned nearby in a space of personality traits."
This is what's not so obvious to me. If they are uncorrelated then that suggests that they are not actually positioned nearby in the space of personality traits - though I will have to increase the scale of my factor analysis to test it more carefully.
"More useful to you would probably be this bit of advice: The impact of personality on behavior is strongest when options are open."
I've heard of this theory before. It seems plausible but sometimes it was pretty hard to write nice items from people's responses in the qualitative survey.
Consider adding TTS to your stack 🙃
One problem with concrete items is that they mean different things in different places. Take engaging in politics, for example. I have lived in Utah, Boston, and Mexico. It is very common to engage in politics in Boston, less so in Utah, and even less so in Mexico. So if you ask about politics to infer if someone is easy-going, Boston will come out looking extraordinarily high strung. Concrete items may be better because they require less interpretation, but even then people are quite bad at remembering even basic things. For example, you can ask large samples of heterosexual men and women their number of partners and the math just doesn't add up; men over-report (comparatively). Not to say we shouldn't use concrete items, just not obvious to me that they are better.
In my view there is a personality space which, thankfully, we can reduce to low dimensions. That's where any method starts. We can find that space via factor analysis of survey items or adjectives. I don't get so hung up on specific factors, per say. Any direction in personality space is real. Though perhaps a word like "creative" is interpreted more contextually than "agreeable". In personality space, I would interpret this as it referring to several directions, kind of a diffuse factor. I wouldn't exactly call that real, it's the descriptor's fault, not a lack of veracity of individual (and intra-individual) variation along those axes. One of my problems with the Big Five is that the interstitial space then gets short shrift; instruments are designed only to measure personality at each pole. But any other rotation in that space is just as real!
Much of this extends for other data reduction techniques as well such as network analysis, though the interpretation is slightly different than "factor".
It seems that to explore personality space we need data generated from humans, usually self report (though I have also used word vectors). With adjectives these are filtered through millions of people who evolved to gossip, ie find important individual differences and articulate them. I put a lot of stock in that filter, but one can get pretty similar results from items (Revelle and Condon found the Big Five as well as the little 27: https://www.sciencedirect.com/science/article/pii/S0191886920300945).
One of the things I hope with this test is that I can collect enough data to do a culture-level factor analysis. I ask about the country people live in as well as what their religion is, which gives a crude measure of culture. Maybe I will discover that by controlling for culture, I can get better results.
Have I mentioned that it would be cool to see your analysis of Aella's dataset? She also is experimenting on the best types of items and it would be interesting to see if her intuitions are buttressed by quantitative analysis
You mean her chaos survey? I agree, but I don't think I have access to the data.
She gives researchers access, especially if they link to her
By the way, did you ever ask Aella for data / did Aella ever give you data?
Average across enough items however, and different items will have this sort of bias in different directions, which at the test level, is much preferable to the global reference group effects that bias normal items.
Throw enough crap at the wall until it sticks, and you may even be able to find normal items which are minimally-biased and which have minimal reference group effects.
Yep. My prediction is that 1. Tailcalled will do this, and that 2. the end result will just be a rediscovery of the HEXACO model, maybe plus schizotypy/psychoticism.