Critique my intelligence research study questionnaire?

IQ Research
1

This thread is to invite critique for my research study. My supervision team are familiar with cognitive testing but from a neuropsych background, and as the only intelligence researcher per se at my university I’m throwing my questionnaire out to the community for some feedback. The basic concept is to operationalize an equation described by Arthur Jensen where life outcomes (success) = ability * personal characteristics * opportunity. Although my interest is in very high intelligence, I’m recruiting participants across the ability spectrum to draw some comparisons. The life outcomes in question have been based on large-scale talent searches but hopefully phrased to be accessible to all (I have my own reasons for including some outcomes with very low base rates - this will be clear when I write the study up!) Part 1 - Life Outcomes: Highest level of educational attainment, socioeconomic status, self-perceived success, life satisfaction (Diener) and the following specific life outcomes (check all that apply):

A. Academic and Scholarly Accomplishments • Academic contest or prize (e.g., Olympiad, essay competition) • Prestigious fellowship or award (e.g., Rhodes, Fulbright, Guggenheim) • Attendance at a highly ranked or elite institution (please specify) • Academic post at selective or research-intensive university • Tenure, professorship, or equivalent senior academic status B. Creative Accomplishments in the Arts and Humanities • Work(s) of art exhibited or sold • Theatrical, cinematic, or television production credit • Music published or commercially released • Literary publication (book, article, poetry, etc.) • Other significant cultural contribution (please describe) C. Business and Entrepreneurship • Founded or co-founded a business or social enterprise • Developed or introduced an innovative product, process, or practice • Achieved senior leadership or executive role in business, industry, or policy D. STEM and Technical Output • Award or distinction in mathematics, science, or technology • Author or co-author of peer-reviewed journal article • Authored other scientific publication (book, edited volume, technical report) • Patent(s) secured (number if known) • Software, algorithm, or application developed • Recipient of grant or research funding E. Rare or Prestigious Awards • Peak subject-specific award (e.g., Fields Medal, Turing Award, Pulitzer, etc.) • Nobel Prize or comparable international distinction • Named in Who’s Who or equivalent registry • Elected Fellow or Member of a national academy, Royal Society, or other prestigious body F. Societal, Cultural, or Public Influence • Contribution to public intellectual life (writing, commentary, activism) • Influence on broader societal, cultural, or intellectual trends • Service or election to public office • Demonstration of special talents or skills not covered above

Part 2: Personal Characteristics - I’ve focused on ambition here to keep this simple, using Hirschi & Spurk’s (2021) Ambition Scale. Part 3: Opportunity - Family background at birth (SES and education level of mother, father and any grandparent). Then the next part of the questionnaire that is my own creation asks about family and early environment support, as follows:

  1. My family or main caregivers actively advocated for my educational or career opportunities while I was growing up.

  2. My family or main caregivers knew how to access selective schools, scholarships, or academic competitions.

  3. My family or main caregivers had the knowledge to help me through the higher education application process and what to expect.

  4. My family or main caregivers had professional or social connections that helped open doors for you (such as introductions or recommendations)?

  5. I considered my home environment emotionally supportive of my learning or talent pursuits.

  6. I grew up surrounded by books, cultural activities, or conversations that stimulated curiosity.

  7. High educational or professional attainment was “normal” or expected in my family environment growing up.

  8. My grandparents or extended family provided encouragement, financial help, or mentoring.

  9. The final part of the questionnaire is a “Resource Generator” - asking about types of instrumental support the individual can access: A. Educational and Developmental Resources (Rel = a dropdown indicating their relationship to the person; Freq. = frequency of contact)

    Do you know someone who could… Yes No Unsure Rel. Freq.

    1 help you gain entry to an advanced or selective educational program?
    2 assist you in securing a scholarship or study funding?
    3 give you access to special learning materials, or resources such as IT, labs, or instruments?
    4 mentor you in a particular academic or creative area?
    5 connect you with others with similar types of talents or motivation?
    6 advise you on university or graduate-school applications?
    7 write a strong academic or employment reference for you?
    B. Professional and Career Development

    8 introduce you to influential people in a field of interest?
    9 help you obtain employment suited to your talents and interests?
    10 act as a professional sponsor or advocate?
    11 invite you to present, exhibit, or publish?
    12 arrange an internship, placement, or collaborative project?
    13 advise you on negotiation of salary, contracts, or promotions?
    14 connect you with investors or funders for a business or idea?
    15 help you gain public or media recognition for your work?
    C. Cultural and Creative Capital

    16 introduce you to professional artists, writers, or musicians?
    17 help you exhibit, publish, or perform creative work?
    18 connect you with a cultural or academic institution (museum, publisher, university)?
    19 nominate you for an award, fellowship, or residency?
    20 provide expert-level feedback on artistic or scholarly work?

D. Wider Milieu and Opportunity Networks

This section focuses on whether you know people outside your immediate
circle who could help make your talents visible, connect you to new arenas, or
act as champions or discoverers of your work.

21 notice or “spot” a talent and bring it to wider attention (for
example, by recommending or nominating you)?

22 introduce you to influential figures beyond your existing field (e.g.
cross-disciplinary or media contacts)?

23 open doors to selective or prestigious networks (e.g. professional
societies, cultural circles, think-tanks)?

24 act as a sponsor, patron, or champion who promotes you to
decision-makers?

25 create opportunities for public exposure of your ideas or projects
(press, conferences, exhibitions, online platforms)?

26 provide informal but pivotal introductions (“I know someone you should
meet”) leading to unexpected chances?

27 offer endorsement or credibility that helps others take your work
seriously?

28 connect you to cross-class or high-status social networks that you
otherwise would not access?

E. Reflection

How satisfied are you with your current access to these forms of help?
(1 = Very dissatisfied → 5 = Very satisfied)Have you ever been “discovered” or offered an unexpected opportunity
through a contact or coincidence? [Open text - optional]
Is there any kind of support you wish had been available but wasn’t? [Open
text - optional]

Any other comments? [Open text - optional]

Obviously, there’s a lot of theoretical background I can’t go into here - but what I’m trying to do is not just capture ability (the RIOT) and life outcomes but dive into the “forgotten” variable of opportunity - which always seems to be mentioned and then batted aside in studies, rather than properly operationalised and explored.

2

Gwenyth,

It’s a great study. I’m glad you’re looking into a wide variety of outcomes. It’s a good sign that you’re aware of the ones with a low base rate. I would suggest adding more outcomes that are expected to have base rates in the 1-25% range. SMPY studies (like this one and this one) list some of these outcomes you may find useful.

But the inclusion of non-career and non-educational outcomes is good. There’s more to life than work and school.

7 Likes
3

@Gwyneth This is ambitious and well-structured. Two quick thoughts: (1) Your “opportunity” operationalization is excellent - the Resource Generator especially captures social capital mechanisms that talent research typically handwaves. Consider adding a temporal component though - access to resources at different life stages (childhood vs early career vs mid-career) might matter differently. (2) For very high IQ participants, you might hit ceiling effects on the RIOT limiting variance in that range. Have you considered supplementing with SAT/GRE scores or other high-ceiling measures for your upper tail? Otherwise this fills a real gap in the literature.

4

I also think there might be a risk for significant recall bias. Successful people tend to remember their families as more supportive than they were, while unsuccessful high-IQ individuals may blame lacking opportunities when other factors are at play. Your family support items rely entirely on subjective perception, so I suggest adding objective markers (parents’ occupations and education, family income bracket, school type, and specific resources available) to see where memory diverges from facts.

5

@elipischke344 Agreed. This is a perfect opportunity for discrepancy analysis. We should specifically look at cases where the subjective support score is high but the objective resource score is low, and vice versa.

6

Thank you all, these are really great suggestions!

Most of my recruitment base will not have taken SAT/GRE, as I’m based in the UK and am likely to get participants from across the English speaking world. I’ve tried to make the measures as Western-world generic as possible.

I’m still considering how to control for age (career stage, etc.). Unfortunately, that’s the drawback of only being able to conduct a snapshot study.

Parental SES bracket and education have been included (as well as any grandparent if known, to gauge longer-term family background), so I think that should at least somewhat mitigate any recall bias in the family background part of the questionnaire.

It seems that where talent studies have analysed a very large number of variables, they generally do not correct for multiple comparisons but instead caution that these are exploratory studies. I am aware this can be tricky to explain to those who come from the experimental side of the disciplinary divide.