CARMA: Syllabus for an “Everything I wasn’t told during my master’s degree” course

This is a living syllabus for my University of Helsinki course. Location: Unioninkatu 35, room 114, 12:15-13:45 on dates mentioned below. Target audience is non-mathematical students in social sciences.

Critical Appraisal of Research Methods and Analysis (CARMA) – Evaluating and not getting fooled by data in scientific and practical research contexts

Description: Research claims in news, science, and business can mislead people, either purposefully or inadvertently. How and why does this happen, and what mistakes, misconceptions and pitfalls should one avoid when evaluating data? This course will help participants assess data-based statements, and offer some tools to avoid getting fooled by them. It is meant for students who aspire to future careers, which involve undertaking, interpreting or commissioning research. This could include science in academic or other institutions, consumer/marketing research in business settings, evidence-based decision making as policy makers or journalists, among others. The course does not require specialising in quantitative methods, although basic familiarity can be useful.

I

  • The crisis of confidence in social and life sciences: State of affairs (4 September 2019) – slides

Learning objectives: Become acquainted with the recent developments regarding the so-called “replication crisis”. 

        1. Replication crisis: how it all started (this time around).
        2. Consequences of problematic practices.
        3. You’re not alone in misinterpreting p-values.

II

  • From questionable research practices and biased stories, to better evidence and/or decisions (11 September 2019) – slides

Learning objectives: Understand what the research community is doing to improve the quality of published research. Extrapolate to non-academic settings.

        1. Transparency and Openness Promotion (TOP) guidelines, and their rationale.
        2. Tricky rule-of-thumb questions to ask when being presented research (1/2: null findings). 

III

  • Magnificient mistakes and where to find them (18 September 2019) – slides

Learning objectives: Recognise some particular pitfalls in evidential statements. Understand that decisions in the field do not need to rely on correct predictive statements, let alone scientific evidence.

        1. Tricky rule-of-thumb questions to ask when being presented research (2/2: “statistically significant” findings). 
        2. Ways tests can fail: Type I/II mistakes. Type M and Type S mistakes.
        3. The difference between evidence of absence and absence of evidence: Black Swans and the Turkey Problem.
        4. When you don’t need to be right: green lumber, and a first taste of convexity.
        5. Heuristics: Simple rules that make us smart.

IV

  • On interpreting data nudes instead of summary tables (25 September 2019)

Learning objectives: Understand the rationale for visualising data, and what can be hidden when reporting summary statistics only. Learn to spot some common tricks used to visualise data in a favourable way to the presenter.

        1. Introduction to data nudes and The End of Average. 
        2. What gets lost in looking at numbers alone: Hidden uncertainty in the absence of distributions.
        3. Demons with(in) axes: Slaying or summoning effects with presentation tricks.

V

  • Complex systems and why they ruin everything [straightforward] (2 October 2019)

Learning objectives: Become familiar with general features of so-called complex systems. Understand how they can be thought of in the context of practical interventions.

        1. Complexity… explained? 
        2. Pathway thinking & complexity thinking in behaviour change science.
        3. Don’t camp at 1st order effects in dragon season.

VI

  • Interventions, randomisation and big data (9 October 2019)

Learning objectives: Understand the rationale behind randomising and experimenting/intervening, as well as some limitations of big data.

        1. The added value of experiments for inference.
        2. Why social interventions often fail.
        3. When is it safe(r) to intervene.

VII

  • Dynamic/idiographic phenomena, and hidden assumptions (16 October 2019)

Learning objectives: Describe the concepts of ergodicity and stationarity. Understand how they can mislead when not taken into account when e.g. assessing risks. 

      1. Models are stupid, stupid (helpful?) golems.
      2. Interaction vs. component dominant dynamics.
      3. The best map fallacy.
      4. (Model-free analysis)

 

2 thoughts on “CARMA: Syllabus for an “Everything I wasn’t told during my master’s degree” course

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s