Complexity methods for behavioural sciences: YouTube channel and resources

In 2019 I attended an exciting summer school; Complexity Methods for Behavioural Science: A Toolbox for Studying Change. Later, we – that is, the University of Helsinki Behaviour Change and Wellbeing Group – had the opportunity to invite Fred Hasselman, who devised the course, to Finland. He gave an overview talk as well as a 3-day workshop, which I recorded with the success expected from a guy holding an actual video camera for the first time of his life. This page collates resources regarding the course.

For all the recordings, see our YouTube channel. There are two playlists; one for short snippets and another one for full-length lectures. Here are some tweets on the course, with links to further resources.

Lecture 0 (video, slides) – Overview of complexity science and its applications in behavioural sciences. Also see shorter snippets on ergodicity, interaction- vs. component-dominant dynamics, and my interview with Fred.

Lecture 1 (video, slides 1-25) – Introduction to Complexity Science: Dissipative systems, Self-Organization, Self-Organised Criticality (SOC), Phase transition, Interaction Dominant Dynamics, Emergence, Synchronisation.

Lecture 2 (video, slides 25-74) – Introduction to the mathematics of change: Logistic Map, Return Plot, Attractors. [The beginning of the lecture was cut due to camera problems; please find a great introduction to the logistic map here.]

Lecture 3 (video, slides) – Basic Time Series Analysis: Autocorrelation Function, Sample Entropy, Relative Roughness

Lecture 4 (video, slides) – Detecting (nonlinear) structure in time series: Fractal Dimension, Detrended Fluctuation Analysis, Standardised Dispersion Analysis

Lecture 5 (video, slides) – Quantifying temporal patterns in unordered categorical time series data: Categorical Auto-Recurrence Quantification Analysis (RQA)

Lecture 6 (video, slides) – Quantifying temporal patterns in continuous time series data: Phase-space reconstruction, early warning signals of phase transitions

 

Matti spiral

Literature:

  1. Mathews, K. M., White, M. C., & Long, R. G. (1999). Why Study the Complexity Sciences in the Social Sciences? Human Relations, 52(4), 439–462. https://doi.org/10.1023/A:1016957424329 [INTRO COMPLEXITY SCIENCE]
  2. Richardson, M. J., Kallen, R. W., & Eiler, B. A. (2017). Interaction-Dominant Dynamics, Timescale Enslavement, and the Emergence of Social Behavior. In Computational Social Psychology (pp. 121–142). New York: Routledge. [INTERACTION-DOMINANCE]
  3. Molenaar, P. C., & Campbell, C. G. (2009). The new person-specific paradigm in psychology. Current directions in psychological science, 18(2), 112-117. [ERGODICITY]
  4. Kello, C. T., Brown, G. D., Ferrer-i-Cancho, R., Holden, J. G., Linkenkaer-Hansen, K., Rhodes, T., & Van Orden, G. C. (2010). Scaling laws in cognitive sciences. Trends in cognitive sciences, 14(5), 223-232. [SCALING PHENOMENA]
  5. Lewis, M. D. (2000). The promise of dynamic systems approaches for an integrated account of human development. Child development, 71(1), 36-43. [STATE SPACE, DYNAMICS]
  6. Olthof, M., Hasselman, F., Strunk, G., van Rooij, M., Aas, B., Helmich, M. A., … Lichtwarck-Aschoff, A. (2019). Critical Fluctuations as an Early-Warning Signal for Sudden Gains and Losses in Patients Receiving Psychotherapy for Mood Disorders. Clinical Psychological Science, 2167702619865969. [DYNAMIC COMPLEXITY]
  7. Olthof, M., Hasselman, F., Strunk, G., Aas, B., Schiepek, G., & Lichtwarck-Aschoff, A. (2019). Destabilization in self-ratings of the psychotherapeutic process is associated with better treatment outcome in patients with mood disorders. Psychotherapy Research, 0(0), 1–12. https://doi.org/10.1080/10503307.2019.1633484 [DYNAMIC COMPLEXITY]
  8. Richardson, M., Dale, R., & Marsh, K. (2014). Complex dynamical systems in social and personality psychology: Theory, modeling and analysis. In Handbook of Research Methods in Social and Personality Psychology (pp. 251–280). [INTRO COMPLEXITY SCIENCE – Social and personality psychology]
  9. Wallot, S., & Leonardi, G. (2018). Analyzing Multivariate Dynamics Using Cross-Recurrence Quantification Analysis (CRQA), Diagonal-Cross-Recurrence Profiles (DCRP), and Multidimensional Recurrence Quantification Analysis (MdRQA) – A Tutorial in R. Frontiers in Psychology, 9. https://doi.org/10.3389/fpsyg.2018.02232 [MULTIDEMINSIONAL RQA]
  10. Webber Jr, C. L., & Zbilut, J. P. (2005). Recurrence quantification analysis of nonlinear dynamical systems. In Tutorials in contemporary nonlinear methods for the behavioral sciences (pp. 26–94). Retrieved from http://www.saistmp.com/publications/spiegorqa.pdf [RQA]
  11. Marwan, N. (2011). How to avoid potential pitfalls in recurrence plot based data analysis. International Journal of Bifurcation and Chaos, 21(04), 1003–1017. https://doi.org/10.1142/S0218127411029008 [RQA parameter estimation]
  12. Boeing, G. (2016). Visual Analysis of Nonlinear Dynamical Systems: Chaos, Fractals, Self-Similarity and the Limits of Prediction. Systems, 4(4), 37. https://doi.org/10.3390/systems4040037 [LOGISTIC MAP, DERTERMINISTIC CHAOS]
  13. Kelty-Stephen, D. G., Palatinus, K., Saltzman, E., & Dixon, J. A. (2013). A Tutorial on Multifractality, Cascades, and Interactivity for Empirical Time Series in Ecological Science. Ecological Psychology, 25(1), 1–62. https://doi.org/10.1080/10407413.2013.753804 [MULTI-FRACTAL ANALYSIS]
  14. Kelty-Stephen, D. G., & Wallot, S. (2017). Multifractality Versus (Mono-) Fractality as Evidence of Nonlinear Interactions Across Timescales: Disentangling the Belief in Nonlinearity From the Diagnosis of Nonlinearity in Empirical Data. Ecological Psychology, 29(4), 259–299. https://doi.org/10.1080/10407413.2017.1368355 [(MULTI-)FRACTAL ANALYSIS]
  15. Hawe, P. (2015). Lessons from Complex Interventions to Improve Health. Annual Review of Public Health, 36(1), 307–323. https://doi.org/10.1146/annurev-publhealth-031912-114421
  16. Rickles, D., Hawe, P., & Shiell, A. (2007). A simple guide to chaos and complexity. Journal of Epidemiology & Community Health, 61(11), 933–937. https://doi.org/10.1136/jech.2006.054254. [INTRO COMPLEXITY SCIENCE – Public health]
  17. Pincus, D., Kiefer, A. W., & Beyer, J. I. (2018). Nonlinear dynamical systems and humanistic psychology. Journal of Humanistic Psychology, 58(3), 343–366. https://doi.org/10.1177/0022167817741784.  [INTRO COMPLEXITY SCIENCE – Positive psychology]
  18. Gomersall, T. (2018). Complex adaptive systems: A new approach for understanding health practices. Health Psychology Review, 0(ja), 1 – 34. https://doi.org/10.1080/17437199.2018.1488603. [INTRO COMPLEXITY SCIENCE – Health psychology]
  19. Nowak, A., & Vallacher, R. R. (2019). Nonlinear societal change: The perspective of dynamical systems. British Journal of Social Psychology, 58(1), 105-128. https://doi.org/10.1111/bjso.12271. [INTRO COMPLEXITY SCIENCE – Societal change]
  20. Carello, C., & Moreno, M. (2005). Why nonlinear methods. In Tutorials in contemporary nonlinear methods for the behavioral sciences (pp. 1–25). Retrieved from https://nsf.gov/pubs/2005/nsf05057/nmbs/chap1.pdf [INTERACTION DOMINANCE, ERGODICITY]
  21. Liebovitch, L. S., & Shehadeh, L. A. (2005). Introduction to fractals. In Tutorials in contemporary nonlinear methods for the behavioral sciences (pp. 178–266). Retrieved from https://nsf.gov/pubs/2005/nsf05057/nmbs/chap5.pdf [FRACTAL ANALYSIS]

 

 

Suomenkielisiä työkaluja COVID-19 taisteluun; yksilöille, yrityksille ja päätöksentekijöille

This post curates Finnish translations (mostly NECSI guidelines) for stopping the Coronavirus pandemic. Tälle sivulle olen koonnut hyvinä pitämiäni suomenkielisiä tekstejä. Suomentajana Thomas Brand, ellei toisin mainita.

Marraskuussa 2019 sain stipendin turvin mahdollisuuden osallistua Nassim Talebin riskinhallintaryhmän koulutukseen New Yorkissa. Siellä käsittelimme pandemiankaltaisia riskejä ja toimintaa niiden välttämiseksi. Muutamaa kuukautta myöhemmin pääsinkin elämään painajaista nähdessäni, että käytännössä kaikki länsimaat toimivat täysin vastoin varovaisuusperiaatetta (ts. joukkotuhon uhka on aina vältettävä agressiivisin toimin), luottaen “parhaaseen nykytietoon” viiveellä ilmenevän riskin torjumisen sijaan.

scientific_briefing
Kokous, joka käytiin vuoden 2020 alussa jokaisessa maailman maassa. Lähde: xkcd

Alla hyviä kirjoituksia, jotka ovat pääosin alunperin NECSI-instituutin tuottamia.  NECSI:lla on pitkä historia hallitusten ja järjestöjen kuten WHO:n konsultoinnissa mm. Ebola ja Zikavirus-epidemioita nitistettäessä, mutta myös muissa kompleksisissa ongelmissa, joihin perinteinen matemaattinen mallinnus ei pure. Koronavirus-pandemiaan liittyvään vapaaehtoisten globaaliin verkostoon voi liittyä täältä; tekemistä on käännöksistä some-aktiviteettiin, maskien ompeluun, hengityslaitteiden suunnitteluun, verkkosivujen ja mobiilisovellusten luomiseen ym.!

Lyhyitä perusohjeistuksia:

Ehdotuksia henkilökohtaiselle toiminnalle tilanteen parantamiseksi:

Jos koet lieviä tai kohtalaisia oireita:

Jos osaat ommella, tai muuten luotat kätevyyteesi:

Ohjeita elinkeinoelämän toimijoille:

Ohjeita ja esseitä yhteiskunnallisille päättäjille:

Mallintamiseen, ennakointiin ja pandemiatutkimukseen liittyviä kirjoituksia

 

koronaohjeet

Coronavirus, lifestyle diseases and the Shadow Mean

In this post, I introduce fat-tailed distributions and the concept of the Shadow Mean, with implications to how seriously multiplicative events should be taken in the society. [Addendum: If you want a technical treatment of the proper Shadow Mean approach instead of my caricature, see this]

I keep getting struck by how often we see well-meaning educated people comparing phenomena such as terrorism and epidemics to the “as much or more” dangerous lifestyle diseases. I even saw one of the smartest health psychologists I know commit this error in their professorial inauguration speech. Note, that I’m not against preventing non-communicable diseases; in fact, that’s what my dissertation is about. But we need to be vigilant on how risks work.

Here’s a chart from the aforementioned presentation, where you can clearly see that, all else equal, we should be diverting almost all our prevention resources to the biggest killers, which are lifestyle diseases:

Rik causes of death

The problem is, that all else is not equal. Why?

It has to do with a concept called “Shadow Mean” (capitalised for ominosity), which relates to “fat tailed” distributions. I’ll explain more later.

But let us first consider some properties of the Coronavirus pandemic, and how they differ from the common flu – and, by extension, to lifestyle diseases. To do so, I’ll give the floor to Luca Dellanna (Twitter, website), who kindly contributed his thoughts to this blog:


Luca Dellanna: Six unintuitive properties of the current pandemic

1/6: Asymmetry (part I)

“The cost of paranoia is bounded. The sooner we get paranoid, quicker we can get a handle on things, sooner we can confidently go back to business as usual the cost of “letting it happen” is unbounded. Here is the tradeoff in the US: Restrict international travel now and maintain our ability to move freely domestically or keep the flows coming and inevitably have to restrict movement both internationally and domestically. The choice is clear.” – Joe Norman (link)

There is enough evidence that the pandemic is inevitable. The only question is how big and how fast we want it.

The costs of preventing the pandemic are mostly linear. Closing down schools today for one month costs roughly as much as closing them for one month in April. Closing down 3 schools costs roughly half as closing down 6 (assuming the same size).

Instead, the costs of letting the pandemic grow are nonlinear.

Letting the pandemic run today might mean 100 more people infected tomorrow. Letting the pandemic run next week might mean 1000 more people infected the following day.

And it gets worse (see the next point).

2/6: Nonlinearities

“In the US, we have 2.3 million people in prison. I cannot imagine a way to stop #coronavirus from spreading like wildfire among that population. How will federal, state, & local authorities handle this?” – Jon Stokes (link)

Another example of the non-linear consequences of the pandemic.

A pandemic that “knocks-off” (i.e. prevents from working, for any reason) 0.1% of the workforce is bad but not that bad.

A pandemic that “knocks-off” (i.e. prevents from working, for any reason) 0.1% of the workforce in a clustered way is much worse: it means that some companies lose a large percentage of their workforce for a few days or weeks and must close the operations (whereas others are directly unaffected).

A pandemic that “knocks-off” (i.e. prevents from working, for any reason) 0.2% of the workforce is ten times worse than a 0.1% pandemic – for there are less workers to covers those who are sick, for one company closing creates problems downstream the supply chain, and so on.

The worst case is so bad that it makes sense planning for it even if it has low chances to happen (which is itself a strong assumption on too uncertain variables).

3/6: Impact

“The difference between the flu and the coronavirus is that between a tide and a tsunami. The same amount of water, but the impact is different because the tsunami arrives all at once.” – Roberto Burioni (link)

As I explained on Twitter, the problem is not (only) the current mortality, but the mortality we can get if our healthcare system gets overwhelmed. People won’t receive the care they need, even for conditions unrelated to the coronavirus.

“If a juggler can juggle 4 balls letting them drop 1% of time,  then he can also juggle 10 balls letting them drop 1% of time.” – this is how most people estimate mortality. As if healthcare was a fully elastic system.

4/6: Asymmetry (part II)

“Asymmetry. Convex decision. So long as there is no risk of harm from masks & disinfectants, the decision is wise, in spite of the absence of evidence– Nassim Nicholas Taleb (link)

Face masks do not offer full protection, but they do offer some protection. As long as you remove them carefully and they don’t make you sweat (so that you’re tempted to touch your face), they’re better than nothing.

Their cost is minimal and bounded, their benefit is large and unbounded (at least for you: they might save your life).

Of course, there is the argument that face masks are finite and they should be allocated where they’re the most needed. It’s a valid argument. But let’s focus on the asymmetry of the cost-benefit, because it applies to another method as well: washing hands and disinfecting.

Their cost is extremely low. I’m baffled that so few people are doing it first thing while arriving home.

Don’t be penny-wise but pound-foolish with your time.

5/6: Testing

“True epidemic in Iran and South Korea, community spread in Italy, confirmed transmission from Iran to multiple countries, the US basically isn’t testing anybody… and as far as I can tell it’s gauche even to mention [the virus] in public in the United States.” – @toad_spotted (link)

If a country doesn’t like to talk about a problem, it will have to talk about that problem.

Problems grow the size they need for you to acknowledge them.
The virus is already here, it’s just not evenly detected. – Balajis Srinivasan (link)

6/6: Infection

“I just realized that when people say ‘yeah but you won’t die’ they mean ‘yeah you’ll become a carrier and make everyone you see sick but not die’.” – Paul McKellar (link)

There are many replies to “the coronavirus is not that mortal”.

  • “15% mortality in older people (80+ years old) almost means a Russian Roulette if they get infected”.
  • One’s chances of dying depend on the number of infected people he meets in his day-to-day (because the more he meets, the more the chances he gets the virus).
  • We don’t know! There are many reasons that prevent us from pinpointing the mortality of the virus in a way that is predictive of the future. We should assume the worst scenarios until we can rule them out. (Why? Because asymmetry and nonlinearities; the content of points #1 and #4 above.)

Luca

[Luca’s newsletter is pretty much the only one I’ve ever found positively thought-provoking; if you want to hear more of his ideas, subscribe here]


 

Horizontally challenged tails

What does this have to do with lifestyle diseases? Well, while the incidence of the common flu is quite unlikely to quadruple from one year to the next, it is much, much less likely, that the incidence of e.g. cardiovascular disease would do the same.

Let’s look at an example. In the left plot below, you see what a mortality rate from a fat tailed distribution would look like. There are two years, when you have an extreme case – something psychologists are used to just eliminating from the data. Note, that outliers are different from extremes; an outlier may be a badly measured observation, whereas an extreme lies within the conceivable boundaries of the phenomenon.

fat and thin tails
Figure by me; code available here

The left plot could signify a viral epidemic. Say we are living year 26; the mean observed annual mortality would be around 900, and you probably aren’t too worried; things are almost exclusively very calm. But, given the fat-tailed distribution, extreme values are possible and upon surviving year 27, the mean would be almost 6000. Before it’s seen, this is known as the Shadow Mean; there are yet unobserved cases we can infer from the mechanics that produce the fat-tailed distribution, but which are not (yet) observed empirically.

Contrast the situation with that on the right plot, which could signify deaths from accidents in a country like Finland. In 900 years, we still have not observed one with over 2500 deaths (nb. this is just simulated data from a thin-tailed distribution). The mean is about 1000 and if we omit the maximum observation, it remains practically identical.

lawnmowers
Figure by Stefan Gasic; see his work here!

N-th order matters

Time and second-order effects – that is, things that happen as an indirect consequence of an event – are of great importance when something extreme happens. Let us run a small scenario. Finland has 5½ million people. Let us consider that 25% would get infected (with a maximum of, say, 50%), and 5% (max. 20%) would require care in a hospital. This would already mean, that we would suddenly have 70 000 (max 550 000) extra patients in the healthcare system, which has been “streamlined” for years. Very different scenario than having the same number of extra patients over the course of a year or a decade – one, which lays fertile ground to second-order effects. These include the impact on people, who wouldn’t have big problems under normal situations, due to having hospital care capacity readily available.

Finally: This is not fearmongering or a call for hysteria. Cold-headed rational decision making calls for taking precautions here. If you stock up so that you can self-quarantine yourself for 14 days in the case of getting ill, and do it gradually by buying little extra every time you go to the store anyway, you are making a good decision. Here’s one more figure by Luca, illustrating the point:

Image
Figure by Luca Dellanna; source

Relevant resources and references:

 

CARMA: Critical Appraisal of Research Methods and Analysis

This is the syllabus for my University of Helsinki course. Target audience is non-mathematical students in social sciences. The 2019 class consisted of social psychologists, social workers, sociologists and political scientists, so it’s quite a mishmash of topics I considered of high importance in life, research and everything.

UPDATE: Some people have been asking about how to cite this; OSF page with DOI, which includes the materials, is here

Critical Appraisal of Research Methods and Analysis (CARMA) – Evaluating and not getting fooled by data in scientific and practical research contexts

 


marta horror

(the violence is real, though)


 

Description: Research claims in news, science, and business can mislead people, either purposefully or inadvertently. How and why does this happen, and what mistakes, misconceptions and pitfalls should one avoid when evaluating data? This course will help participants assess data-based statements, and offer some tools to avoid getting fooled by them. It is meant for students who aspire to future careers, which involve undertaking, interpreting or commissioning research. This could include science in academic or other institutions, consumer/marketing research in business settings, evidence-based decision making as policy makers or journalists, among others. The course does not require specialising in quantitative methods, although basic familiarity can be useful.

Note: a lot of slides contain “animation” that doesn’t work if you watch the presentation on a scrolling mode instead of having one full slide on the screen at a time. So, download or zoom in.

I

  • The crisis of confidence in social and life sciences: State of affairs (4 September 2019) – slides

Learning objectives: Become acquainted with the recent developments regarding the so-called “replication crisis”.

        1. Replication crisis: how it all started (this time around).
        2. Medicine, you were supposed to be the best of us!
        3. Consequences of problematic practices.
        4. You’re not alone in misinterpreting p-values.

II

  • From questionable research practices and biased stories, to better evidence and/or decisions (11 September 2019) – slides

Learning objectives: Understand what the research community is doing to improve the quality of published research. Extrapolate to non-academic settings.

        1. Transparency and Openness Promotion (TOP) guidelines to fight bad science.
        2. Transforming publication practices with pre-prints
        3. Disentangling confirmatory and exploratory research.
        4. Tricky rule-of-thumb questions to ask when being presented research (1/2: “null findings”).

III

  • Magnificient mistakes and where to find them (18 September 2019) – slides

Learning objectives: Recognise some particular pitfalls in evidential statements. Understand that decisions in the field do not need to rely on correct predictive statements, let alone scientific evidence.

        1. Tricky rule-of-thumb questions to ask when being presented research (2/2: “statistically significant” findings).
        2. Ways tests can fail: Type I/II mistakes. Type M and Type S mistakes.
        3. The difference between evidence of absence and absence of evidence: Black Swans and the Turkey Problem.
        4. When you don’t need to be right: green lumber, and a first taste of convexity.
        5. Heuristics: Simple rules that make us smart.

IV

  • On interpreting data nudes instead of summary tables (25 September 2019) – slides

Learning objectives: Understand the rationale for visualising data, and what can be hidden when reporting summary statistics only. Learn to spot some common tricks used to visualise data in a favourable way to the presenter.

        1. A crude redux to evidence of absence.
        2. Data Nudes vs. Shitty Tables.
        3. The End of Average.
        4. What gets lost in looking at numbers alone: Uncertainty hidden in the absence of distributions.
        5. Demons with(in) axes: Slaying or summoning effects with presentation tricks.
        6. Dose-response effects masked by averages.

V

  • Complex systems and why they ruin everything straightforward (2 October 2019) – slides

Learning objectives: Become familiar with general features of so-called complex systems. Understand how they can be thought of in the context of practical interventions.

        1. Intro to complexity, and general features of complex systems. 
        2. Interaction vs. component dominant systems.
        3. Don’t camp at 1st order effects in dragon season.
        4. Navigating the Four Quadrants

VI

  • Never cross Heraclitus’ river, if it’s on average 1 meter deep: Interventions and their offspring (9 October 2019) – slides

Learning objectives: Understand the rationale behind interventions and experimenting/intervening in complex systems, as well as some limitations of big data.

        1. Change comes in a triad.
        2. Sales tricks to counter, use and abuse.
        3. Pathway thinking & complexity thinking in behaviour change science.
        4. Failures and unexpected effects of social interventions.
        5. When is it safe(r) to intervene?

VII

  • Dynamic/idiographic phenomena, and hidden assumptions (16 October 2019) – slides

Learning objectives: Describe the concepts of ergodicity and stationarity. Understand how they can mislead when not taken into account when e.g. assessing risks.

      1. Assumptions, schmassumptions; mind your foundations!
      2. Damned world not sitting still: Ergodicity & stationarity
      3. The idiographic approach to science
      4. The best map fallacy
      5. The precautionary principle for policy and interventions
      6. Frequency vs. consequences of being wrong: What matters more?
      7. Recap on the course: The Fourth Quadrant will find you, so better put your house in order

 

Student evaluations, comments, and feedback

Some students provided spontaneous feedback, and I everyone an opportunity to give evaluations. These are comprehensive answers i.e. there is no publication bias or selective reporting here!


Great course!! Even if the statistics are not exactly your thing, this course will give you a lot of useful information and a better look on the research field. I feel that I did benefit a lot from this course. The teaching was great and got me interested in the things that haven’t interest me before.

  • Anonymous student

Thank you Matti for this exiting and engaging course! I enjoyed substantially ambitious and well-prepared lectures. Even though I’m focusing on qualitative methodology in my own work, I found this course important and highly interesting.

  • Valter, a Social Work major

Can highly recommend the course. It shows that the teacher knows what he is talking about and is interested in the topics presented. The course can be a bit difficult but it’s teached in a fun way with concrete examples. Definitely not a boring course. The teacher is not boring either.

  • Anonymous student

A great course, I learned a lot. After the course I find two learning outcomes especially important; learning to better evaluate research, but especially learning to treat academy as an institution.

The course had A LOT of stuff, and was sometimes a bit difficult to follow and keep up the connections between topics. With some improving for the structure and creating clear bridges from one topic to another, this course will be even more beneficial.

  • Anonymous participant

This course is an eye opener, it makes you have a different but more clear understanding of research particularly and the world in general. The teaching style was excellent and the content was practical. Personally, I found it easy to relate to my field of study and i’m sure anyone else would find it very practical too, regardless of their research being qualitative or quantitative.

  • Selestino, Public Policy major

Highly-stimulating overview of a range of interrelated complex topics. Presented in an engaging manner and involving multiple interdisciplinary perspectives, this course can change how you think.

  • Antti, social psychology major

The course shows and discusses many issues of contemporary quantitative research methods and provides tools and tips how to become a better researcher. Not a critical course towards quantitiative research methods though, so don’t think about taking the course as an excuse for not learning the methods!

I would recommend the course for first year master students who have some prior knowledge of quantitative research methods. You don’t have to know how to use them though as there are no quantitative excercises in the course.

In summary, a great remedy for any traumas that you might have from trying to learn quantitative research methods. The course itself doesn’t heal the wounds though as those skills are not teached but the lecturer does provide great sources where you can hone those skills on your own time. Hopefully, a second course where those skills are excercised will soon follow.

  • Aku, a 6th year social psychology student

The course was well designed and teacher’s enthusiasm and expertise motivated me to do my best. Altough, I was suprised to notice that the evaluation of the course was based on the assessment scale of pass and fail. After investing a lot of time and effort in doing the assignments, it would have been instructive to know in what scale did I performed. Nonetheless, I learned a lot in this course and it opened new perspectives which I can utilize in my masters thesis.

  • Henna, a sociology major

Big thanks for the course! It was a very interesting and fun set even though it included a lot of new things to be learned in a fast phase. You are very skilled at explaining things very clearly and in an entertaining way by using (for some reason often fatal :D) examples. Not that entertainment is the most essential aspect of a course but at least it helps to concentrate and remember the content. The course had a good balance of lecturing and group discussions, albeit it wasn’t always easy to come up with discussion points since there was so much to take in. Still, it was nice to hear what materials others had been reading or what they remember from the lectures. You were also very good at taking and answering questions in many different ways to ensure everyone understood the underlying point, and I never felt that I could not ask something I did not understand no matter “simple” the question.

CARMA for the win!

  • Social Psychology Master’s Student

I would recommend this course for every student because it gives you many new viewpoints concerning validity of scientific methods.

  • Anonymous participant

Thank you for the lecture course, Matti. Your passion to these topics really shows with the enthusiasm you presented the numerous examples in class, with the blog and tweets and with the breathtaking slideshows sometimes consisting of 100 slides or more. I appreciate you bringing up the importance of open science and “hacks”, with which it is possible to take the other direction with science. And honestly, without all of the examples with which you tied the topics to real life, I probably wouldn’t have had the slightest idea what this course was about. The in-class discussions didn’t work that well, and I think that was because it was hard to tie our thoughts together (and present them in class) because everyone had done assignments in different topics. Discussing itself was alright, though. I liked that the at-home assignments balanced the theory-heavy lectures also, where we could think of the topics more concretely, if we wanted to. All in all, I think this was a rather “easy” course to complete, but I like that, since studying is done for our own sake and for our education, not for teachers. Like critical thinking. And as I stated in the last assignment, during this course I learned that before, I wasn’t at all as critical as I thought I was. So, thanks for that!

  • A 4th year student

Thank you for an excellently organised course! Your effort in the implementation and enthusiasm toward the subject, as well as goals aiming to expand students’ understanding were very visible during the course. This motivated to do the intensive work required by internalising difficult topics.

  • Henna, a sociology major

Thank you for this course, I really liked it! I feel that I now have a deeper understanding of research methodology and am able to do more critical judgments than before. I also wish there would be a second Carma course.

  • A social psychology major

Pathways and complexity in behaviour change research

These are slides of a talk given at the Aalto University Complex Systems seminar. Contrasts two views to changing behaviour; the pathway view and the complexity view, the latter being at its infancy. Presents some Secret Analysis Arts of Recurrence, which Fred Hasselman doesn’t want you to know about. Includes links to resources. If someone perchance saw my mini-moocs (1, 2) and happened to find them useful, drop me a line and I’ll make one of this.

Lifestyle factors are hugely relevant in preventing disease in modern societies; unfortunately people often fail in their attempts to change health behaviour – both their own, as well as that of others’. In recent years, behaviour change design has been conceived as a process where one identifies deficiencies in factors influencing the behaviours (commonly called “determinants”). Complexity thinking suggests putting emphasis on de-stabilisation instead.

The perspective taken here is mostly at the idographic level. At the time of writing, we have behaviour change methods to affect e.g. skills, perceived social norms, attitudes and so forth – but very little on general de-stabilisation of the motivational system as an important predictor of change.

Perspectives are welcome!

ps. Those of you to worry about brainwashing and freedom of thought: Chill. Stuff that powerful doesn’t really exist, and if it did, marketers would know about it and probably rule the world. [No, they don’t rule the world, I’ve been there]

pps. Forgot to put it in the slides, but this guy Merlijn Olthof will perhaps one day tweet about his work about destabilisation in psychotherapy contexts. Meanwhile, you can e.g. be his 10th Twitter follower, or keep checking his Google Scholar profile, as there’s a new piece coming out soon!

Why you should share Data Nudes instead of just Shitty Tables

This post summarises what I wanted to say with a recent paper published in Health Psychology and Behavioural Medicine, which includes an RMarkdown website supplement with code. Related slideshow and a video walkthrough is available here. Note: If it’s not obvious, These are my opinions as the first author, and may or may not be shared with collaborators who are nice people and surely wouldn’t use such foul language in public.

Some Problems in Summarising and Presenting Data

Many research reports include lots of variables, presented in tables comparing two or more groups, say an intervention and a control, or males and females. Readers often look at the means and standard deviations, looking for statistically significant differences between the two. What’s the problem?

1. It’s often not clear what significance even means, or whether some correction for multiple testing has been applied.

First of all, following the logic of Neyman-Pearson hypothesis testing, to keep error rate under the alpha level, one would have to correct for multiple testing, and it is unclear how many tests one should correct for when hypotheses are not pre-specified. Ignoring this – especially, where it is unclear how to heed the recommendation to justify one’s alpha level – error rates can become surprisingly high, much more than the conventionally assumed 5%.

2. In the absence of randomisation, increased sample size leads to detecting more and more tiny differences.

When there has not been randomisation (as in the case of genders or baseline cohort descriptions), the null hypothesis of zero difference is never true, and its rejection only depends on statistical power. We are pretty much never interested in whether the populations differ by any arbitrarily small amount on any of the presented variables. What usually matters, is whether this difference is large enough to make a difference, that is, how big is the effect size. Two caveats follow: Firstly, in behavioural field trials, your participants are rarely independent from each other, but come clustered in e.g. classrooms (students), hospitals (patients) or offices (9-to-5 mental patients). Secondly, you almost always need to randomise clusters instead of individuals (here‘s why), which gives statistical power a huge ass-whooping.

Not accounting for the multilevel structure of the data when calculating effect sizes inflates the standard errors, possibly even making zero effects appear as medium-sized ones. But it is not a trivial task to derive trustworthy effect sizes for nested data (Lai & Kwok 2016). Although some solutions exist, they have not yet been empirically validated for finite populations in the second or third levels, nor is there currently a straightforward software implementation available – to my knowledge, that is. Therefore, a sensible option may be to present the means with their corresponding confidence intervals, encouraging the readers to refrain from merely considering non-overlapping intervals between groups as dichotomous hypothesis tests. In Shitty Table 1 you can see how this is done. That seem clear to you? Don’t worry, there are alternatives!

shitty table 1
Shitty Table 1. Means and confidence intervals for lots of things. Click to enlarge. Source.

3. The shape of the distribution may matter much, much more than simple arithmetic mean.

Difference between two means is fun and neat, but only informative for approximately normal or symmetric distributions, which are not the norm in social and life sciences. See reading list in the end. But hey, surely everyone reports things like skewness and kurtosis? [Of course they don’t, and even if they did, a minority of social scientists could actually interpret the numbers.] Look at Shitty Table 2 to see for yourself, whether you consider this a good way to convey information.

shitty table 2
Shitty Table 2. Means, standard deviations and some distributional properties of a single variable in different educational tracks the participants were nested in. Nur = Practical nurse, HRC = Hotel, restaurant and catering studies, BA = Business and administration, IT = Business information technology. Click to enlarge. Source.

An aside as regards the means: Few individual participants are described by the group-level summary statistics. In fact, using Daniels’ definition of an ‘approximately average individual’ as falling in the middle 30% of the range of values, only 1.50% of participants can be considered ‘average’ on all of the primary outcome measures (see supplementary website, section https://git.io/fpOy1). Also see this and this blog post, as well as the papers listed in the end.

Data Wants to be Seen Naked

star trek android GIF

In our paper, we present some ways behaviour change researchers could visualise their data, discuss some limitations and provide links to R code. Many, many other dedicated sources do this better, so feel free to check out this or this, for example. A principle I particularly like is to, whenever possible, include the raw data in the visualisation. This is because in abstractions, I personally have a hard time keeping in mind that I’m dealing with individuals operating in the world (complex dynamic systems in complex dynamic systems), and the raw data tends to ground me to some reality.

pretty picture 1
Pretty Picture 1. Visualising the information in Shitty Table 1 with raw data. Click to enlarge.

Data-visualisation and data exploration techniques (e.g. network analysis) can help reveal the dynamics involved in complex multi-causal systems – a challenging task with Shitty Tables. Data visualisations are crucial supplements to large numerical tables of descriptive statistics. With visualisations, researchers can communicate large amounts of information – including the associated uncertainty – in an accessible format, without requiring extensive mathematical expertise from the reader. This is important for researchers who intend to build on previous results, and in the paper we argue that such practices may also reduce problems that have led to the recent loss of confidence in the reproducibility and replicability of research findings in social and life sciences. Fully open data sharing would be ideal, but this is not always possible due to privacy concerns and, at the time of writing, remains a lamentably rare practice. In addition, open data does not necessarily accommodate stakeholders with low technical expertise in data analysis and visualisation, such as clinicians, patients and policy makers.

The benefits of presenting complex data visually should encourage researchers to publish extensive analyses and descriptions as website supplements, which would increase the speed and quality of scientific communication, as well as help to address the crisis of reduced confidence in research findings.

pretty picture 2
Pretty Picture 2. Visualising the information in Shitty Table 2. Shows hours of accelerometer-measured moderate-to-vigorous physical activity for different educational tracks. Midpoints of diamonds indicate means, endpoints 95% credible intervals. Individual observations are presented under the density curves, with random scatter on the y-axis to ease inspection. Nur = Practical nurse, HRC = Hotel, restaurant and catering, BA = Business and administration, IT = Information and communications technology.

In Pretty Picture 2, looking closely you can observe that boys did more moderate-to-vigorous physical activity (x-axis is average daily hours) in every educational track. In spite of this, girls appeared more active when combining the educational tracks (shown as rows in the figure), because there is much more people in the practical nurse track, ,as well as those people being mostly girls. This is also known as the Simpson’s paradox, and is best investigated by visualising data.

pretty picture 3.PNG
Pretty Picture 3. See paper for elaboration.

Conventional approaches would have e.g. left the reader with an impression that the means of the multimodal or skewed variables (see Pretty Picture 1) are interpretable as central tendencies, and that the sample is homogenous (see Pretty Picture 2). Transparent and accessible sharing of data characteristics, analyses and analytical choices is imperative for increasing confidence in research findings; if nothing else, the elaborate supplements can act as a platform to present robustness tests and assumption explorations in.

pretty picture 4
Pretty Picture 4. See paper for elaboration.

Reading list

The paper described in this post:

  • Heino, M. T. J., Knittle, K., Fried, E., Sund, R., Haukkala, A., Borodulin, K., … Hankonen, N. (2019). Visualisation and network analysis of physical activity and its determinants: Demonstrating opportunities in analysing baseline associations in the let’s move it trial. Health Psychology and Behavioral Medicine, 7(1), 269–289. https://doi.org/10.1080/21642850.2019.1646136
  • Supplementary website: Link

On data visualisation:

  • Tay, L.Parrigon, S.Huang, Q., & LeBreton, J. M. (2016). Graphical descriptives a way to improve data transparency and methodological rigor in psychologyPerspectives on Psychological Science11(5), 692701

On hypothesis testing for non-prespecified comparisons:

  • de Groot AD. The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica. 2014;148:188–94.
  • Nosek BA, Ebersole CR, DeHaven AC, Mellor DT. The preregistration revolution. Proceedings of the National Academy of Sciences. 2018;201708274.

On effect sizes for cluster randomised situations:

  • Lai MHC, Kwok O-m. Estimating Standardized Effect Sizes for Two- and Three-Level Partially Nested Data. Multivariate Behavioral Research. 2016;51:740–56.
  • Lai MHC, Kwok O-m, Hsiao Y-Y, Cao Q. Finite population correction for two-level hierarchical linear models. Psychological methods. 2018;23:94.

On distributional shapes:

  • Choi, S. W. (2016). Life is lognormal! What to do when your data does not follow a normal distribution. Anaesthesia71(11), 1363-1366.
  • Saxon, E. (2015). Beyond bar chartsBMC Biology13(1), 60. doi: 10.1186/s12915-015-0169-6
  • Taleb, N. N. (2007). Black swans and the domains of statistics. The American Statistician61(3), 198-200.
  • van Rooij, M. M., Nash, B., Rajaraman, S., & Holden, J. G. (2013). A fractal approach to dynamic inference and distribution analysis. Frontiers in physiology, 4, 1.
  • Weissgerber, T. L.Garovic, V. D.Savic, M.Winham, S. J., & Milic, N. M. (2016). From static to interactive: Transforming data visualization to improve transparencyPLOS Biology14(6), e1002484. doi: 10.1371/journal.pbio.1002484
  • Weissgerber, T. L.Milic, N. M.Winham, S. J., & Garovic, V. D.(2015). Beyond bar and line graphs: time for a new data presentation paradigmPLOS Biology13(4), e1002128. doi: 10.1371/journal.pbio.1002128

On averages:

  • Daniels, G. S. (1952). The“average man”?Wright-Patterson Air Force Base, OHAir Force Aerospace Medical Research Lab.
  • Rose, T. (2016). The end of average: How to succeed in a world that values sameness. Penguin UK.
  • Rousselet, G. A., Pernet, C. R., & Wilcox, R. R. (2017). Beyond differences in means: Robust graphical methods to compare two groups in neuroscienceEuropean Journal of Neuroscience46(2), 17381748. doi: 10.1111/ejn.13610
  • Trafimow, D., Wang, T., & Wang, C. (2018). Means and standard deviations, or locations and scales? That is the question!New Ideas in Psychology503437. doi: 10.1016/j.newideapsych.2018.03.001