Suomen tie ulos kriisistä

Yaneer Bar-Yam is a complexity scientist, who has worked with and warned about pandemics for 15 years. His interview with Esa-Pekka Pälvimäki and Thomas Brand (in English) regarding the COVID-19 situation in Finland can be found here; these are some of my notes and video extracts.

Nämä ovat muistiinpanoni Esa-Pekka Pälvimäen ja Thomas Brandin toimittamasta kompleksisuustieteilijä ja pandemiatutkija Yaneer Bar-Yamin haastattelusta, jossa tämä kommentoi Suomen Koronavirustilannetta. Bar-Yam on yksi kompleksisuustieteiden isistä, ja äärimmäisen kunnioitettu tutkija. Ks. myös Suomenkielisiä työkaluja COVID-19 taisteluun.

Ensisijainen asia: On ymmärrettävä, että voimme päästä taudista eroon. Voimme lopettaa tämän taudin siinä, missä olemme lopettaneet muitakin tauteja: SARS, MERS ja Ebola eivät ole globaaleja riesoja. Tästä lisää myöhemmin.

“On maita, jotka ovat toimineet viisaasti ja päässeet taudista eroon; ne ovat [historian silmissä] voittajia. Suomi ei ole vielä siellä… jos Suomi haluaa päästä johtajien joukkoon, sen tulee toimia nopeasti ja voimakkaasti taudin hävittämiseksi.”

Kaksi tietä ulos kriisistä:

  1. Kahden viikon sulku johtaa siihen, että uudet tautitapaukset loppuvat lähes kokonaan. Niillä alueilla, joilla edelleen on tapauksia, sulkua tulee jatkaa. Hallituksen tulisi tukea kaupunkien ja muiden yhteisöjen päätösvaltaa siinä, että nämä voivat säädellä itse omia rajoituksiaan.
  2. Viiden viikon kansallinen sulku: On paljon maita, jotka ovat menestyneet COVID-taistossa kansallisen sulun avulla (esim. Etelä-Korea, Kreikka, Islanti, Luxemburg, Kroatia; ks. kuva). Tähän joukkoon kuuluvat maat voivat avata matkustusrajoituksia toistensa välillä. Toim. huom. yksikään maa ei ole peitonnut virusta ilman päättäväisiä vastatoimia.

Mikä on yhteisöjen rooli epidemian torjunnassa? Mikäli lainvoimainen ulkonaliikkumiskielto tai muut liikkumisrajoitteet ovat mahdottomia, pandemiavaste voidaan tehdä yhteisöissä; viestinä on, että olemme samassa veneessä ja kaikki haluavat päästä takaisin normaaliin – palataan siis normaaliin mahdollisimman nopeasti! Kaikki eivät suosituksia tietenkään tule noudattamaan, mutta jos suurin osa niin tekee, se riittää. Yleinen ja hyväksi havaittu tekniikka epidemian hallintaan on ovelta ovelle kulkeminen ja yhteisön jäsenten voinnin tiedusteleminen; ovatko he terveitä, sairaita, tarvitsevatko he jotakin? Tätä voi pari viikkoa tehdä yhteisön jäsen, Suomessa kenties taloyhtiön suojelu/turvallisuusvastaava?

Yhteisöissä, joissa tauti leviää poikkeuksellisen vahvasti, tulee puhua johtajille ja kertoa, että taudista ja sen tuottamasta kärsimyksestä voidaan päästä eroon. Ei ole mitään tärkeämpää kuin se, että yhteisöt saadaan ottamaan omistajuus ja vastuu omista jäsenistään. Heitä, heidän huoliaan ja ongelmiaan tulee kuunnella ja kysyä, kuinka heitä voitaisiin parhaalla tavalla auttaa.

 

Käynnistyykö leviäminen väistämättä uudestaan, jos tautia kantava henkilö pääsee tartunnoista vapaalle alueelle? Uudet tartunta-aallot eivät ole tarpeellisia. Tartuntatauteja on hävitetty ennenkin, ja samoin voidaan Koronaviruskin hävittää: paikallisesti ja globaalisti. Kyse on valinnasta. Esim. 1-3 tapausta voidaan aina pysäyttää kontaktijäljityksen ja altistuneiden eristämisen avulla; voimme myös toimillamme vaikuttaa siihen, että tapausten ilmaantuminen on hyvin epätodennäköistä. Mutta jos tapauksia on esim. kymmenen, tarvitaan järeämpiä toimia.

Palaako tauti aina ja ikuisesti ulkomailta, kunnes rokotus on saatavilla; eihän minkään maan talous kestä niin pitkiä rajoituksia? Ei, taudin hävittäminen saadaan tehtyä viikoissa. Suomessa se saataisiin poistettua monista paikoista kahdessa viikossa, toisissa kolmessa tai useammassa. Viidessä-kuudessa viikossa se katoaisi kaikkialta. Tähän liittyy kiinnostava harha: taudin alkuvaiheessa ajateltiin taudittoman maailman kestävän ikuisesti, ja nyt ajatellaan taudin kestävän ikuisesti. Ei – normaalitila ei kestä ikuisesti, eikä poikkeustila kestä ikuisesti. SARS ja MERS eivät päätyneet nekään kiertämään maailmaa ikuisesti.

 

Entä laumaimmuniteetti? Laumaimmuniteetin hankinnan kustannus on valtava, eikä ole selvää, että se toimisi. Jos emme tee suurempia toimia, yritämme pitää tautitapaukset alhaalla ja odotamme rokotetta sekä laumaimmuniteettia, siinä voi mennä vuosia, ja se voi maksaa 250 000 henkeä.

Keskustelussa sivuttiin seitsemää ensimmäistä kohtaa 9-kohtaisesta toimintaohjelmasta (ks. COVID-19: How to Win, sekä suomennetut ohjeet: COVID-19 -taistelusuositukset poliittisille päätöksentekijöille | Miksi viiden viikon lockdown voi pysäyttää COVID19-epidemian? | Milloin voimme jälleen avata yhteiskunnan?)

  1. Kaikkien – yritysten, yhteisöjen ja hallituksen – saaminen mukaan ponnistukseen.
  2. Sulku (lockdown): Fyysisen etäisyyden (6–9 metriä; 2m ei riitä) pitäminen, tartuntojen rajoittaminen perheryhmissä (positiiviseksi testatut henkilöt lähetetään karanteeniin esim. hotelliin oman asunnon sijaan).
  3. Tapausten tunnistaminen ja eristäminen (miellyttäviin paikkoihin, esim. hotelleihin) ajoissa.
  4. Kasvosuojainten käyttäminen, erityisesti välttämättömissä palveluissa.
  5. Edes jonkinasteiset matkustusrajoitukset. Liikkumisen rajoittaminen tarpeellisiksi nähtyihin palveluihin on parempi kuin se, että kaikki liikkuvat mielin määrin, mikä tuo tartuntatapaukset paikkoihin joissa niitä ei välttämättä muuten olisi.
  6. Välttämättömien palveluiden käytön saaminen turvalliseksi. Turvalliset työtilat, etätyömahdollisuudet, ruokakauppojen kotiin-/kadullekuljetukset, jne.
  7. Laajamittainen testaus, jotta tiedetään missä tarvitaan lisärajoituksia ja missä rajoituksia voidaan höllentää. Tietokonetomografiaa voidaan käyttää testaamisen nostamiseen uudelle tasolle; se tuottaa hyvin vähän vääriä negatiivisia havaintoja.

 

“Kuka tahansa, joka nykyään sanoo, ettei ole olemassa informaatiota, jonka perusteella kasvosuojainten voi sanoa olevan hyödyllisiä taudin leviämisen kannalta, on sokea. He pitävät maskia suun ja nenän sijaan silmillään. Näyttö on olemassa, tieteellinen ymmärrys on olemassa; tämän viestin pitää olla selkeä.”

– Yaneer Bar-Yam (39:34)

Valtava virhe vastatoimissa on, että ajattelemme ja toimimme kuin tämä olisi influenssa. Mutta siitä ei ole kyse; voimme oppia enemmän vakavista tartuntatautitapauksista selvinneiltä mailta, kuin voimme menneestä toiminnastamme vanhojen perus-influenssojen parissa. Esimerkiksi Ebola on tullut paikallisen häviämisen jälkeen takaisin vain siksi, että se on palannut eläinten kautta ihmisiin.

 

Vaihtoehtoinen strategia ei ole Flatten the Curve, ts. “Pidä tapausmäärät alhaalla ja odottele”.

 

Tässä vielä video kokonaisuudessaan:

 

Complexity methods for behavioural sciences: YouTube channel and resources

In 2019 I attended an exciting summer school; Complexity Methods for Behavioural Science: A Toolbox for Studying Change. Later, we – that is, the University of Helsinki Behaviour Change and Wellbeing Group – had the opportunity to invite Fred Hasselman, who devised the course, to Finland. He gave an overview talk as well as a 3-day workshop, which I recorded with varying success. This page collates resources regarding the course.

For all the recordings, see our YouTube channel. There are two playlists; one for short snippets and another one for full-length lectures. Here are some tweets on the course, with links to further resources. For additional slides, see here. See the end of the post for literature!

  • Lecture 0 (video, slides)Overview of complexity science and its applications in behavioural sciences. Also see shorter snippets on ergodicity, interaction- vs. component-dominant dynamics, and my interview with Fred.
  • Lecture 1 (video, slides [1-25])Introduction to Complexity Science: Dissipative systems, Self-Organization, Self-Organised Criticality (SOC), Phase transition, Interaction Dominant Dynamics, Emergence, Synchronisation.
  • Lecture 2 (video, slides [26-74])Introduction to the mathematics of change: Logistic Map, Return Plot, Attractors. [The beginning of the lecture was cut due to camera problems; please find a great introduction to the logistic map here.]
  • Lecture 3 (video, slides)Basic Time Series Analysis: Autocorrelation Function, Sample Entropy, Relative Roughness.
  • Lecture 4 (video, slides [34 onwards, see also this, this and this]) – Detecting (nonlinear) structure in time series: Fractal Dimension, Detrended Fluctuation Analysis, Standardised Dispersion Analysis.
  • Lecture 5 (video, slides [1-16])Quantifying temporal patterns in unordered categorical time series data: Categorical Auto-Recurrence Quantification Analysis (RQA).
  • Lecture 6 (video, slides [17-52])Quantifying temporal patterns in continuous time series data: Continuous Auto-Recurrence Quantification Analysis, Phase-space reconstruction.
  • Lecture 7 (video, slides [52-70]) – Recurrence Quantification Analysis in practice: Data preparation for RQA, “General recipe” (i.e. RQA workflow), lagged/windowed RQA, RQA in detecting cognitive phase transitions, RQA in neural imaging.
  • Lecture 8 (video, slides) – Multivariate Recurrence Quantification Analysis: Cross-Recurrence Quantification Analysis (CRQA), applications in interpersonal synchronisation dynamics (leader-follower behaviour), Diagonal Cross-Recurrence Profiles (DCRP).
  • Lecture 9 (video, slides) – Multivariate Time Series Analysis – Dynamic Complexity & Phase Transitions in Psychology: Self-ratings as a research tool, the importance of sampling frequency, dynamic complexity as an early warning signal in psychopathology.
  • Lecture 10 (video, slides [1-37]) – Introduction to graph theory, with applications of network science: Complex networks, hyperset theory, network-based complexity measures, small-world networks.
  • Lecture 11 (video, slides [38-80]) – Multiplex recurrence networks for non-linear multivariate time series analysis: Recurrence networks, change profiles of ecological momentary assessments as an alternative to raw scores. Also see this paper!

 

Matti spiral

Literature:

Three recent papers directly related to the course’s topics:

Hasselman, F., & Bosman, A. M. T. (2020). Studying Complex Adaptive Systems with Internal States: A Recurrence Network Approach to the Analysis of Multivariate Time Series Data Representing Self-Reports of Human Experience. Frontiers in Applied Mathematics and Statistics, 6. https://doi.org/10.3389/fams.2020.00009

Heino, M. T. J., Knittle, K. P., Noone, C., Hasselman, F., & Hankonen, N. (2020). Studying behaviour change mechanisms under complexity [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/fxgw4

Olthof, M., Hasselman, F., & Lichtwarck-Aschoff, A. (2020). Complexity In Psychological Self-Ratings: Implications for research and practice [Preprint]. PsyArXiv. https://doi.org/10.31234/osf.io/fbta8

More resources on complexity:

  1. Mathews, K. M., White, M. C., & Long, R. G. (1999). Why Study the Complexity Sciences in the Social Sciences? Human Relations, 52(4), 439–462. https://doi.org/10.1023/A:1016957424329 [INTRO COMPLEXITY SCIENCE]
  2. Richardson, M. J., Kallen, R. W., & Eiler, B. A. (2017). Interaction-Dominant Dynamics, Timescale Enslavement, and the Emergence of Social Behavior. In Computational Social Psychology (pp. 121–142). New York: Routledge. [INTERACTION-DOMINANCE]
  3. Molenaar, P. C., & Campbell, C. G. (2009). The new person-specific paradigm in psychology. Current directions in psychological science, 18(2), 112-117. [ERGODICITY]
  4. Kello, C. T., Brown, G. D., Ferrer-i-Cancho, R., Holden, J. G., Linkenkaer-Hansen, K., Rhodes, T., & Van Orden, G. C. (2010). Scaling laws in cognitive sciences. Trends in cognitive sciences, 14(5), 223-232. [SCALING PHENOMENA]
  5. Lewis, M. D. (2000). The promise of dynamic systems approaches for an integrated account of human development. Child development, 71(1), 36-43. [STATE SPACE, DYNAMICS]
  6. Olthof, M., Hasselman, F., Strunk, G., van Rooij, M., Aas, B., Helmich, M. A., … Lichtwarck-Aschoff, A. (2019). Critical Fluctuations as an Early-Warning Signal for Sudden Gains and Losses in Patients Receiving Psychotherapy for Mood Disorders. Clinical Psychological Science, 2167702619865969. [DYNAMIC COMPLEXITY]
  7. Olthof, M., Hasselman, F., Strunk, G., Aas, B., Schiepek, G., & Lichtwarck-Aschoff, A. (2019). Destabilization in self-ratings of the psychotherapeutic process is associated with better treatment outcome in patients with mood disorders. Psychotherapy Research, 0(0), 1–12. https://doi.org/10.1080/10503307.2019.1633484 [DYNAMIC COMPLEXITY]
  8. Richardson, M., Dale, R., & Marsh, K. (2014). Complex dynamical systems in social and personality psychology: Theory, modeling and analysis. In Handbook of Research Methods in Social and Personality Psychology (pp. 251–280). [INTRO COMPLEXITY SCIENCE – Social and personality psychology]
  9. Wallot, S., & Leonardi, G. (2018). Analyzing Multivariate Dynamics Using Cross-Recurrence Quantification Analysis (CRQA), Diagonal-Cross-Recurrence Profiles (DCRP), and Multidimensional Recurrence Quantification Analysis (MdRQA) – A Tutorial in R. Frontiers in Psychology, 9. https://doi.org/10.3389/fpsyg.2018.02232 [MULTIDEMINSIONAL RQA]
  10. Webber Jr, C. L., & Zbilut, J. P. (2005). Recurrence quantification analysis of nonlinear dynamical systems. In Tutorials in contemporary nonlinear methods for the behavioral sciences (pp. 26–94). Retrieved from http://www.saistmp.com/publications/spiegorqa.pdf [RQA]
  11. Marwan, N. (2011). How to avoid potential pitfalls in recurrence plot based data analysis. International Journal of Bifurcation and Chaos, 21(04), 1003–1017. https://doi.org/10.1142/S0218127411029008 [RQA parameter estimation]
  12. Boeing, G. (2016). Visual Analysis of Nonlinear Dynamical Systems: Chaos, Fractals, Self-Similarity and the Limits of Prediction. Systems, 4(4), 37. https://doi.org/10.3390/systems4040037 [LOGISTIC MAP, DERTERMINISTIC CHAOS]
  13. Kelty-Stephen, D. G., Palatinus, K., Saltzman, E., & Dixon, J. A. (2013). A Tutorial on Multifractality, Cascades, and Interactivity for Empirical Time Series in Ecological Science. Ecological Psychology, 25(1), 1–62. https://doi.org/10.1080/10407413.2013.753804 [MULTI-FRACTAL ANALYSIS]
  14. Kelty-Stephen, D. G., & Wallot, S. (2017). Multifractality Versus (Mono-) Fractality as Evidence of Nonlinear Interactions Across Timescales: Disentangling the Belief in Nonlinearity From the Diagnosis of Nonlinearity in Empirical Data. Ecological Psychology, 29(4), 259–299. https://doi.org/10.1080/10407413.2017.1368355 [(MULTI-)FRACTAL ANALYSIS]
  15. Hawe, P. (2015). Lessons from Complex Interventions to Improve Health. Annual Review of Public Health, 36(1), 307–323. https://doi.org/10.1146/annurev-publhealth-031912-114421
  16. Rickles, D., Hawe, P., & Shiell, A. (2007). A simple guide to chaos and complexity. Journal of Epidemiology & Community Health, 61(11), 933–937. https://doi.org/10.1136/jech.2006.054254. [INTRO COMPLEXITY SCIENCE – Public health]
  17. Pincus, D., Kiefer, A. W., & Beyer, J. I. (2018). Nonlinear dynamical systems and humanistic psychology. Journal of Humanistic Psychology, 58(3), 343–366. https://doi.org/10.1177/0022167817741784.  [INTRO COMPLEXITY SCIENCE – Positive psychology]
  18. Gomersall, T. (2018). Complex adaptive systems: A new approach for understanding health practices. Health Psychology Review, 0(ja), 1 – 34. https://doi.org/10.1080/17437199.2018.1488603. [INTRO COMPLEXITY SCIENCE – Health psychology]
  19. Nowak, A., & Vallacher, R. R. (2019). Nonlinear societal change: The perspective of dynamical systems. British Journal of Social Psychology, 58(1), 105-128. https://doi.org/10.1111/bjso.12271. [INTRO COMPLEXITY SCIENCE – Societal change]
  20. Carello, C., & Moreno, M. (2005). Why nonlinear methods. In Tutorials in contemporary nonlinear methods for the behavioral sciences (pp. 1–25). Retrieved from https://nsf.gov/pubs/2005/nsf05057/nmbs/chap1.pdf [INTERACTION DOMINANCE, ERGODICITY]
  21. Liebovitch, L. S., & Shehadeh, L. A. (2005). Introduction to fractals. In Tutorials in contemporary nonlinear methods for the behavioral sciences (pp. 178–266). Retrieved from https://nsf.gov/pubs/2005/nsf05057/nmbs/chap5.pdf [FRACTAL ANALYSIS]

 

 

Pathways and complexity in behaviour change research

These are slides of a talk given at the Aalto University Complex Systems seminar. Contrasts two views to changing behaviour; the pathway view and the complexity view, the latter being at its infancy. Presents some Secret Analysis Arts of Recurrence, which Fred Hasselman doesn’t want you to know about. Includes links to resources. If someone perchance saw my mini-moocs (1, 2) and happened to find them useful, drop me a line and I’ll make one of this.

Lifestyle factors are hugely relevant in preventing disease in modern societies; unfortunately people often fail in their attempts to change health behaviour – both their own, as well as that of others’. In recent years, behaviour change design has been conceived as a process where one identifies deficiencies in factors influencing the behaviours (commonly called “determinants”). Complexity thinking suggests putting emphasis on de-stabilisation instead.

The perspective taken here is mostly at the idographic level. At the time of writing, we have behaviour change methods to affect e.g. skills, perceived social norms, attitudes and so forth – but very little on general de-stabilisation of the motivational system as an important predictor of change.

Perspectives are welcome!

ps. Those of you to worry about brainwashing and freedom of thought: Chill. Stuff that powerful doesn’t really exist, and if it did, marketers would know about it and probably rule the world. [No, they don’t rule the world, I’ve been there]

pps. Forgot to put it in the slides, but this guy Merlijn Olthof will perhaps one day tweet about his work about destabilisation in psychotherapy contexts. Meanwhile, you can e.g. be his 10th Twitter follower, or keep checking his Google Scholar profile, as there’s a new piece coming out soon!

Randomised experiments (mis?)informing social policy in complex systems

In this post, I vent about anti-interdisciplinarity, introduce some basic perspectives of complexity science, and wonder whether decisions on experimental design actually lead us to end up in a worse place than where we were, before we decided to use experimental evidence to inform social policy.

People in our research group recently organised a symposium, Interdisciplinary perspectives on evaluating societal interventions to change behaviour (talks watchable here), as part of a series called Behaviour Change Science & Policy (BeSP). The idea is to bring together people from various fields from philosophy to behavioural sciences, medicine and beyond, in order to better tackle problems such as climate change and lifestyle diseases.

One presentation touched upon Finland’s randomised controlled trial to test the effects of basic income on employment (see also report on first year results). In crude summary, they did not detect effects of free money on finding employment. (Disclaimer: They had aimed for 80% statistical power, meaning that if all your assumptions regarding the size of the effect are correct, in the long term, 20% of the time you’d get no statistically significant effect in spite of there being a real effect.)

During post-symposium drinks, I spoke with an economist about the trial. I was wondering, how come they used individual instead of cluster randomisation – randomising neighbourhoods, for example. The answer was resource constraints; much larger sample sizes are needed for the statistics to work. To me it seemed clear, that it’s a very different situation if one person in a network of friends got free money, as compared to if everyone did. The economist wondered: “How come there could be second-order effects when there were no first-order effects?” The conversation took a weird turn. Paraphrasing:

Me: Blahblah compelling evidence from engineering and social sciences to math and physics that “more is different”, i.e. phenomena play out differently depending on the scale at consideration… blahblah micro-level interactions create emergent macro-level patterns blahblah.

Economist: Yeah, we’re not having that conversation in our field.

Me: Oh, what do you mean?

Economist: Well, those are not things discussed in our top journals, or considered interesting subjects to research.

Me: I think they have huge consequences, and specifically in economics, this guy in Oxford just gave a presentation on what he called “Complexity economics“. He had been doing it for some decades already, I think he originally had a physics background…

Economist: No thanks, no physicists in my economics.

Me: Huh?

Economist: [exits the conversation]

Now, wasn’t that fun for a symposium on interdisciplinary perspectives.

I have a lot of respect for the mathematical prowess of economists and econometricians, don’t get me wrong. One of my favourites is Scott E. Page, though I only know him due to an excellent course on complexity (also available as an audio book). I do probably like him, because he breaks out of the monodisciplinary insulationist mindset economists are often accused of. Page’s view of complexity actually relates to our conversation. Let’s see how.

First off, he describes complexity (and most social phenomena of interest) as arising from four factors, which can be thought as tuning knobs or dials. Complexity arises, when each dial is not tuned into either of the extremes, which is where equilibria arise, but somewhere in the middle. And complex systems tend to reside far from equilibrium, permanently.

To dig more deeply into how the attributes of interdependence,
connectedness, diversity, and adaptation and learning generate
complexity, we can imagine that each of these attributes is a dial that
can be turned from 0 (lowest) to 10 (highest).

Scott E. Page

  • Interdependence means the extent of how much one person’s actions affect those of another’s. This dial ranges from complete independence, where one person’s actions do not affect others’ at all, to complete dependence, where everyone observes and tries to perfectly match all others’ actions. In real life, we see both unexpected cascades (such as the US decision makers’ ethanol regulations, leading to the Arab Spring), as well as some, but never complete, independence – that is, manifestations that do not fit into either extreme of the dial, but lie somewhere in between.
  • Connectedness refers to how many other people a person is connected to. The extremes range from people living in a cabin in the woods all alone, to hypersocial youth living in Instagram trying to keep tabs on everyone and everything. The vast majority of people lie somewhere in between.
  • Diversity is the presence of qualitatively different types of actors: If every person is a software engineer, mankind is obviously doomed… But the same happens if there’s only one engineer, one farmer etc. Different samples of real-world social systems (e.g. counties) consist of intermediate amounts of diversity, lying somewhere in between.
  • Adaptation and learning refer to the extent of the actors’ smartness. This ranges from following simple, unchanging rules, to being perfectly rational and informed, as assumed in classical economics. In actual decision making, we see “bounded rationality”, reliance on rules of thumb and tradition, as well as both optimising and satisficing behaviours – the “somewhere in between”.

The complexity of complex systems arises, when diverse, connected people interact on the micro-level, and by doing so produce “emergent” macro-level states of the world, to which they adapt, creating new unexpected states of the world.

You might want to read that one again.

Back to basic income: When we pick 2000 random individuals around the country and give them free money, we’re implicitly assuming they are not connected to any other people, and/or that they are completely independent the actions of others’. We’re also assuming that they are either the same, or that it’s not interesting that they are of different types. And so forth. If we later compare their employment data to that of those who were not given basic income, the result we get is an estimate of the causal effect in the population, if all assumptions would hold.

But consider how these assumptions may fail. If the free money was perceived as a permanent thing, and given to people’s whole network of unemployed buddies, it seems quite plausible that they would adapt their behaviour as a response of the dynamics of their social network changing. This might even be different in cliques of certain people, who might use the safety net of basic income to collectively found companies and take risks, and cliques of other people, who might alter their daily drinking behaviour to match the costs with the predictable income – for better or worse. But when you randomise individually and ignore how people cluster in networks, you’re studying a different thing. Whether it’s an interesting thing or a silly thing, is another issue.

Now, it’s easy to come up with these kinds of assumption-destroying scenarios, but a whole different ordeal to study them empirically. We need to simplify reality in order to deal with it. The question is this: How much of an abstraction can a map (i.e. a model in a research study, making those simplified assumptions) be, in order to still represent reality adequately? This is also an ontological question, because if you take the complexity perspective seriously, you say bye-bye to the kind of thinking that allows you to dream up predictable effects a button-press (such as a policy change) has over the state of a system. People who act in—or try steering—complex systems, control almost nothing but influence almost everything.

An actor in a complex system controls almost nothing but influences almost everything.

Scott E. Page

Is some information, some model, still better than none? Maybe. Maybe not. In Helsinki, you’re better off without a map, than with a map of Stockholm – the so-called “Best map fallacy” (explained here in detail). Rare, highly influential events drive the behaviour of complex systems: the Finnish economy was not electrified by average companies starting to sell more, but by Nokia hitting the jackpot. And these events are very hard, if not impossible, to predict✱.

Ok, back to basic income again. I must say that the people who devised the experiment were not idiots, and included e.g. interviews of people to get some idea about unexpected effects. I think that this type of an approach is definitely necessary when dealing with complexity, and all social interventions should include qualitative data in their evaluation. But, again, unless the unemployed don’t interact, with randomisation done individually you’re studying a different thing than when it’s done in clusters. I do wonder if it would have been possible to include some matched clusters, to see if any qualitatively different dynamics take place, when you give basic income to a whole area instead of randomly picked individuals within it.

Complex systems organizational map.jpg
The society is a complex system, and must be studied as such. Figure: Hiroki Sayama (click to enlarge)

But, to wrap up this flow of thought, I’m curious if you think it is possible to randomise a social intervention individually AND always keep in mind that the conclusions are only valid if there are no interactions between people’s behaviour and that of their neighbours. Or is it inevitable that that the human mind smoothes out the details?

Importantly: Is our map better now, than it was before? Will this particular experiment go in history as—like the economist stated in “there were no first-order effects”—basic income not having any effect on job seeking? (remember, aim was only 80% statistical power). Lastly, I want to say I consider it unforgiveable to only work within one discipline and disregard the larger world you’re operating in: When we bring science to policy making, we must be doubly cautious of the assumptions our conclusions stand on. Luckily, transparent scientific methodology allows us to be explicit about them.

Let me hear your thoughts, and especially objections, on Twitter, or by email!

✱ One solution is to harness convexity, which can be oversimplified like this:

  1. Unpredictable things will happen, and they will make you either better or worse off.
  2. Magnitude of an event is different from it’s effect on you, i.e. there are huge events that don’t impact you at all, and small events that are highly meaningful to you. Often that impact depends on the interdependence and connectedness dials.
  3. To an extent, you can control the impact an event has on you.
  4. You want to control exposure in such a way, that surprise losses are bounded, while surprise gains are as limitless as possible.

Complexity considerations for intervention (process) evaluation

For some years, I’ve been partly involved in the Let’s Move It intervention project, which targeted dysfunctional physical activity and sedentary behaviour patterns of older adolescents, by affecting their school environment as well as social and psychological factors.

I held a talk at the closing seminar; it was live streamed and is available here (on stage starting from about 1:57:00 in the recording). But if you were there, or are otherwise interested in the slides I promised, they are now here.

For a demonstration of non-stationary processes (which I didn’t talk about but which are mentioned in these slides), check out this video and an experimental mini-MOOC I made. Another blog post touching on some of the issues is found here.

 

blogiin kuva

Misleading simplifications and where to find them (Slides & Mini-MOOC 11min)

The gist: to avoid getting fooled by them, we need to name our simplifying assumptions when modeling social scientific data. I’m experimenting with this visual approach to delivering information to those who think modeling is boring; feedback and improvement suggestions very welcome! [Similar presentation with between-individual longitudinal physical activity networks, presented at the Finnish Health Psychology conference: here]

I’m not as smooth as those talking heads on the interweb, so you may want just the slides. Download by clicking on the image below or watch at SlideShare.

SLIDE DECK:

misleading assumptions 1st slide

Mini-MOOC:

 

Note: Jan Vanhove thinks we shouldn’t  become paranoid with model assumptions; check his related blog post here!

The secret life of (complex dynamical) habits

It was recently brought to my attention that there exist such things as time and context, the flow of which affects human affairs considerably. Then there was this Twitter conversation about what habits actually are. In this post, I try to make sense of how to view health behavioural habits from the perspective of dynamical systems / complexity theory. I mostly draw from this article.

Habits are integral to human behaviour, and arguably necessary to account for in intervention research 1–3. Gardner 1 proposes a definition of habit as not a behaviour but “a process by which a stimulus generates an impulse to act as a result of a learned stimulus-response association”. Processes being seldom stable for all eternity, a complex dynamical systems perspective would propose some consequences of this definition.

What does it mean, when a process—such as habit—is stable? One way of conceiving this is considering the period of stability as a particular state a system can be in, while being subject to change. Barrett 4 proposes four features of dynamic system stability, in which a system’s states depend on the interactions among its components, as well as the system’s interactions with its environment.

corpus clock
Corpus Clock of Cambridge, where I’m writing this. The clock behaves chaotically so that it’s accurate every five minutes. A time-eating locust on top reminds us that neither habits, nor other human endeavours, escape this passage. Photo: Jim Linwood

First of all, stability always has a time frame, and stabilities at different time frames (such as stability over a month and a year) are interdependent. We ought to consider, how these time scales interact. For example, some factors which determine one’s motivation to go to the gym, such as mood, fluctuate on the scale from minutes to hours. Others may fluctuate on the daily level, and can be influenced by how much one slept the previous night or how stressful one’s workday was, whereas others fluctuate weekly. Then again, some—which increasingly resemble dispositions or personality factors—may be quite stable across decades. When inspecting a health behaviour, we ought to be looking at minimum the process which takes place on a time scale one level faster, and one lever slower than the one we are purportedly interested in 4. For example, how do daily levels of physical activity relate to weekly ones, and how do montly fluctuations affect the weekly fluctuations? Health psychologists could also classify each determinant of a health behaviour, based on the time scale it is thought to operate on. For example, if autonomous forms of motivation 5 seem to predict physical activity quite well cross-sectionally, we could attempt to measure it for a hundred days and investigate what the relevant time-scales of fluctuations are, in relation to those of the target behaviour. Such an exercise could also be helpful for deciding on the sampling frequency of experience sampling studies.

Second, processes in systems such as people have their characteristic attractor landscapes, and these landscapes can possibly be spelled out, along with the criteria associated with them. By attractors I mean here behaviours a person is drawn to, and an attractor landscape is the conglomerate of these behaviours. The cue-structure of the behaviours can be quite elaborate. For example, a person may smoke only, when they have drank alcohol (1) in a loud environment (2), among a relatively large group (3) of relatively unfamiliar people (4), one or two of whom are smokers (5); a situation where it is easier to have a private conversation if one joins another to go out for a cigarette. This highlights how the process of this person’s smoking habit can be very stable (mapping to the traditional conception of “habitual”), while also possibly being highly infrequent.

Note: Each of the aforementioned conditions for this person to smoke are insufficient by themselves, although all are needed to trigger smoking in this context. As a whole, they are sufficient to cause the person to smoke, but not always necessarily needed, because the person may smoke in some more-or-less limited other conditions, too. These conditions can also be called INUS (referring to Insufficient but Necessary criteria of an Unnecessary but Sufficient context for the behaviour) 6. Let that sink in a bit. As a corollary, if a criterion really is necessary, it may be an attractive target for intervention.

Third, the path through which change happens matters, a lot. Even when all determinants of behaviour are at a same value, the outcome may be very different depending on previous values of the outcome. This phenomenon is known as hysterisis, and it has been observed in various fields from physics (e.g. the form of a magnetic field depends on its past) to psychology (e.g. once a person becomes depressed due to excess stress, the stress level must be much lower to switch back to the normal state, than was needed for the shift to depression; 7). As a health behaviour example, just imagine how much easier it is to switch from a consistent training regime to doing no exercise at all, compared to doing it the other way around. Another way to think about is to consider that systems are “influenced by the residual stability of an antecedent regime” 4. As a consequence of stability being “just” a particular type of a path-dependent dynamic process 4,8, we need to consider the history leading up to the period where a habit is active. This forces investigators to consider attractor patterns and sensitivity to initial conditions: When did this stable (or attractor) state come about? If interactions in a system create the state of the system, which bio-psycho-social interactions are contributing to the stable state in question?

Fourth, learning processes such as those happening due to interventions usually affect a cluster of variables’ stabilities, not just one of them. To change habits, we naturally need to consider which changeable processes should be targeted, but it is probably impossible to manipulate these processes in isolation. This has been dubbed the “fat finger problem” (Borsboom 2018, personal communication); trying to change a specific variable, like attempting to press a specific key on the keyboard with gloves on, almost invariably ends up affecting neighbouring variables. Our target is dynamic and interconnected, often calling for coevolution of the intervention and the intervened.

It is obvious that people can relapse to their old habitual (attractor) behaviour after an intervention, and likely that extinction, unlearning and overwriting of cue-response patterns can help in breaking habits, whatever the definition. But the complex dynamics perspective puts a special emphasis on understanding the time scale and history of the intervenable processes, as well as highlighting the difficulty of changing one process while holding others constant, as the classical experimental setup would propose.

I would be curious of hearing thoughts about these clearly unfinished ideas.

  1. Gardner, B. A review and analysis of the use of ‘habit’ in understanding, predicting and influencing health-related behaviour. Health Psychol. Rev. 9, 277–295 (2015).
  2. Wood, W. Habit in Personality and Social Psychology. Personal. Soc. Psychol. Rev. 21, 389–403 (2017).
  3. Wood, W. & Rünger, D. Psychology of Habit. Annu. Rev. Psychol. 67, 289–314 (2016).
  4. Barrett, N. F. A dynamic systems view of habits. Front. Hum. Neurosci. 8, (2014).
  5. Ryan, R. M. & Deci, E. L. Self-determination theory: Basic psychological needs in motivation, development, and wellness. (Guilford Publications, 2017).
  6. Mackie, J. L. Causes and Conditions. Am. Philos. Q. 2, 245–264 (1965).
  7. Cramer, A. O. J. et al. Major Depression as a Complex Dynamic System. PLoS ONE 11, (2016).
  8. Roe, R. A. Test validity from a temporal perspective: Incorporating time in validation research. Eur. J. Work Organ. Psychol. 23, 754–768 (2014).

 

Is it possible to unveil intervention mechanisms under complexity?

In this post, I wonder what complex systems, as well as the nuts and bolts of mediation analysis, imply for studying processes of health psychological interventions.

Say we make a risky prediction and find an intervention effect that replicates well (never mind for now that replicability is practically never tested in health psychology). We could then go on to investigating boundary conditions and intricacies of the effect. What’s sometimes done is a study of “mechanisms of action”, also endorsed by the MRC guidelines for process evaluation (1), as well as the Workgroup for Intervention Development and Evaluation Research (WIDER) (2). In such a study, we investigate whether the intervention worked as we thought it should have worked (in other words, to test the program theory; see previous post). It would be spectacularly useful to decision makers, if we could disentangle the mechanisms of the intervention; “by increasing autonomy support, autonomous motivation goes up and physical activity ensues”. But attempting to evaluate this opens a spectacular can of worms.

Complex interventions include multiple interacting components, targeting several facets of a behaviour on different levels of the environment the individual operates in (1). This environment itself can be described as a complex system (3). In complex, adaptive systems such as the society or a human being, causality is thorny issue (4): Feedback loops, manifold interactions between variables over time, path-dependence and sensitivity to initial conditions make it challenging at best to state “a causes b” (5). But what does it even mean to say something causes something else?

Bollen (6) presents three conditions for causal inference: isolation, association and direction. Isolation means that no other variable can reasonably cause the outcome. This is usually impossible to achieve strictly, which is why researchers usually aim to control for covariates and thus reach a condition of pseudo-isolation. A common, but not often acknowledged problem is overfitting; adding covariates to a model leads to also fitting the measurement error they carry with them. Association means there should be a connection between the cause and the effect – in real life, usually a probabilistic one. In social sciences, a problem arises as everything is more or less correlated with everything else, and high-dimensional datasets suffer of the “curse of dimensionality”. Direction, self-evidently, means that the effect should flow from one direction to the other, not the other way around. This is highly problematic in complex systems. For an example in health psychology, it seems obvious that depression symptoms (e.g. anxiety and insomnia) feed each other, resulting in self-enforcing feedback loops (7).

When we consider the act of making efficient inferences, we want to be able to falsify our theories of the world (9); something that’s only recently really starting to be understood among psychologists (10). An easy-ish way about this, is to define the smallest effect size of interest (SESOI) a priori, ensure one has proper statistical power and attempt to reject the hypotheses that effects are larger than the upper bound of the SESOI, and lower than the lower bound. This procedure, also known as equivalence testing (11) allows for rejecting the falsification of statistical hypotheses in situations, where a SESOI can be determined. But when testing program theories of complex interventions, there may be no such luxury.

The notion of non-linear interactions with feedback loops makes the notion of causality in a complex system an evasive concept. If we’re dealing with complexity, it is a situation where even miniscule effects can be meaningful when they interact with other effects: even small effects can have huge influences down the line (“the butterfly effect” in nonlinear dynamics; 8). It is hence difficult to determine the SESOI for intermediate links in the chain from intervention to outcome. And if we only say we expect an effect to be “any positive number”, this leads to the postulated processes, as described in intervention program theories, being unfalsifiable: If a correlation of 0.001 between intervention participation and a continuous variable would corroborate a theory, one would need more than six million participants to detect it (at 80% power and an alpha of 5%; see also 12, p. 30). If researchers are unable to reject the null hypothesis of no effect, they cannot determine whether there is evidence for a null effect, or if a more elaborate sample was needed (e.g. 13).

Side note: One could use Bayes factors to compare whether a point null data generator (effect size being zero) would predict the data better than, for example, an alternative model where most effects are near zero but half of them over d = 0.2. But still, the smaller effects you consider potentially important, the less the data can distinguish between alternative and null models. A better option could be to estimate, how probable it is that the effect has a positive sign (as demonstrated here).

In sum, researchers are faced with an uncomfortable trade-off: Either they must specify a SESOI (and thus, a hypothesis) which does not reflect the theory under test or, on the other hand, unfalsifiability.

A common way to study mechanisms is to conduct a mediation analysis, where one variable’s (X) impact on another (Y) is modelled to pass through a third variable (M). In its classical form, one expects the path X-Y to go near zero, when M is added to the model.

The good news is, that nowadays we can do power analyses for both simple and complex mediation models (14).  The bad news is, that in the presence of randomisation of X but not M, the observed M-Y relation entails strong assumptions which are usually ignored (15). Researchers should e.g. justify why there exist no other mediating variables than the ones in the model; leaving variables out is effectively the same as assuming their effect to be zero. Also, the investigator should demonstrate why no omitted variables affect both M and Y – if there are such variables, the causal effect may be distorted at best and misleading at worst.

Now that we know it’s bad to omit variables, how do we avoid overfitting the model (i.e. be fooled by looking too much into what the data says)? It is very common for seemingly supported theories to fail to generalise to slightly different situations or other samples (16), and subgroup claims regularly fail to pan out in new data (17). Some solutions include ridge regression in the frequentist framework and regularising priors in the Bayesian one, but the simplest (though not the easiest) solution would be cross-validation. In cross-validation, you basically divide your sample in two (or even up to n) parts, use the first one to explore and the second one to “replicate” the analysis. Unfortunately, you need to have a large enough sample so that you can break it down to parts.

What does all this tell us? Mainly, that investigators would do well to heed Kenny’s (18) admonition: “mediation is not a thoughtless routine exercise that can be reduced down to a series of steps. Rather, it requires a detailed knowledge of the process under investigation and a careful and thoughtful analysis of data”. I would conjecture that researchers often lack such process knowledge. It may also be, that under complexity, the exact processes become both unknown and unknowable (19). Tools like structural equation modelling are wonderful, but I’m curious if they are up to the task of advising us about how to live in interconnected systems, where trends and cascades are bound to happen, and everything causes everything else.

These are just relatively disorganised thoughts, and I’m curious to hear if someone can shed hope to the situation. Specifically, hearing of interventions that work consistently and robustly, would definitely make my day.

ps. If you’re interested in replication matters in health psychology, there’s an upcoming symposium on the topic in EHPS17 featuring Martin Hagger, Gjalt-Jorn Peters, Rik Crutzen, Marie Johnston and me. My presentation is titled “Disentangling replicable mechanisms of complex interventions: What to expect and how to avoid fooling ourselves?

pps. A recent piece in Lancet (20) called for a complex systems model of evidence for public health. Here’s a small conversation with the main author, regarding the UK Medical Research Council’s take on the subject. As you see, the science seems to be in some sort of a limbo/purgatory-type of place currently, but smart people are working on it so I have hope 🙂

complexity rutter twitter.PNG

https://twitter.com/harryrutter/status/876219437430517761

 

Bibliography

 

  1. Moore GF, Audrey S, Barker M, Bond L, Bonell C, Hardeman W, et al. Process evaluation of complex interventions: Medical Research Council guidance. BMJ. 2015 Mar 19;350:h1258.
  2. Abraham C, Johnson BT, de Bruin M, Luszczynska A. Enhancing reporting of behavior change intervention evaluations. JAIDS J Acquir Immune Defic Syndr. 2014;66:S293–S299.
  3. Shiell A, Hawe P, Gold L. Complex interventions or complex systems? Implications for health economic evaluation. BMJ. 2008 Jun 5;336(7656):1281–3.
  4. Sterman JD. Learning from Evidence in a Complex World. Am J Public Health. 2006 Mar 1;96(3):505–14.
  5. Resnicow K, Page SE. Embracing Chaos and Complexity: A Quantum Change for Public Health. Am J Public Health. 2008 Aug 1;98(8):1382–9.
  6. Bollen KA. Structural equations with latent variables. New York: John Wiley. 1989;
  7. Borsboom D. A network theory of mental disorders. World Psychiatry. 2017 Feb;16(1):5–13.
  8. Hilborn RC. Sea gulls, butterflies, and grasshoppers: A brief history of the butterfly effect in nonlinear dynamics. Am J Phys. 2004 Apr;72(4):425–7.
  9. LeBel EP, Berger D, Campbell L, Loving TJ. Falsifiability Is Not Optional. Accepted pending minor revisions at Journal of Personality and Social Psychology. [Internet]. 2017 [cited 2017 Apr 21]. Available from: https://osf.io/preprints/psyarxiv/dv94b/
  10. Morey R D, Lakens D. Why most of psychology is statistically unfalsifiable. GitHub [Internet]. in prep. [cited 2016 Oct 23]; Available from: https://github.com/richarddmorey/psychology_resolution
  11. Lakens D. Equivalence Tests: A Practical Primer for t-Tests, Correlations, and Meta-Analyses [Internet]. 2016 [cited 2017 Feb 24]. Available from: https://osf.io/preprints/psyarxiv/97gpc/
  12. Dienes Z. Understanding Psychology as a Science: An Introduction to Scientific and Statistical Inference. Palgrave Macmillan; 2008. 185 p.
  13. Dienes Z. Using Bayes to get the most out of non-significant results. Quant Psychol Meas. 2014;5:781.
  14. Schoemann AM, Boulton AJ, Short SD. Determining Power and Sample Size for Simple and Complex Mediation Models. Soc Psychol Personal Sci. 2017 Jun 15;194855061771506.
  15. Bullock JG, Green DP, Ha SE. Yes, but what’s the mechanism? (don’t expect an easy answer). J Pers Soc Psychol. 2010;98(4):550–8.
  16. Yarkoni T, Westfall J. Choosing prediction over explanation in psychology: Lessons from machine learning. FigShare Httpsdx Doi Org106084m9 Figshare. 2016;2441878:v1.
  17. Wallach JD, Sullivan PG, Trepanowski JF, Sainani KL, Steyerberg EW, Ioannidis JPA. Evaluation of Evidence of Statistical Support and Corroboration of Subgroup Claims in Randomized Clinical Trials. JAMA Intern Med. 2017 Apr 1;177(4):554–60.
  18. Kenny DA. Reflections on mediation. Organ Res Methods. 2008;11(2):353–358.
  19. Bar-Yam Y. The limits of phenomenology: From behaviorism to drug testing and engineering design. Complexity. 2016 Sep 1;21(S1):181–9.
  20. Rutter H, Savona N, Glonti K, Bibby J, Cummins S, Finegood DT, et al. The need for a complex systems model of evidence for public health. The Lancet [Internet]. 2017 Jun 13 [cited 2017 Jun 17];0(0). Available from: http://www.thelancet.com/journals/lancet/article/PIIS0140-6736(17)31267-9/abstract

 

Replication is impossible, falsification unnecessary and truth lies in published articles (?)

joonasautocomic
Is psychology headed towards being a science conducted by “zealots”, or to a post-car (or train) crash metaphysics, where anything goes because nothing is even supposed to replicate?

I recently peer reviewed a partly shocking piece called “Reproducibility in Psychological Science: When Do Psychological Phenomena Exist?“ (Iso-Ahola, 2017). In the article, the author makes some very good points, which unfortunately get drowned under very strange statements and positions. Me, Eiko Fried and Etienne LeBel addressed those shortly in a commentary (preprint; UPDATE: published piece). Below, I’d like to expand upon some additional thoughts I had about the piece, to answer Martin Hagger’s question.

On complexity

When all parts do the same thing on a certain scale (planets on Newtonian orbits), their behaviour is relatively easy to predict for many purposes. Same thing, when all molecules act independently in a random fashion: the risk that most or all beer molecules in a pint move upward at the same time is ridiculously low, and thus we don’t have to worry about the yellow (or black, if you’re into that) gold escaping the glass. Both situations are easy-ish systems to describe, as opposed to complex systems where the interactions, sensitivity to initial conditions etc. can produce a huge variety of behaviour and states. Complexity science is the study of these phenomena, which have become increasingly common since the 1900s (Weaver, 1948).

Iso-Ahola (2017) quotes (though somewhat unfaithfully) the complexity scientist Bar-Yam (2016b): “for complex systems (humans), all empirical inferences are false… by their assumptions of replicability of conditions, independence of different causal factors, and transfer to different conditions of prior observations”. He takes this to mean that “phenomena’s existence should not be defined by any index of reproducibility of findings” and that “falsifiability and replication are of secondary importance to advancement of scientific fields”. But this is a highly misleading representation of the complexity science perspective.

In Bar-Yam’s article, he used an information theoretic approach to analyse the limits of what we can say about complex systems. The position is that while full description of systems via empirical observation is impossible, we should aim to identify the factors which are meaningful in terms of replicability of findings, or the utility of the acquired knowledge. As he elaborates elsewhere: “There is no utility to information that is only true in a particular instance. Thus, all of scientific inquiry should be understood as an inquiry into universality—the determination of the degree to which information is general or specific” (Bar-Yam, 2016a, p. 19).

This is fully in line with the Fisher quote presented in Mayo’s slides:

Fisher quote Mayo

The same goes for replications; no single one-lab study can disprove a finding:

“’Thus a few stray basic statements contradicting a theory will hardly induce us to reject it as falsified. We shall take it as falsified only if we discover a reproducible effect which refutes the theory. In other words, we only accept the falsification if a low-level empirical hypothesis which describes such an effect is proposed and  corroborated’ (Popper, 1959, p. 66)” (see Holtz & Monnerjahn, 2017)

So, if the high-quality non-replication replicates, one must consider that something may be off with the original finding. This leads us to the question of what researchers should study in the first place.

On research programmes

Lakatos (1971) posits a difference between progressive and degenerating research lines. In a progressive research line, investigators explain a negative result by modifying the theory in a way which leads to new predictions that subsequently pan out. On the other hand, coming up with explanations that do not make further contributions, but rather just explain away the negative finding, leads to a degenerative research line. Iso-Ahola quotes Lakatos to argue that, although theories may have a “poor public record” that should not be denied, falsification should not lead to abandonment of theories. Here’s Lakatos:

“One may rationally stick to a degenerating [research] programme until it is overtaken by a rival and even after. What one must not do is to deny its poor public record. […] It is perfectly rational to play a risky game: what is irrational is to deceive oneself about the risk” (Lakatos, 1971, p. 104)

As Meehl (1990, p. 115) points out, the quote continues as follows:

“This does not mean as much licence as might appear for those who stick to a degenerating programme. For they can do this mostly only in private. Editors of scientific journals should refuse to publish their papers which will, in general, contain either solemn reassertions of their position or absorption of counterevidence (or even of rival programmes) by ad hoc, linguistic adjustments. Research foundations, too, should refuse money.” (Lakatos, 1971, p. 105)

Perhaps researchers should pay more attention which program they are following?

As an ending note, here’s one more interesting quote: “Zealotry of reproducibility has unfortunately reached the point where some researchers take a radical position that the original results mean nothing if not replicated in the new data.” (Iso-Ahola, 2017)

For explorative research, I largely agree with these zealots. I believe exploration is fine and well, but the results do mean nearly nothing unless replicated in new data (de Groot, 2014). One cannot hypothesise and confirm with the same data.

Perhaps I focus too much on the things that were said in the paper, not what the author actually meant, and we do apologise if we have failed to abide with the principle of charity in the commentary or this blog post. I do believe the paper will best serve as a pedagogical example to aspiring researchers, on how strangely arguments could be constructed in the olden times.

ps. Bar-Yam later commented on this blog post, confirming the mis(present/interpr)etation of his research by the author of the reproducibility paper:

baryam

pps. Here’s Fred Hasselman‘s comment on the article, from the Frontiers website (when you scroll down all the way to the bottom, there’s a comment option):

1. Whether or not a posited entity (e.g. a theoretical object of measurement) exists or not, is a matter of ontology.

2. Whether or not one can, in principle, generate scientific knowledge about a posited entity, is a matter of epistemology.

3. Whether or not the existence claim of a posited entity (or law) is scientifically plausible depends on the ability of a theory or nomological network to produce testable predictions (predictive power) and the accuracy of those predictions relative to measurement outcomes (empirical accuracy).

4. The comparison of the truth status of psychological theoretical constructs to the Higgs Boson is a false equivalence: One is formally defined and deduced from a highly corroborated model and predicts the measurement context in which its existence can be verified or falsified (the LHC), the other is a common language description of a behavioural phenomenon “predicted” by a theory constructed from other phenomena published in the scientific record of which the reliability is… unknown.

5. It is the posited entity itself -by means of its definition in a formalism or theory that predicts its existence- that decides how it can be evidenced empirically. If it cannot be evidenced using population statistics, don’t use it! If the analytic tools to evidence it do not exist, develop them! Quantum physics had to develop a new theory of probability, new mathematics to be able to make sense of measurement outcomes of different experiments. Study non-ergodic physics, complexity science, emergence and self-organization in physics, decide if it is sufficient, if not, develop a new formalism. That is how science advances and scientific knowledge is generated. Not by claiming all is futile.

To summarise: The article continuously confuses ontological and epistemic claims, it does not provide a future direction even though many exist or are being proposed by scholars studying phenomena of the mind, moreover the article makes no distinction between sufficiency and necessity in existence claims, and this is always problematic.

Contrary to the claim here, a theory (and the ontology and epistemology that spawned it) can enjoy high perceived scientific credibility even if some things cannot be known in principle, or if there’s always uncertainty in measurements. It can do so by being explicit about what it is that can and cannot be known about posited entities.

E.g. Quantum physics is a holistic physical theory, also in the epistemic sense: It is in principle not possible to know anything about a quantum system at the level of the whole, based on knowledge about its constituent parts. Even so, quantum physical theories have the highest predictive power and empirical accuracy of all scientific theories ever produced by human minds!

As evidenced by the history of succession of theories in physics, successful scientific theorising about the complex structure of reality seems to be a highly reproducible phenomenon of the mind. Let’s apply it to the mind itself!

Bibliography:

Bar-Yam, Y. (2016a). From big data to important information. Complexity, 21(S2), 73–98.

Bar-Yam, Y. (2016b). The limits of phenomenology: From behaviorism to drug testing and engineering design. Complexity, 21(S1), 181–189. https://doi.org/10.1002/cplx.21730

de Groot, A. D. (2014). The meaning of “significance” for different types of research [translated and annotated by Eric-Jan Wagenmakers, Denny Borsboom, Josine Verhagen, Rogier Kievit, Marjan Bakker, Angelique Cramer, Dora Matzke, Don Mellenbergh, and Han L. J. van der Maas]. Acta Psychologica, 148, 188–194. https://doi.org/10.1016/j.actpsy.2014.02.001

Holtz, P., & Monnerjahn, P. (2017). Falsificationism is not just ‘potential’ falsifiability, but requires ‘actual’ falsification: Social psychology, critical rationalism, and progress in science. Journal for the Theory of Social Behaviour. https://doi.org/10.1111/jtsb.12134

Iso-Ahola, S. E. (2017). Reproducibility in Psychological Science: When Do Psychological Phenomena Exist? Frontiers in Psychology, 8. https://doi.org/10.3389/fpsyg.2017.00879

Lakatos, I. (1971). History of science and its rational reconstructions. Springer. Retrieved from http://link.springer.com/chapter/10.1007/978-94-010-3142-4_7

Meehl, P. E. (1990). Appraising and amending theories: The strategy of Lakatosian defense and two principles that warrant it. Psychological Inquiry, 1(2), 108–141.

Weaver, W. (1948). Science and complexity. American Scientist, 36(4), 536–544.