Navigation auf uzh.ch

Suche

Department of Psychology

Newsletter #8: Frühjahrssemester 25

Welcome

Dear members of the Department of Psychology,

The 8th newsletter of the Open Science Initiative at the Department of Psychology promises to be an exciting read!

This time, we have some news about the Open Science Prize 2025, provide a short overview of multiverse analysis, and report from the Data Protection Week!

Questions, suggestions, and contributions to this newsletter may be sent toopenscience@psychologie.uzh.ch. The next newsletter will appear at the beginning of HS25.

Warm regards,
Your Open Science Initiative

 

Topics

Table of contents

Open Science Prize 2025

We received many applications for the Open Science Prize 2025 in all categories. This year, we would like to announce two changes: First, after many years of sponsorship by the URPP Dynamics of Healthy Aging, the prizes for papers by PhD students and by postdocs will be sponsored by PSYCH. We gratefully acknowledge this support! Second, the winners of the Open Science Prize 2025 will be announced on May 26th during the ExPra congress and the MaDoKo, which will provide a festive context.

Multiverse Analysis

[Martin Götz]

“Female hurricanes are deadlier than male hurricanes” (Jung et al., 2014), “Women's mate preferences change across the ovulatory cycle” (Gangestad et al., 2007), and "Frontal alpha asymmetry (FAA) is a measure of depressive disorders (DD)" (Kemp et al., 2010). Our literature is brimming with bold statements supposedly lending themselves to future research and practical implications, but a considerable number of them might (not yet) hold unequivocal answers to the research questions (cf. Kołodziej et al., 2021; Simonsohn et al., 2020; Stern et al., 2019).

Simmons et al. (2011) introduced the term researcher degrees of freedom (RDF) to describe the wide array of equally valid and theoretically sound choices available at each stage of the research process (e.g., Gelman & Loken, 2013; Götz & O’Boyle, 2023; Zitzmann et al., 2024). RDF induce variability in research methods, even when the objective remains constant. This means that multiple researchers beginning with the same fundamental question and limited to theoretically or empirically justified options can still arrive at vastly different conclusions owing to RDF. Moreover, RDF contribute to multiple comparisons issues that ultimately increase false-positive rates, obscuring the true robustness of scientific claims and the resulting knowledge base when published (e.g., Gelman & Loken, 2013; Götz & O’Boyle, 2023; Steegen et al., 2016).

The cumulative nature of choices throughout the research process means that individual RDF leading to significantly different conclusions need not be drastically different themselves, such as deciding between qualitative and quantitative approaches. Instead, RDF often involve seemingly minor differences in the research process, such as one researcher measuring a demographic control variable using categories (e.g., age bands) and another requesting an exact value. These comparable RDF at various decision points can individually and collectively affect distant outcomes, potentially affecting the overall acceptance or rejection of the theory, model, or intervention under study. Thus, there is a need to aggregate and contextualize results from one set of decisions (i.e., a universe) within a broader set resulting from the combination of similarly viable decisions (i.e., a multiverse).

A multiverse represents a finite space containing alternative decisions and respective options that, if selected, would be defensible for a given research endeavor. For instance, if measuring age using age ranges contradicted theory and norms in developmental psychology, it would not be a viable decision for researchers in that field. Consequently, any pathways or universes incorporating this choice would be excluded from the multiverse. However, even when restricted to viable universes with minor variations, a multiverse encompassing thousands, if not millions, of unique universes is likely to emerge. For example, a researcher making only three decisions (e.g., outlier treatment, control variable inclusion, outcome variable operationalization) with four options per decision (e.g., outlier removal, winsorization, log transformation, robust estimation) will generate 43 or 64 universes. A fourth decision option would create 256 universes, and a fifth option per decision would produce over 1000 universes, each potentially conforming to theory, existing literature, and research norms.

Multiverse analysis represents a somewhat novel methodological approach to acknowledge, illustrate, and investigate the impact of RDF on the (in)variance of a scientific finding (e.g., Götz et al., 2024; Sarma et al., 2023; Steegen et al., 2016). This highly versatile method allows researchers to clearly think about their choices in the research process, the respective alternatives for these choices, and their potential impact on their findings; we consider it a thought-provoking approach to addressing the robustness of one’s research findings, and would like to invite you to learn more about it using the following tutorials.

Tutorials

  • Götz, M., Sarma, A., & O’Boyle, E. H. (2024). The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse. International Journal of Psychology, 59(6), 1003–1014. https://doi.org/10.1002/ijop.13229
  • Heyman, T., & Vanpaemel, W. (2022). Multiverse analyses in the classroom. Meta-Psychology, 6. https://doi.org/10.15626/MP.2020.2718
  • Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637

References

  • Gangestad, S. W., Garver-Apgar, C. E., Simpson, J. A., & Cousins, A. J. (2007). Changes in women’s mate preferences across the ovulatory cycle. Journal of Personality and Social Psychology, 92(1), 151–163. https://doi.org/10.1037/0022-3514.92.1.151
  • Gelman, A., & Loken, E. (2013). The garden of forking paths: Why multiple comparisons can be a problem, even when there is no “fishing expedition” or “p-hacking” and the research hypothesis was posited ahead of time (pp. 1–17). Department of Statistics, Columbia University. http://stat.columbia.edu/~gelman/research/unpublished/forking.pdf
  • Götz, M., & O’Boyle, E. H. (2023). Cobblers, let’s stick to our lasts! A song of sorrow (and of hope) about the state of personnel and human resource management science. In M. R. Buckley, A. R. Wheeler, J. E. Baur, & J. R. B. Halbesleben (Eds.), Research in personnel and human resources management (Vol. 41, pp. 7–92). Emerald. https://doi.org/10.1108/S0742-730120230000041004
  • Götz, M., Sarma, A., & O’Boyle, E. H. (2024). The multiverse of universes: A tutorial to plan, execute and interpret multiverses analyses using the R package multiverse. International Journal of Psychology, 59(6), 1003–1014. https://doi.org/10.1002/ijop.13229
  • Jung, K., Shavitt, S., Viswanathan, M., & Hilbe, J. M. (2014). Female hurricanes are deadlier than male hurricanes. Proceedings of the National Academy of Sciences, 111(24), 8782–8787. https://doi.org/10.1073/pnas.1402786111
  • Kemp, A. H., Griffiths, K., Felmingham, K. L., Shankman, S. A., Drinkenburg, W., Arns, M., Clark, C. R., & Bryant, R. A. (2010). Disorder specificity despite comorbidity: Resting EEG alpha asymmetry in major depressive disorder and post-traumatic stress disorder. Biological Psychology, 85(2), 350–354. https://doi.org/10.1016/j.biopsycho.2010.08.001
  • Kołodziej, A., Magnuski, M., Ruban, A., & Brzezicka, A. (2021). No relationship between frontal alpha asymmetry and depressive disorders in a multiverse analysis of five studies. eLife, 10, e60595. https://doi.org/10.7554/eLife.60595
  • Sarma, A., Kale, A., Moon, M. J., Taback, N., Chevalier, F., Hullman, J., & Kay, M. (2023). multiverse: Multiplexing alternative data analyses in R notebooks. Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems, 1–15. https://doi.org/10.1145/3544548.3580726
  • Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22(11), 1359–1366. https://doi.org/10.1177/0956797611417632
  • Simonsohn, U., Simmons, J. P., & Nelson, L. D. (2020). Specification curve analysis. Nature Human Behaviour, 4(11), 1208–1214. https://doi.org/10.1038/s41562-020-0912-z
  • Steegen, S., Tuerlinckx, F., Gelman, A., & Vanpaemel, W. (2016). Increasing transparency through a multiverse analysis. Perspectives on Psychological Science, 11(5), 702–712. https://doi.org/10.1177/1745691616658637
  • Stern, J., Arslan, R. C., Gerlach, T. M., & Penke, L. (2019). No robust evidence for cycle shifts in preferences for men’s bodies in a multiverse analysis: A response to Gangestad, Dinh, Grebe, Del Giudice, and Emery Thompson (2019). Evolution and Human Behavior, 40(6), 517–525. https://doi.org/10.1016/j.evolhumbehav.2019.08.005
  • Zitzmann, S., Wagner, W., Lavelle-Hill, R., Jung, A. J., Jach, H., Loreth, L., Lindner, C., Schmidt, F. T. C., Edelsbrunner, P. A., Schaefer, C. D., Deutschländer, R., Schauber, S. K., Krammer, G., Wolff, F., Hui, B., Fischer, C., Bardach, L., Nagengast, B., & Hecht, M. (2024). On the role of variation in measures, the worth of underpowered studies, and the need for tolerance among researchers: Some more reflections on Leising et al. from a methodological, statistical, and social-psychological perspective. Personality Science, 5, 27000710241257413. https://doi.org/10.1177/27000710241257413

Insights from the Data Protection Week at UZH

[Lisa-Katrin Kaufmann]

From January 22 to 24, 2025, the UZH held its first-ever Data Protection Week, organized by the Data Stewards and Open Science Services. The event provided practical tools and insights for researchers handling personal or sensitive data. Topics included legal and ethical considerations for handling personal data, best practices for securely storing and anonymizing data, and strategies for sharing data effectively in collaborative projects. The week featured hybrid lectures and on-site workshops, with active contributions from participants. These inputs will inform a forthcoming White Paper summarizing key insights.

Missed it?
Slides from the presentations are available online (https://zenodo.org/communities/dpwuzh/records), providing a valuable resource for those who couldn’t attend. The full program is available here (
https://www.openscience.uzh.ch/dam/jcr:a3ee0b56-afb9-447e-a5d3-68d6e8648c56/Program_Data-Protection-Week_v20250116.pdf).

For additional guidance and tools, explore the following resources:

Open Science Services by the University Library: Access additional resources and support through the University Library. https://ub.uzh.ch/en/wissenschaftlich-arbeiten/Rechtliche-Aspekte/data-protection.html

Events

Final words

This newsletter is published once a semester – feel free to contact us if you have any questions: openscience@psychologie.uzh.ch

As this newsletter is only published once per semester, we are unable to inform you about events scheduled at (rather) short notice. Thus, we recommend subscribing to the mailing list of UZH’s Center for Reproducible Science to stay up to date on offers for further training and scientific exchange on open science at UZH, such as the ReproducibiliTea Journal Club.

Current members of the open science initiative

Prof. Dr. Johannes Ullrich (Leitung); Dr. Walter Bierbauer; Prof. Dr. Renato Frey; Dr. Martin Götz; M.Sc. Patrick Höhener; Dr. Sebastian Horn; Dr. Lisa Kaufmann; M.Sc. Sophie Kittelberger; Dr. André Kretzschmar; M.Sc. Pascal Küng; Prof. Dr. Nicolas Langer; Dr. Susan Mérillat; Dr. Joanna Rutkowska; Dr. Robin Segerer; Prof. Dr. Carolin Strobl; Dr. Lisa Wagner; M.Sc. Jasmin Weber; Dr. Katharina Weitkamp; M.Sc. Natascha Wettstein