Agenda » Syllabus
Basic course (Modules 1-4)
Basic Course (Module 1)
Basic concepts 1
Meta-Analysis where the effect size is consistent across studies
Expand
Introduction
There is a widespread belief that the goal of a meta-analysis is to estimate the mean effect size for a set of studies. While this is sometimes correct it is helpful to think of meta-analyses as falling into two groups. When the impact of a treatment is essentially similar for all studies in the analysis, it does make sense to focus on the mean effect, which is also the common effect. However, when the effect size varies substantially across studies we need to quantify the extent of dispersion and consider the clinical or substantive importance of that dispersion. This will be a key element in the course.
Cannon Analysis
We start with a simple meta-analysis to compare the relative utility of two treatments for preventing cardiovascular events. I use this to outline the elements of a meta-analysis that we will be exploring in later modules. I also use this to show how we can perform a simple analysis from start to finish, including generating a high-resolution plot and writing a report.
Tamiflu Symptom Relief
This is an analysis of randomized controlled trials (RCTs) that compared the duration of flu symptoms in patients treated with Tamiflu to those treated with a placebo. This is the second example where the effect size is consistent across studies.
How a meta-analysis works.
I use a series of fictional studies to show what happens as we add studies to a meta-analysis. When the effect size is consistent across studies, we focus on the common effect size. When the effect size varies across studies we estimate the mean effect size, but we also need to estimate the dispersion in effects and consider the implications of this dispersion for the utility of the intervention.
Fixed-effect vs. random-effects
Every meta-analysis must be based on a statistical model. The model tells us how the studies were sampled and how we can generalize from them to other studies or populations. We discuss how to select a model, and also how to avoid common mistakes related to this issue.
Learning Objectives
At the conclusion of this module students will be able to:
- Perform a meta-analysis where the effect-size is consistent across studies
- Understand how to choose a statistical model
- Report the results for a meta-analysis where the effect size is consistent
- Identify and avoid common mistakes related to choosing a statistical model
Basic Course (Module 2)
Basic concepts 2
Meta-Analysis where effect size varies across studies
Expand
Effect size indices
The effect size is the unit of currency in a meta-analysis. We compute an effect size for each study. Then, we pool these values to estimate the common (or mean) effect size, and the dispersion in effects. We discuss how to choose an effect size index, and how to understand and explain the meaning of the effect size.
Effect size vs p-values
In primary studies that compare outcomes in two groups, there are two general approaches that researchers apply. One is to test the null hypothesis of no effect and report a p-value. The other is to estimate the effect size, and report that effect size along with a confidence interval. In this module I show why we almost always want to focus on the effect size approach. Then I extend this to meta-analysis, where it is imperative that we work with the effect size from each study rather than the p-value.
What studies to include
In any meta-analysis we can choose to work with a narrowly defined population and a specific variant of the intervention. In this case, we assume that the effect size will be reasonably consistent across studies, and our goal will be to estimate this effect size. Alternatively, we can choose to include an array of populations and/or variants of the intervention. In this case, our goal will be to assess the dispersion in effects and possibly to see what moderators are associated with this dispersion. We explain how to make these decisions and how to map them to the inclusion/exclusion criteria.
Heterogeneity
In most meta-analyses, the effect size varies from study to study. It’s important to understand how much the effect size varies, and to consider the clinical or substantive implications of this variation. In this module, I start by reviewing how we think about heterogeneity in a primary study. Then, I show that the same ideas apply in a meta-analysis.
Case study: ADHD analysis
This is an analysis of seventeen randomized controlled trials (RCTs) that assessed the impact of methylphenidate on adults with ADHD (attention deficit hyperactivity disorder). This is an opportunity to develop a feel (in practice) for the concepts we earlier discussed in the abstract.
Learning Objectives
At the conclusion of this module students will be able to:
- Understand how to determine what kinds of studies to include in an analysis
- Understand how to quantify heterogeneity in a meta-analysis
- Identify and avoid common mistakes
Basic Course (Module 3)
How to perform and report a meta-analysis
using continuous and dichotomous outcomes
Expand
The I2 statistic
The vast majority of meta-analyses use the I2 statistic to quantify heterogeneity. Many use this statistic to quantify heterogeneity as being low, moderate or high. While this practice is ubiquitous it is nevertheless incorrect. I2 is a proportion, not an absolute amount. It tells us what proportion of the variance is due to differences in the true effects rather than sampling error. It does not tell us how much variance there is. Classifications of heterogeneity based on I2 are uninformative at best, and sometimes misleading. I explain this in detail, and then discuss the prediction interval, the statistic that provides the information that researchers believe (incorrectly) is being reported by I2.
Since many students find this idea so hard to accept, I include a video clip from one of my in-person workshops where Julian Higgins (the co-creator of I2) was kind enough to speak and confirm that he completely agrees with this point.
Case study: The St. John’s Wort analysis
This is an analysis of studies where patients with severe depression were randomized randomized to received either hypericum or a placebo. The studies looked at the impact of the treatment on the depression. We will use this analysis to discuss how to estimate the mean effect size and the dispersion in effects working with dichotomous outcomes.
Learning Objectives
In this module students will gain experience in all steps related to an analysis including
- How to enter data
- How to run the analysis
- How to understand and report the results
- How to properly quantify heterogeneity in effects
- How to create a high-resolution plot
- How to perform sensitivity analyses
- To avoid mistakes related to heterogeneity
Basic Course (Module 4)
How to perform and report a meta-analysis
using incidence, correlation, and risk difference
Expand
Overview
We will work through a series of analyses from start to finish. This will provide concrete examples of the topics introduced earlier and an opportunity for students to become comfortable performing these analyses on their own. The datasets include the following (among others)
Case study: The mitral-valve analysis
While most meta-analyses work with studies that compared two groups, we can also use meta-analysis to work with studies that looked at the risk of an event in one group – such as incidence or prevalence. The mitral valve analysis is one example. This analysis looks at the risk of death following mitral-valve surgery in elderly patients.
Case study: The reading analysis
This is an analysis of studies that reported the correlation between children’s vocabulary skills at a very young age and their language skills when they entered school. A high correlation would mean that the early assessment could be employed to identify children who might benefit from early intervention.
Case study: The Tocilizumab analysis
This is an analysis of observational studies of patients who had been hospitalized for Covid-19. In each study some patients were treated with tocilizumab and others were not. The analysis looked at the relationship between treatment and risk of death. The paper reported that patients treated with the drug were more likely to survive and that this relationship was statistically significant. However, we will reanalyze the data and show that the truth is more complicated. While the drug was associated with increased survival on average, it was actually associated with an increased risk of death in roughly 35% of studies. This analysis provides an example of why it is important to quantify heterogeneity properly and take account of the heterogeneity when assessing the potential utility of an intervention,
Learning Objectives
In this module students will gain experience in all steps related to an analysis including
- How to enter data
- How to run the analysis
- How to understand and report the results
- How to properly quantify heterogeneity in effects
- How to create a high-resolution plot
- How to perform sensitivity analyses
- To avoid mistakes related to heterogeneity
Advanced course (Modules 5-8)
Advanced Course (Module 5)
Subgroup Analyses –
Comparing the effect size in different sets of studies
Expand
Overview
In the basic course we introduced a number of meta-analyses where the effect size varied across studies, and we learned how to estimate the mean effect size and the heterogeneity in effects. In the advanced course we return to those data sets and learn how to explain some of that heterogeneity using subgroup analysis and meta-regression (analogous to ANOVA and multiple regression). We also learn how to assess the potential impact of publication bias in these analyses.
Comparing subgroups
In a primary study we may use a t-test or a one-way analysis of variance to compare the mean score in two or more groups of people. In a meta-analysis we may use a subgroup analysis to compare the mean effect size in two or more subgroups of studies.
Case study: The ADHD Analysis
We will revisit the analysis to look at the relationship between dose and patient type
Case study: The Mitral-valve analysis
We will revisit the analysis and compare the risk for studies that employed different variants of the procedure
Case study: St. John’s Wort analysis
We will revisit the analysis to compare the effect size for different types of patients
Case study: Reading
We will revisit the analysis to see if the correlation varies depending on the child’s age
Case study: Tocilizumab
We will revisit the analysis to see if the effect varies the type of patient
Learning Objectives
In this module students will learn
- How to run a subgroups analysis
- How to select a statistical model for a subgroups analysis
- How to estimate Tau-squared in the presence of subgroups
- That subgroup comparisons are observational
- How to report the results of a subgroup analysis
- How to avoid common mistakes when comparing subgroups
Advanced Course (Module 6)
Meta-regression –
Assessing the relationship of covariates with effect size
Expand
Overview
In a primary study we may use multiple regression to assess the relationship between covariates and scores. In a meta-analysis we may use meta-regression to assess the relationship between covariates and effect sizes
Case study: The ADHD Analysis
We will revisit this analysis to look at the relationship between dose and effect sizes
Case study: The Mitral-valve Analysis
We will revisit this analysis to look for covariates related to the risk of death
Case study: The St. Johns’ Wort Analysis
We will revisit this analysis to look for covariates related to the effect size.
Case study: The Reading Analysis
We will revisit this analysis to look for covariates related to the effect size.
In each case we will isolate the unique impact of specific factors while controlling for potential confounds. We will discuss the meaning of tau-squared and prediction intervals in the presence of covariates. We will discuss the meaning and limitations of the R-squared analog. We will discuss how to plot the impact of continuous and categorical predictors, and interactions
Learning Objectives
In this module students will learn
- How to perform a meta-regression in CMA
- How to work with continuous covariates
- How to work with categorical covariates
- How to work with sets of covariates
- How to understand the I-sq analog
- How to understand the R-sq analog
- How to plot results of a meta-regression
- How to assess the unique impact of covariates
- How to use the Knapp-Hartung adjustment in meta-regression
- How to avoid common mistakes in meta-regression
Advanced Course (Module 7)
Working with complex data sets
Multiple outcomes, time points, comparisons, subgroups
Expand
Complex data sets
A meta-analysis may include studies that report data for complex data sets. This may include data for two or more independent subgroups. It may include data for two or more outcomes or timepoints based on the same people. It may include studies that reported data for two or more treatments vs. a common control group.
Learning Objectives
In this module students will learn
- How to work with studies that present data for two or more independent subgroups
- How to work with studies that present data for two or more outcomes based on the same subjects
- How to work with studies that present data for two or more timepoints based on the same subjects
- How to work with studies that present data for studies that reported data for two or more treatments vs. a common control group.
Advanced Course (Module 8)
Publication bias, limitations of the random effects model
Risk of bias, and other issues
Expand
Publication bias
Most meta-analyses are based on studies pulled from the literature. Since studies are more likely to be published if they report statistically significant results, there is a tendency for the literature to incorporate a biased subset of studies. It follows that a meta-analysis based on these studies will tend to overestimate the magnitude of the effect. We discuss how to address this problem.
Case study: The ADHD Analysis
Case study: The Mitral-valve Analysis
Case study: The St. Johns’ Wort Analysis
Case study: The Reading Analysis
Case study: The tocilizumab analysis
Limitations of the random-effects model
While meta-analysis provides an exceptional tool, it is imperative that we understand the limitations of this tool. We will discuss the limitations of the random-effects model. In particular, we will discuss limitations that apply when we are working with a small number of studies, and how to deal with these cases.
Learning Objectives
In this module students will learn
- How to assess the potential impact of publication bias
- How to generalize from the studies in the analysis to a wider universe of studies
- How to work with analyses when there are only a small number of studies
- How to use the Knapp-Hartung adjustment
- When it makes sense to perform a meta-analysis
- How to avoid common mistakes associated with these issues
Testimonials
"The course content is wonderful and Dr. Borenstein is extremely patient and helpful in his interactions. Even though I have undertaken Meta-Analysis studies, I did pick up additional nuances that add to the clarity of understanding. Thanks a lot Dr. Borenstein!"
Moutusi Maity - IIM Lucknow - Sydney 2020
"Thank you very much for the wonderful workshop at Kent State University. I really enjoyed it. I particularly like the way you organize the course, starting with the concept, then applications and examples, and finally common mistakes."
Jingzhen (Ginger) Yang, PhD, MPH - Associate Professor, Department of Social and Behavioral Sciences, College of Public Health, Kent State University