## Agenda » Syllabus

## Basic course (Modules 1-4)

## Basic Course (Module 1)

Basic concepts 1

Meta-Analysis where the effect size is consistent across studies

Expand

##### Introduction

There is a widespread belief that the goal of a meta-analysis is to estimate the mean effect size for a set of studies. While this is sometimes correct it is helpful to think of meta-analyses as falling into two groups. When the impact of a treatment is essentially similar for all studies in the analysis, it does make sense to focus on the mean effect, which is also the common effect. However, when the effect size varies substantially across studies we need to quantify the extent of dispersion and consider the clinical or substantive importance of that dispersion. This will be a key element in the course.

##### Cannon Analysis

We start with a simple meta-analysis to compare the relative utility of two treatments for preventing cardiovascular events. I use this to outline the elements of a meta-analysis that we will be exploring in later modules. I also use this to show how we can perform a simple analysis from start to finish, including generating a high-resolution plot and writing a report.

##### Tamiflu Symptom Relief

This is an analysis of randomized controlled trials (RCTs) that compared the duration of flu symptoms in patients treated with Tamiflu to those treated with a placebo. This is the second example where the effect size is consistent across studies.

##### How a meta-analysis works.

I use a series of fictional studies to show what happens as we add studies to a meta-analysis. When the effect size is consistent across studies, we focus on the common effect size. When the effect size varies across studies we estimate the mean effect size, but we also need to estimate the dispersion in effects and consider the implications of this dispersion for the utility of the intervention.

##### Fixed-effect vs. random-effects

Every meta-analysis must be based on a statistical model. The model tells us how the studies were sampled and how we can generalize from them to other studies or populations. We discuss how to select a model, and also how to avoid common mistakes related to this issue.

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

At the conclusion of this module students will be able to:

- Perform a meta-analysis where the effect-size is consistent across studies
- Understand how to choose a statistical model
- Report the results for a meta-analysis where the effect size is consistent
- Identify and avoid common mistakes related to choosing a statistical model

## Basic Course (Module 2)

Basic concepts 2

Meta-Analysis where effect size varies across studies

Expand

##### Effect size indices

The effect size is the unit of currency in a meta-analysis. We compute an effect size for each study. Then, we pool these values to estimate the common (or mean) effect size, and the dispersion in effects. We discuss how to choose an effect size index, and how to understand and explain the meaning of the effect size.

##### Effect size vs p-values

In primary studies that compare outcomes in two groups, there are two general approaches that researchers apply. One is to test the null hypothesis of no effect and report a p-value. The other is to estimate the effect size, and report that effect size along with a confidence interval. In this module I show why we almost always want to focus on the effect size approach. Then I extend this to meta-analysis, where it is imperative that we work with the effect size from each study rather than the p-value.

##### What studies to include

In any meta-analysis we can choose to work with a narrowly defined population and a specific variant of the intervention. In this case, we assume that the effect size will be reasonably consistent across studies, and our goal will be to estimate this effect size. Alternatively, we can choose to include an array of populations and/or variants of the intervention. In this case, our goal will be to assess the dispersion in effects and possibly to see what moderators are associated with this dispersion. We explain how to make these decisions and how to map them to the inclusion/exclusion criteria.

##### Heterogeneity

In most meta-analyses, the effect size varies from study to study. It’s important to understand how much the effect size varies, and to consider the clinical or substantive implications of this variation. In this module, I start by reviewing how we think about heterogeneity in a primary study. Then, I show that the same ideas apply in a meta-analysis.

##### Case study: ADHD analysis

This is an analysis of seventeen randomized controlled trials (RCTs) that assessed the impact of methylphenidate on adults with ADHD (attention deficit hyperactivity disorder). This is an opportunity to develop a feel (in practice) for the concepts we earlier discussed in the abstract.

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

At the conclusion of this module students will be able to:

- Understand how to determine what kinds of studies to include in an analysis
- Understand how to quantify heterogeneity in a meta-analysis
- Identify and avoid common mistakes

## Basic Course (Module 3)

How to perform and report a meta-analysis

using continuous and dichotomous outcomes

Expand

##### The I^{2} statistic

The vast majority of meta-analyses use the I^{2} statistic to quantify heterogeneity. Many use this statistic to quantify heterogeneity as being low, moderate or high. While this practice is ubiquitous it is nevertheless incorrect. I^{2} is a proportion, not an absolute amount. It tells us what proportion of the variance is due to differences in the true effects rather than sampling error. It does not tell us how much variance there is. Classifications of heterogeneity based on I^{2} are uninformative at best, and sometimes misleading. I explain this in detail, and then discuss the prediction interval, the statistic that provides the information that researchers believe (incorrectly) is being reported by I^{2}.

Since many students find this idea so hard to accept, I include a video clip from one of my in-person workshops where Julian Higgins (the co-creator of I^{2}) was kind enough to speak and confirm that he completely agrees with this point.

##### Case study: The St. John’s Wort analysis

This is an analysis of studies where patients with severe depression were randomized randomized to received either hypericum or a placebo. The studies looked at the impact of the treatment on the depression. We will use this analysis to discuss how to estimate the mean effect size and the dispersion in effects working with dichotomous outcomes.

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

In this module students will gain experience in all steps related to an analysis including

- How to enter data
- How to run the analysis
- How to understand and report the results
- How to properly quantify heterogeneity in effects
- How to create a high-resolution plot
- How to perform sensitivity analyses
- To avoid mistakes related to heterogeneity

## Basic Course (Module 4)

How to perform and report a meta-analysis

using incidence, correlation, and risk difference

Expand

##### Overview

We will work through a series of analyses from start to finish. This will provide concrete examples of the topics introduced earlier and an opportunity for students to become comfortable performing these analyses on their own. The datasets include the following (among others)

##### Case study: The mitral-valve analysis

While most meta-analyses work with studies that compared two groups, we can also use meta-analysis to work with studies that looked at the risk of an event in one group – such as incidence or prevalence. The mitral valve analysis is one example. This analysis looks at the risk of death following mitral-valve surgery in elderly patients.

##### Case study: The reading analysis

This is an analysis of studies that reported the correlation between children’s vocabulary skills at a very young age and their language skills when they entered school. A high correlation would mean that the early assessment could be employed to identify children who might benefit from early intervention.

##### Case study: The Tocilizumab analysis

This is an analysis of observational studies of patients who had been hospitalized for Covid-19. In each study some patients were treated with tocilizumab and others were not. The analysis looked at the relationship between treatment and risk of death. The paper reported that patients treated with the drug were more likely to survive and that this relationship was statistically significant. However, we will reanalyze the data and show that the truth is more complicated. While the drug was associated with increased survival on average, it was actually associated with an increased risk of death in roughly 35% of studies. This analysis provides an example of why it is important to quantify heterogeneity properly and take account of the heterogeneity when assessing the potential utility of an intervention,

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

In this module students will gain experience in all steps related to an analysis including

- How to enter data
- How to run the analysis
- How to understand and report the results
- How to properly quantify heterogeneity in effects
- How to create a high-resolution plot
- How to perform sensitivity analyses
- To avoid mistakes related to heterogeneity

## Advanced course (Modules 5-8)

## Advanced Course (Module 5)

Subgroup Analyses –

Comparing the effect size in different sets of studies

Expand

##### Overview

In the basic course we introduced a number of meta-analyses where the effect size varied across studies, and we learned how to estimate the mean effect size and the heterogeneity in effects. In the advanced course we return to those data sets and learn how to explain some of that heterogeneity using subgroup analysis and meta-regression (analogous to ANOVA and multiple regression). We also learn how to assess the potential impact of publication bias in these analyses.

##### Comparing subgroups

In a primary study we may use a t-test or a one-way analysis of variance to compare the mean score in two or more groups of people. In a meta-analysis we may use a subgroup analysis to compare the mean effect size in two or more subgroups of studies.

##### Case study: The ADHD Analysis

We will revisit the analysis to look at the relationship between dose and patient type

##### Case study: The Mitral-valve analysis

We will revisit the analysis and compare the risk for studies that employed different variants of the procedure

##### Case study: St. John’s Wort analysis

We will revisit the analysis to compare the effect size for different types of patients

##### Case study: Reading

We will revisit the analysis to see if the correlation varies depending on the child’s age

##### Case study: Tocilizumab

We will revisit the analysis to see if the effect varies the type of patient

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

In this module students will learn

- How to run a subgroups analysis
- How to select a statistical model for a subgroups analysis
- How to estimate Tau-squared in the presence of subgroups
- That subgroup comparisons are observational
- How to report the results of a subgroup analysis
- How to avoid common mistakes when comparing subgroups

## Advanced Course (Module 6)

Meta-regression –

Assessing the relationship of covariates with effect size

Expand

##### Overview

In a primary study we may use multiple regression to assess the relationship between covariates and scores. In a meta-analysis we may use meta-regression to assess the relationship between covariates and effect sizes

##### Case study: The ADHD Analysis

We will revisit this analysis to look at the relationship between dose and effect sizes

##### Case study: The Mitral-valve Analysis

We will revisit this analysis to look for covariates related to the risk of death

##### Case study: The St. Johns’ Wort Analysis

We will revisit this analysis to look for covariates related to the effect size.

##### Case study: The Reading Analysis

We will revisit this analysis to look for covariates related to the effect size.

In each case we will isolate the unique impact of specific factors while controlling for potential confounds. We will discuss the meaning of tau-squared and prediction intervals in the presence of covariates. We will discuss the meaning and limitations of the R-squared analog. We will discuss how to plot the impact of continuous and categorical predictors, and interactions

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

In this module students will learn

- How to perform a meta-regression in CMA
- How to work with continuous covariates
- How to work with categorical covariates
- How to work with sets of covariates
- How to understand the I-sq analog
- How to understand the R-sq analog
- How to plot results of a meta-regression
- How to assess the unique impact of covariates
- How to use the Knapp-Hartung adjustment in meta-regression
- How to avoid common mistakes in meta-regression

## Advanced Course (Module 7)

Working with complex data sets

Multiple outcomes, time points, comparisons, subgroups

Expand

##### Complex data sets

A meta-analysis may include studies that report data for complex data sets. This may include data for two or more independent subgroups. It may include data for two or more outcomes or timepoints based on the same people. It may include studies that reported data for two or more treatments vs. a common control group.

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

In this module students will learn

- How to work with studies that present data for two or more independent subgroups
- How to work with studies that present data for two or more outcomes based on the same subjects
- How to work with studies that present data for two or more timepoints based on the same subjects
- How to work with studies that present data for studies that reported data for two or more treatments vs. a common control group.

## Advanced Course (Module 8)

Publication bias, limitations of the random effects model

Risk of bias, and other issues

Expand

##### Publication bias

Most meta-analyses are based on studies pulled from the literature. Since studies are more likely to be published if they report statistically significant results, there is a tendency for the literature to incorporate a biased subset of studies. It follows that a meta-analysis based on these studies will tend to overestimate the magnitude of the effect. We discuss how to address this problem.

##### Case study: The ADHD Analysis

##### Case study: The Mitral-valve Analysis

##### Case study: The St. Johns’ Wort Analysis

##### Case study: The Reading Analysis

##### Case study: The tocilizumab analysis

##### Limitations of the random-effects model

While meta-analysis provides an exceptional tool, it is imperative that we understand the limitations of this tool. We will discuss the limitations of the random-effects model. In particular, we will discuss limitations that apply when we are working with a small number of studies, and how to deal with these cases.

##### Exercise

Practice exercise and quiz

for the issues discussed in this module

##### Learning Objectives

In this module students will learn

- How to assess the potential impact of publication bias
- How to generalize from the studies in the analysis to a wider universe of studies
- How to work with analyses when there are only a small number of studies
- How to use the Knapp-Hartung adjustment
- When it makes sense to perform a meta-analysis
- How to avoid common mistakes associated with these issues

###### Testimonials

"Michael Borenstein is an extremely clear teacher. He explains meta-analyses procedures in a systematic and 'simple' way, meaning that he describes the concepts without complicated words. I got a good overview of how to use CMA in my future work. Thank you for the support to develop more self-efficacy with respect to performing meta-analyses and interpreting the results."

**Silvia Titze - University of Graz, Austria**

"The course content is wonderful and Dr. Borenstein is extremely patient and helpful in his interactions. Even though I have undertaken Meta-Analysis studies, I did pick up additional nuances that add to the clarity of understanding. Thanks a lot Dr. Borenstein!"

**Moutusi Maity - IIM Lucknow - Sydney 2020**