Beyond ANCOVA: A New Method for Excluding the Influence of Covariates in Comparing Group Means
by
Jeffrey S. Kane, Ph. D.
Professional Statistical Services
Abstract
Analysis of covariance purports to exclude the influence of one or more covariates from the statistical comparison of the levels of an independent variable on their mean dependent variable values. The method relies on the assumption that the pooled within-group slope of the regression of the dependent variable on the covariate(s) accurately reflects the dependent variable-covariate relationship at all levels of the independent variable. Data that fails to satisfy this assumption is commonly encountered and renders ANCOVA inapplicable, leaving the analyst without recourse in pursuing the objective of excluding covariate variance from the comparison of group means. Even where within-group regression coefficients for the covariate do not differ significantly, to whatever extent that they depart from exact equality, ANCOVA will fail to fully exclude the extraneous variance. Failure to fully exclude such extraneous variance causes inflation of the error term, thereby reducing the method's power and increasing its susceptibility to Type II error. A new method, analysis of covariate residuals (ANCOVRES), is proposed that uses the within-group slope for each independent variable level as the basis for completely removing covariate variance from the dependent variable within the respective level. This approach maximizes the power of ANOVA or t-tests applied to detect the significance of group differences on the covariate-independent portion of the dependent variable. More generally, it provides a method of excluding extraneous covariate variance from the comparison of group means that is not subject to ANCOVA's restrictive assumption of homogeneity of regression slopes.
Consider the common situation in which pre- and post-treatment assessments are collected for treatment and control groups that are either samples from pre-existing populations or are the result of random assignment to condition. It is often the case that pre- and post-treatment assessment scores will be correlated with each other. In this situation it is entirely legitimate to seek to isolate the variance in the post-treatment scores that is not associated with pre-treatment scores in order to focus the treatment vs. control comparison exclusively on post-treatment status and its standard error. The legitimacy and indeed, appropriateness of excluding such extraneous variance from the dependent variable scores of groups being compared can be generalized to any covariates that do not cause, are not influenced by, and are not tantamount to being parallel or equivalent measures of, the independent variable that differentiates the groups.
The general statistical methodology available for excluding such extraneous covariate variance is analysis of covariance (ANCOVA). With the exception of its application under a rarely occurring condition (i.e., when the covariate-dependent variable relationship is identical in the groups being compared), ANCOVA is incapable of achieving the complete exclusion of covariate variance from the dependent variable scores on which group means are to be compared. The reasoning underlying this contention is presented below, followed by the proposal of a new method that satisfies the requirement of completely excluding covariate variance from group mean comparisons. For the sake of simplicity of exposition, this presentation will refer to the case of a single covariate and a single two category independent variable. However, the observations and remedies offered are generalizable to analyses involving multiple covariates, independent variables with more than two categories, and/or multiple independent variables.
ANCOVA's Inability to Completely Exclude Covariate Variance
ANCOVA purports to exclude covariate variance from the dependent variable by computing residual scores for the full sample from the prediction of the dependent variable by the pooled within-group regression of the dependent variable on the covariate. This use of the pooled within-group estimate of the covariate regression coefficient requires that the within-group regression coefficients do not differ from each other and hence, from the pooled within-group regression coefficient, to a significant, or even to any non-negligible, degree. This equality of within-group regression coefficients is conventionally tested by evaluating the significance of the independent variable-by-covariate interaction. When the latter test is significant, ANCOVA is rendered inapplicable, leaving the analyst with no way to accomplish the needed control of the covariate. However, even when the latter test fails to reach significance, differences between the within-group coefficients exceeding any negligible level will result in the residuals of the pooled within-group regression representing over-corrections for one group and under-corrections for the other group. This in turn will result in the inclusion of variance associated with the covariate in the dependent variable residual scores of each group defined on the independent variable. This covariate-associated residual dependent variable variance will have the effect of artifactually altering the differences between group means, inflating the error variance, or both.
A New Approach to Excluding a Covariate's Influence on Group Mean Comparisons
ANCOVA's inability to fully exclude the influence of a covariate from the comparison of group means on a dependent variable in the absence of perfect regression homogeneity is due to the method's reliance on the pooled group or full-sample estimates of the relationship between the dependent variable and the covariate. The alternative proposed here to remedy this deficiency begins by computing the residuals from the regression of the dependent variable on the covariate separately within each group. This method will produce scores within each group that are completely independent of the covariate, regardless of how widely the slopes differ between groups in the untransformed data.
After completing the process of computing the residuals from the within-group regressions, one is left with residual scores for which the mean is exactly zero in each group. In order to proceed with the comparison of the group means it is necessary to determine what the original means would be with the level of the covariate held constant between the groups. A reasonable choice for the constant level of the covariate to use in adjusting the group means is the combined sample mean of the covariate. Assuming approximately normal distributions of both groups on the covariate, the choice of the covariate mean as the constant level to apply to each of the groups positions the covariate adjustment point midway between the covariate means of the two groups. If the distribution of the covariate in one or more of the groups is skewed or contains outliers, the median could usefully be substituted for the mean.
The adjusted group means are then computed by substituting the constant level chosen for the covariate for the value of the covariate in each group's equation for the regression of the dependent variable on the covariate. These operations are expressed in equation 1.
(1)
The covariate adjusted mean for each group is then added to the respective group's residual scores. The resulting sets of group scores can then be compared using either ANOVA or (when the number of groups = 2) a t-test.
A shortcut for computing these same scores is to apply equation 2 to the data.
(2)
It is proposed that this new analytical method be named analysis of covariate residuals, the natural acronym for which is ANCOVRES. The advantage of this method is that it completely removes the influence of a covariate from the comparison of group means regardless of the degree to which the regression slopes differ between the groups. Even where the degree of regression heterogeneity does not reach significance, this method is still preferable to ANCOVA because it produces more accurate estimates of the group factor and error sums of squares in all cases except where perfect regression homogeneity exists, which is very rare.
A number of authors (e.g., (Maxwell et al., 1985; Keppel, 1991; Winer, Brown, & Michels, 1991) have contended that when regression residuals are analyzed by ANOVA or t-test, the error degrees of freedom should be reduced by one because the slopes are computed from the same data. However, degrees of freedom are defined as the number of values in the final calculation of a statistic that are free to vary. It is not at all clear to this author why the computation of the mean-adjusted residuals by the proposed procedure would cause one of the resulting values to not be free to vary. Specifically, knowing n-1 of the residuals in any group would not enable one to predict the nth value. This may be a concern when a procedure uses the pooled within-group regression slope, but it does not appear to be applicable to the proposed method.
A Computational Example of ANCOVRES
The example to be presented will use the pretest as the covariate and pretest –posttest difference as the dependent variable. These variables were purposely chosen to ensure a strong covariate influence on the dependent variable in illustrating the relatively efficacy of ANCOVA and ANCOVRES in excluding covariate variance.
The data for the example fails to formally satisfy the regression homogeneity assumption (i.e., the p-value of the test for the assumption's violation is less than.05). Figure 1 illustrates the within-group regressions of the change score on the pretest for the raw data, revealing the substantial difference in slopes within each of the study groups as well as the appreciable difference from zero of each group's slope.
Figure 1. Within-group regressions of unmodified pre-post change scores on the pretest in non-homogeneous regression data.
Tables 1 and 2 present the results of applying ANOVA, ANCOVA, and ANCOVRES to this data, revealing that both ANOVA and ANCOVA failed to find the group effect to be significant. In contrast, the p-value for this effect was well below conventional .05 threshold for significance in the result produced by the ANCOVRES method.
Table 1
Comparison of ANOVA, ANCOVA, and ANCOVRES Results for Data that Violates the Regression Homogeneity Assumption: Parts A and B
|
Part A. Descriptive Statistics
Part B. Tests of Regression Homogeneity Assumption ANOVA test of interaction term:
|
Table 2
Comparison of ANOVA, ANCOVA, and ANCOVRES Results for Data that Violates the Regression Homogeneity Assumption: Parts C, D, and E
|
Part C. Within-group Regressions of Change on Pretest
Part D. Comparison of ANOVA, ANCOVA, and ANCOVRES Results
Part E. Adjusted Change Score Means
|
The inability of ANCOVA to detect the group difference as significant is entirely due to the failure of the pooled within-group regression to adequately remove covariate variance, which resulted in virtually no reduction in mean square error relative to its size in the ANOVA. This is graphically illustrated by Figure 2 in which the differences between the slopes of the regressions of the dependent variable on the ANCOVA residuals (i.e., from the pooled within-group regression) are essentially unchanged from the difference that existed before applying the ANCOVA method.
Figure 2. Within-group regressions of ANCOVA pooled regression residuals of pre-post change on pretest in non-homogeneous regression data.
Since the full-sample correlation of the pre-post change with the pretest was only .161, the pooled within group regression failed to remove any appreciable amount of the pretest variance from the pre-post change scores within each group. In contrast, the use of the ANCOVRES method resulted in slopes of zero in each of the groups, as shown in Figure 3.
Figure 3. Within-group regressions of ANCOVRES within-group residualized pre-post change scores on the pretest in non-homogeneous regression data.
Extending ANCOVRES to Larger Models
The proposed method can be extended to the use of multiple covariates. A separate multiple linear regression can be computed for each level of the independent variable and used as the basis for computing the residual scores for the respective level. This method is not restricted to the use of a linear model; polynomial terms can be included, although they tend to capitalize on error to a greater extent, which could lead to excessive reduction of the error term and increased Type I error.
The method can also be used with multiple independent variables. An appropriate way to proceed would be to parallel the steps of a Type II sum of squares ANOVA. Specifically, separate ANOVAs would be conducted for each independent variable and for each interaction effect. For further details on this procedure, please contact the author.
Conclusion
ANCOVA fails to completely exclude the influence of covariates on the response variable within each of the levels of the independent variable except in the rare case where the regression slopes reflecting this influence are identical across all independent variable levels. When the difference between such slopes reaches statistical significance, convention stipulates that ANCOVA is rendered inapplicable. ANCOVRES, the new method proposed here, is applicable regardless of the degree of difference in regression slopes between independent variable levels. Consequently, the proposed method constitutes the first generally applicable recourse in situations where the regression homogeneity assumption of ANCOVA is formally violated. In addition, even where the differences between regression slopes do not reach statistical significance but are nevertheless non-negligible, ANCOVRES offers a more powerful and precise method of assessing group differences on the response variable free from the influence of covariates considered to be extraneous.
References
- Keppel, G. (1991). Design and Analysis: A Researcher’s Handbook. Upper Saddle River, New Jersey: Prentice Hall.
- Maxwell, S. E., Delaney, H. D., & Manheimer, J. M. (1985). ANOVA of residuals and ANCOVA: Correcting an illusion by using model comparisons and graphs. Journal of Educational Statistics, 10, 197-209.
- Winer, B.J., Brown, D.R., & Michels, K.M. (1991). Statistical Principles in Experimental Design (3rd Ed.). McGraw-Hill, New York.