1. Background of the StudyOutliers are observations that appear inconsistent with the remainder of the dataset. Bollen and Jackman [1] defined outliers as observations that are distinct from most of the data points in the sample. Hawkins [2] described outlier as an observation that deviates so much from other observations as to arouse suspicions that it was generated by different mechanisms. Dixon [3] sees outlier as values that are dubious in the eyes of the researcher and contaminants. Outliers are often present in real data but may go unnoticed because nowadays much data are produced by computer without careful inspection or screening. Outliers may be mistakes in data entry or otherwise, accurate but unexpected observations which could shed new light on the phenomenon under study.
In general, outliers can be classified into two types: man-made one and random one. Man-made outliers may arise because of typographical error(s), misreporting of information, incorrect distribution assumption and sampling error. Random outliers, on the other hand, may arise because of random chance for drawing sample from a population. Presence of man-made or random outliers or both would seriously influence the result of statistical analysis including point and interval estimates and type 1 and type 11 errors. Outlier can cause us to misinterpret patterns in plots; it can affect visual resolution of remaining data in plot (forces observations into clusters) and may indicate that our model fails to capture important characteristics of the data. Unusual cases can substantially influence the fit of the ordinary least square model thus, leading to faulty conclusion. Some man-made outliers can be avoided by a strict data entry and rechecking process before conducting a statistical analysis. Data transformation is another way to reduce the influence of outliers (see Barnett and Lewis [4]; Montgomery [5]).
Quite a number of authors have proposed different methods of detecting outliers.
Abuzaid et al. [6], proposed a number of diagrammatical plots and hypothesis testing to detect outliers in the simple circular regression model. The work focused on detecting outlier in the down and Mardia’s circular-circular regression model. Zhang et al. [7] presented a method which applies signal processing techniques to solve important problems in data mining. They introduced an outlier detection approach termed to find out, based on wavelet transform. The main idea in the method is to remove the clusters from the original data and then identify the outliers. Rousseeuw [8] proposed a depth based method to detect outlier. The data points are organized in layers in the data space according to the value of the point depth. Aggarwal and Yu [9] proposed a method which entails studying the projection from the rest of the data in a sparse data with high dimensionality. Arning et al. [10] introduced a method which relies on the observations such that after seeing a series of similar data, an element disturbing the series is considered an outlier.
Hodge and Austin [11] identified an efficient method for on-line classification and outlier detection in multivariate sensor data. This involved a comparative review to identify and distinguish their advantages and disadvantages and introducing a survey of contemporary outlier detection techniques. Sebert et al. [12] and Montgomery [5] asserts that to identify the existence of outliers the standardized residuals are computed and a large standardized residual of (d > 3) indicates outlier. Worden et al. [13] used the concept of discordances to signal deviance from the norm.
Kitagawa [14] and Wing-Kam Fung and Bacon-Shone [15] applied Akaike information criterion (AIC) in detection of outliers by using (quasi) Bayesian approach with predictive likelihood. Belsey et al. [16] suggested standardizing each residual with an estimate of its standard deviation that is independent of the residual. This is accomplished by using as the estimate of variance for the ith residual, the residual mean square from an analysis where that observation has been omitted. The result is a jackknife residual also called a fully studentized residual.
In this study, concentration is on the effect of outlier as well as detection methods on linear regression model. Specifically, we are concerned with observations that differ from the regression plane defined by the data. It is important to identify these types of outliers in regression modeling because the observations when undetected can lead to erroneous parameter estimates and inference from the model. Deleting outliers from a regression model can sometimes give completely different results as bias or distortion of estimates are removed. Identifying outliers in the real world data-base is important for improving the quality of original data and for reducing the impact of outliers. Identifying outliers and high-leverage points, is a fundamental step in the least-squares regression model building process. On this note, it is the purpose of this research to examine the effect of outliers on simple and multiple linear regression, compare different methods of detecting outlier and show its effect on modeling.
2. Methods of Outlier Detection and Effects on Regression ModelThere are many methods for the detection of outliers in linear regression. They may be classified into graphical and analytical methods. The graphical methods include Scatter graph, Boxplot, Williams graph, Rankit graph (or Q-Q Plot) and graph of predicted residuals. The analytical methods are predicted residuals, standardized residuals, studentized residuals, Jack-knife residuals, Cook’s distance, Different-in-fits (DFFITS) and Atkinson’s measure.
The general linear model is given as
y n × 1 = X n × k β k × 1 + ε n × 1 (2.1)
where,
y n × 1 is the observation (dependent variables);
X n × k is the design matrix including a constant;
ε ~ N ( 0 , σ 2 I ) is the error term;
β = ( β 0 , β 1 , β 2 , ⋯ , β k − 1 ) ′ is the coefficient and k is the number of covariates or predictors for each observation.
Then, the least square estimator is given as
β = ( X ′ X ) − 1 ( X ′ Y ) (2.2)
where the fitted (predicted) values for mean of Y are
Y ^ = Χ β = Χ ( Χ ′ Χ ) − 1 ( Χ ′ Y ) = Η Y (2.3)
where H = X ( X ′ X ) − 1 X ′ is the projection matrix (or hat matrix).
The ith diagonal element of H (given by h i = x ′ i ( Χ ′ Χ ) − 1 x i is known as the leverage of the ith observation).
Similarly, the ith element of the residual vector
ε i = Y i − Y ^ i = ( I − H ) Y i (2.4)
where H is as defined in Equation (2.3), I is the identity matrix and Yi is corresponding fitted value ( i = 1 , 2 , ⋯ , n ).
In this study, Outliers were detected using the following methods:
2.1. Studentized and Standardized ResidualsThe Studentized residual is obtained by
ε S . i = ε ⌢ i σ ^ 1 − h i = ε ⌢ i σ ^ 2 ( 1 − h i ) (2.5)
where σ ^ is an appropriate estimate of σ , and the estimate of σ ^ 2 (Mean Residual Sum of Squares) is the internally studentized residuals. h i is leverage points as already defined.
σ ^ 2 = 1 n − k ∑ j = 1 n ε ^ j (2.6)
where k is the number of parameters in the model (Equation (2.1)).
Studentized residuals with large absolute values are considered large. If the regression model is appropriate, with no outlying observations, each Studentized residual follows a t distribution with n − k − 1 degrees of freedom.
CRITICAL: Each deleted residual has a student’s t-distribution with n − k − 1 degrees of freedom.
If the Studentized residual is divided by the estimates of its standard error so that the outcome is a residual with zero mean and standard deviation one, it becomes standardized residual denoted by
ε S T . i = ε ⌢ i s d ( σ ) (2.7)
where s d ( σ ) is the standard deviation of σ ^ 2 in Equation (2.6).
CRITICAL: The standardized residuals, di > 3 potentially indicate outlier.
2.2. Jackknife ResidualsThe jackknife residuals are residuals which with an assumption of normality of errors have a student distribution with (n − k − 1) degrees of freedom. The formula is:
ε J . i = ε ⌢ S . i ( n − k − 1 ) ( n − k − ε ⌢ 2 S . i ) (2.8)
where ε S . i is the Studentized residuals Equation (2.5).
The jackknife residual examines the influence of individual point on the mean quadratic error of prediction.
CRITICAL: Each deleted residual has a student’s t-distribution with n − k − 1 degrees of freedom.
2.3. Predicted ResidualsThe predicted residual or Cross-validated residuals for observation i is defined as the residual for the ith observation that results from dropping the ith observation from the parameter estimates. The sum of squares of predicted residual errors is called the PRESS statistic:
ε P . i = ε i 1 − h i (2.9)
PRESS is called Prediction sum of squares; an assessment of your model’s predictive ability. PRESS, similar to the residual sum of squares, is the sum of squares of the prediction error. In least squares regression, PRESS is calculated with the following formula:
PRESS = ∑ i = 1 n ( ε i 1 − h i ) 2 (2.10)
where ε i = residual and hi = leverage value for the ith observation. In general, the smaller the PRESS value, the better the model’s predictive ability.
2.4. Cook’s DistanceCook’s distance Di of observation, i is defined as the sum of all the changes in the regression model when observation i is removed from it. Cook [17] proposed a statistic for detection of outlier, given as:
D i = ∑ j = 1 n ( y j − y ^ j ( i ) ) 2 k S 2 (2.11)
and S 2 = ( n − k ) − 1 ε ′ ε is the mean squared error of the regression model. Equivalently, it can be expressed using the leverage
D i = ε i 2 k S 2 [ h i ( 1 − h i ) 2 ] (2.12)
Here, Di measures the sum of squared changes in the predictions when observation “i” is not used in estimating β . Di approximately follows F(p, n − p) distribution.
CRITICAL: The cut off value of Cook-Statistic is 4/n.
2.5. Difference-in-Fit (DFFIT)It is the difference between the predicted responses from the model constructed using complete data and the predicted responses from the model constructed by setting the ith observation aside. It is similar to cook’s distance. Unlike cook’s distance, it does not look at all of the predicted values with the ith observation set aside. It looks only at the predicted values for the ith observation. It combines leverage and studentized residual (deleted t residuals) values into one overall measure of how unusual an observation is. DFFIT is computed as follows:
DFFIT = ε i [ n − k − 1 σ 2 ( 1 − h i ) − ε i 2 ] h i 1 − h i (2.13)
where ε i = residual, n = sample size, k = the number of parameters in the model, σ2 = variance and hi = leverage value for the ith observation.
CRITICAL: The cut off value of DFFIT is 2 k n .
2.6. Atkinson’s Measure (A<sub>i</sub>)It enhances the sensitivity of distance measures to high-leverage point. This modified version of cook’s measure Di suggested by Atkinson is even more closely related to Belsey et al. (1980) [16] DFFITS and has the form
A i = | ε J . i | [ n − k k × h i 1 − h i ] (2.14)
where n, k, hi are as defined in Equations (2.13) and ε J . i is the absolute value of Jackknife residuals.
This measure is also convenient for graphical interpretation.
2.7. Scatter Graph and Box PlotScatter plot is a line of best fit (alternatively called “trendline”) drawn in order to study the relationship between the variables measured. For a set of data variables (dimensions) X 1 , X 2 , ⋯ , X k , the scatter plot matrix shows all the pairwise scatter plots of the variables on the dependent variable.
A box plot is a method for graphically depicting groups of numerical data through their quartiles (i.e. Mean, Median Mode, quartiles). Box plots may also have lines extending vertically from the boxes (whiskers) indicating variability outside the upper and lower quartiles. It is also called box-and-whisker plot and box-and-whisker diagram. Outliers may be plotted as individual points and it can be used for outlier detection in regression model, where the primary aim here is not to fit a regression model but find out outliers using regression and to improve a regression model by removing the outliers.
2.8. Willams Graph, Rankit Graph (or Q-Q Plot) and Predicted Residuals GraphThe Williams graph (Williams [18]) has the diagonal elements Hii on the x-axis and the jackknife residuals ε ^ J i on the y-axis. Two boundary lines are drawn, the first for outlier y = t0.95(n − k − i), and the second for high leverages, x = 2k/n. Note: t0.95(n − k − 1) is the 95% quantile of the student distribution with (n − k − 1) degrees of freedom.
The Q-Q plot (or Rankit Graph) has the quantile of the standardized normal distribution μpi for P i = i / ( n + 1 ) on the x-axis and the ordered residuals ε ^ S . i , ε ^ P . i , ε ^ J . i i.e. increasingly ordered values of various types of residuals on the y-axis.
The graph of predicted residuals (or Predicted Residuals Graph) has the predicted residuals ε ^ P i on the x-axis and the ordinary residuals ε ^ i (Equation (3.4)) on the y-axis. The outlier can easily be detected by their location, as they lie outside the line y = x far from its central pattern (Meloun and Militky [19]).