Predictive Efficiency of Restricted and Mixed Regression Estimator

Amit Sharma1 and Manoj Kumar2*

1Department of Statistics,University of Jammu, Jammu - 180 006, INDIA.
2Corresponding Author, Department of Statistics,Panjab University, Chandigarh – 160 014, INDIA.
3Govt.College Bishrampur Surajpur (C. G.), INDIA.
4Department of Physics, S.R.M.G.P.C. Lucknow, INDIA.
amitstat99@gmail.com, mantiwa@gmail.com
(Received on: September 23, Accepted: October 7, 2017)

ABSTRACT

In this article we have considered the composite target function to expose the ridge regression and also the estimators obtained by utilizing the prior non-sample information for prediction purpose. Finally, the performance properties of the estimators are analyzed.

Keywords:Linear regression model, ridge regression estimator, restricted ridge estimator mixed regression estimator, target function, prior non-sample information.

INTRODUCTION

Generally predictions from a linear regression are made either for the actual values of the study variable or for the average values at a time, see. Rao and Toutenberg (1995). However situation may arise in which one may be required to consider the predictions of the both actual and the average values simultaneously. For example, manufacturer of a certain product would like to know the life of his product on the average. On the other hand, a customer who purchases this product would be more interested in knowing the actual life of the product, which he has purchased. Thus, the manufacturer is interested in the prediction of the average values but he may not completely ignore the interest of the customer in the predictions of actual value and so he may assign higher weightage to the prediction of actual values. Similarly, the customer may give more weightage to the prediction of actual value in comparison to that of the average values. Appreciating the need of simultaneous prediction of actual and average values of the study variable, Shalabh (1995) proposed a composite target function, which considers the prediction of both the actual and average values together.
But if the usual assumption of the full column rank of the design matrix X is violated in the linear regression model Y = Xβ +σU is violated, the problem of multicollinearity arises. This leads to unstable estimates of the regression coefficients, which yields unstable predictors, so they (predictors) are practically useless. To overcome this, different remedial actions have been proposed. A popular numerical technique to deal with multicollinearity is ridge regression proposed by Hoerl and Kennard (1970a,b). Consequently, several authors have suggested different estimators, see. E.g., Sarkar (1992, 1998), Liu(1993) and Kaciranlar et al. (1999), Akdeniz,F, Erol,H (2003).
In this paper the author consider the composite target function to expose ridge regression estimator and the estimator obtained by considering the prior non-sample information, for prediction purpose and analyses their performance properties The organization of the paper is as follows; section 2 deals with model specification and presents target function for the prediction of actual and average values of the study variables. In section 3 the author assumes the availability of prior non-sample information and formulates the estimators, viz., ridge regression restricted ridge and mixed ridge. In section 4 two predictors are presented and their properties are analyses. In section 5 the author compares the predictors on the basis of their properties, which were analyzed in section 4 and lastly the results of the theorem presented in section 4 are derived in section 6.

MODEL SPECIFICATIONS AND TARGET FUNCTION

Let us consider a linear regression model

Y = Xβ +σU             (2.1)

Where Y is a (n x 1) vector of n-observations on study variables; X is a (n x p)matrix with full column rank of observations on p-explanatory variables; β is a ( p x1) vector of regression coefficients; σ being an unknown positive scalar and U is a (n x1) vector of n- observable disturbances with mean zero and variance unity.
If b denotes the estimate ofb , then the predictor for values of the study variable with in the sample is generally formulated as T^ = Xbwhich is used for predicting either the actual value of Y or the average value E[Y] = Xβ at a time.

When the situation demands prediction for both the average and actual together, we may define the following target function
T(Y) =αY + (1-α )E(Y) = T        (2.2)

and use T^ = Xb for predicting it, where 0 ≤ α ≤1 is a non stochastic scalar specifying the weightage to be assigned to the prediction of actual and average values of the study variable,see.,Shalabh (1995)

PRIOR NON-SAMPLE INFORMATION

Suppose in addition to the sample information given in (2.1) there exists non-sample information which is in the form of stochastic restrictions i.e.
r = Rβ               (3.1)

Where r is a (J X 1) vector of known elements and R is a (J X p) full column rank matrix of known elements
If we ignore the restrictions (3.1) then the least squares estimator of β is bo = X + Y             (3.2)
Where X + Y = (X'X) X -1X'
And following the ridge regression method of estimation proposed by Hoerl & Kennard (1970a, b) the author shall write
b(k) =W-1X'Y             (3.3)
Where k is a non-negative scalar characterizing the estimator and W = (X'X + kI )
The estimator defined in (3.2) and (3.3) doesn't obey the restriction (3.1), it is not so with the restricted ridge estimator given by

bR(k) = b(k) - (X'X)-1R'[R(X'X)-1 R']-1[Rb(k) - r]        (3.4)
This obeys the restrictions (3.1)
Suppose the restrictions are stochastic in nature, i.e r = Rβ +V             (3.5)
Where V is a (J X 1) random vector with mean zero and variance-covariance matrix W which is positive definite and known.
Further it is assumed that U and V are stochastically independent The mixed ridge type estimator of β is
bM (k) = [I - kW -1] bM             (3.6)
Where bM = [X'X + S2R'Ω-1R]-1 [X'X + S2R'Ω-1r]            (3.7)
S 2 = 1/n - p((Y-Xβ)(Y-Xβ)')            (3.8)
Here S 2 is an unbiased estimator of s 2 ,see., Theil and Goldberger (1961)

PREDICTIONS

Employing (3.2), (3.3) and (3.6) yields the following predictors for the values of study variables

It is easy to see from (2.2) that both the predictors defined in (4.1) and (4.2) are biased with bias vectors

Further their mean squared error matrices and predictive risks are

Where λj 's are the Eigen values of X'X

In order to derive the expression for bias, mean squared error matrix and predictive risk of TM (k) , we employ the small disturbances asymptotic theory introduced by Kadane (1971).

Theorem: When the disturbances in the model (2.1) are normally distributed, the predictive bias, mean squared error matrix and risk of TM (k) up to order o(σ4 ) of approximations is given by

A COMPARISON

On comparing the expression (4.10) and (4.11), the author finds that up to the unit power of k and 0 ≤ a < 0.5, the predictive risk associated with T^R (k) is less than the predictive risk associated with T (k) , if the characterizing scalar k satisfies the constraint

and just reverse holds, if 0.5 < α < equal 1.
For α = 0.5,the predictive risk associated with T^R (k) is smaller than the predictive risk associated with T^(k) , if the lower bound of k is

On comparing the expression (4.10) and (4.14), the author finds that for a specified value of k suggested by Hoerl, Kennard and Baldwin (1975), the predictive risk associated with TM (k) is smaller than the predictive risk associated with T^MODEL SPECIFICATIONS AND TARGET FUNCTION(k) so long as

Case I: When the aim is to predict the average value of the study variable (α = 0)

On comparing the expression (5.4) and (5.5), the author finds that for the average value prediction the predictorT^ (k) R is better than that of T^(k) for a fixed value of k. On comparing the expression (5.4) and (5.6), the author finds that for the average value prediction the predictorTM (k) has smaller risk than that of T^(k) if
n > ( p + 2)

Case II:When the aim is to predict the actual values of the study variable (a = 1)
n > ( p + 2)

On comparing the expressions (5.7) and (5.8) the author finds that for actual value prediction, TR (k) have smaller risk than T^(k) , if the lower bound of k is

On comparing the expression (5.7) and (5.9) we see that

Thus for actual value prediction, TM (k) performs better thanTˆ(k) ,if the terms on the right hand side of the expression (5.7) are positive.

APPENDIX

Employing (2.1) and (2.2) in (3.6) provides

Utilizing the expression (6.6) and (6.7) in (6.1) yield the result (4.12) of the theorem Now


Utilizing expressions (6.9), (6.10), (6.11), (6.12) and (6.13) in (6.8) yields the result (4.13) of the theorem
Again

and thus result (4.14) of the theorem follows from the expression (6.14)

REFERENCES

  1. Hoerl, A.E and Kennard, R.W., "Ridge regression: biased estimation for non-orthogonal problems" Technometrics, 12, 55-67 (1970 a).
  2. Hoerl, A.E and Kennard, R.W., "Ridge regression: biased estimation for non-orthogonal problems" Technometrics, 12, 69-82 (1970 b).
  3. Hoerl, A.E and Kennard, R.W., and Baldwin, K.F.,"Ridge regression: some simulation", Commn. Stat., 4, 105-123 (1975).
  4. Kaciranlar,S., Sakallioglu, S., Akdeniz, F., Styan , G.P.H., and H.J., Werner. "A new biased estimator in linear regression and detailed analysis of widely analyses data on Portland Cement", Sankhya, Series B, 61, 443-459 (1999)
  5. Kadane, J.B., "Comparisons of k-class estimators when the disturbances are small", Econometrica, 39, 723-737 (1971).
  6. Rao, C.R., and H., Toutenburg. Linear Models: Least Squares and Alternatives, Springer, New York (1995).
  7. Shalabh. "Performance of Stein-Rule procedure for simultaneous prediction of actual and average values of the study variable in linear regression model", Bulletin of the International Statistical Institute 56, 1375-90 (1995).
  8. Sarkar, N., "A new estimator combining the ridge regression and restricted least squares methods of estimation" Comm. in Statistics, Series A, 21, 1987-2000 (1992).
  9. Sarkar, N., Erratum: "A new estimator combining the ridge regression and restricted least squares methods of estimation" Comm. in Statistics, Series A, 27, 1019-1020 (1998).
  10. Theil, H and Goldberger, A.S., "On pure and mixed estimation in econometrics", International Economic Review 2, 65-78 (1961).
  11. Akdeniz, F and Erol, H. "Mean Squared Error Matrix Comparison of Some Biased Estimators in linear Regression". Communication in Statistics-theory and Methods 32 (7), 2389-2413 (2003).
map1 map2 map3