equation systems for variables regression

In equation (5), β_cap_zX is a column vector of size (k x 1) where k is the number of regression coefficients in the model (not including z), and γ is a scalar. The expected value of the fitted coefficients when a variable is omitted from the regression (Image by Author)

To simply it, we’ll use the identity E( AB )=E( A )E( B ) : Now let’s look at the second term of equation (4). So far, we have shown that in equation (4), the first term on the right is the column vector β and the third term is the column vector 0, both of size (k x 1). Thus, the third term in equation (4) is effectively extinguished into a column vector of zeroes of size (k x 1). The product of this matrix with the (k x 1) column vector of zeroes is simply a column vector of zeroes of size (k x 1). So the yellow colored bit equates to a matrix of size (k x k). Finally, the yellow colored bit is the inverse of product of X of size (n x k) and its transpose of size (k x n). Thus, the green and gray bits multiplied together is the column vector 0 of size (k x 1). The green colored expectation is simply X’ (which is the transpose of X) and it is of size (k x n). This property of the errors being exogenous implies that the gray colored expectation E( ϵ | X )= 0, where 0 is a column vector of size (n x 1) containing only zeroes.

One of the primary assumptions of the linear regression model is that the errors ϵ, conditioned upon the regression variables X, have a zero mean. The third term of equation (4) simplified using the product of expectations rule (Image by Author) In the above equation, we will substitute y with Xβ + zγ + ϵ from equation (1) as follows: ( X’X ), which is of size (k x k), when multiplied with X’ of size (k x n), yields a matrix of size (k x n) which when multiplied with y of size (n x1) yields a matrix of size (k x 1), which is exactly the dimensions of β_cap. y is a column vector of observed values of size (n x 1).Recollect that the product of two matrices of size (k x n) and (n x k) is a matrix of size (k x k). It’s as if X has been turned on its side. X with its rows and columns interchanged. X is a matrix of regression variables of size (n x k).β_cap is a column vector of fitted regression coefficients of size (k x 1) assuming there are k regression variables in the model including the intercept but excluding the variable that we have omitted.The closed form solution of y = Xβ_cap + e (Image by Author) If we solve the rest of equation (2), namely y = Xβ_cap + e by minimizing the sum of squares of residual errors e, it has a beautiful closed form solution that can be expressed in matrix notation as follows: This is akin to leaving out the term zγ from equation (1) or zγ_cap term from equation (2). Suppose the we fail to include the variable Engine_Size while building the model. Here’s the equation for the fitted model:Ĭity_MPG = β_1_cap + β_2_cap*Car_Volume+ β_3_cap*Curb_Weight + β_4_cap*Engine_Size + e Let’s revisit the regression model for the automobiles data set:Ĭity_MPG = β_1 + β_2*Car_Volume+ β_3*Curb_Weight + β_4*Engine_Size + ϵ The effect of omitting a regression variable We have now prepared the ground for addressing the problem of what happens when you omit regression variables. The ith residual error e_i is the difference between the ith observation y_i and the corresponding ith predicted value from the fitted model. Also in equation (2), the column vector of residual errors e takes the place of the column vector of regression errors ϵ. the estimates of the corresponding population level values of β and γ. Notice the cap or hat “^” symbol over β and γ indicating that they are the fitted values i.e. The fitted partitioned linear regression model (Image by Author)

Equation systems for variables regression