Matrix Algebra Review
Definition. An m*n matrix, Am*n, is a rectangular array of real
numbers with m rows and n
columns. Element in the ith row and the jth column is denoted by aij .
Definition. A vector a of length n is an n*1 matrix with each element
denoted by ai. The ith
element is called the ith component of the vector and n is the dimensionality.
2. A matrix may also be multiplied by a constant c. The product cA is the
matrix that results
from multiplying each element of A by c. Thus
3. The transpose operation A T or A' of a matrix changes the columns into rows
so that the first
column of A becomes the first row of AT , the second column becomes second row, and etc.
So the (i,j)th element in Am*n becomes the (j,i)th in the transpose
4. We can define matrix multiplication A B if the number of elements in a row
of A is the same
as the number of elements in the columns of B. E.g. when A is (p*k) and B is (k*n). An
element of the new matrix AB is formed by taking the inner product of each row of A with
each column of B. The matrix product AB is
Special Square Matrix
1. A square matrix A is said to be symmetric if A = AT or aij= aji for all i and j.
is not symmetric
2. Diagonal Matrix.
3. Identity Matrix.
The identity matrix is a square matrix with ones on the diagonal and zeros
elsewhere . It
follows from the definition of matrix multiplication that the (i, j) enrty of AI is
. So AI = A . Similarly, IA = A .
Therefore matrix I acts like 1 in ordinary multiplication.
The fundamental scalar relation about the existence of an inverse number a-1
, has the following matrix algebra extension .
then B is called the inverse of A and is denoted by A-1.
Other Matrix Properties
1. Trace. The sum of the diagonal elements,
2. A square matrix that does not have a matrix inverse is called a singular
The inverse of a 2*2 matrix is given by
3. A matrix is singular if and only if its determinant is 0. The determinant
of a matrix A is
denoted as |A| The determinant of a 2*2 matrix is given by
Examples 1. Simultaneous equations
We can rewrite the above three equations as a single matrix equation.
Example 2. Variance/Covariance Matrix
For a vector of random variables,
we can write a matrix containing their variances
and their covariances. Let be the variance of Yi and let covij be the covariance between Yi and
Yj , i < j. Then the variance/covariance matrix for is
Also note that the above can be written as
where is the
correlation of Yi and Yj . Note that all of these matrices
are symmetric. Furthermore,
the terms on the diagonal of the variance/covariance matrix must be positive and terms off
the diagonal of the correlation matrix are bounded by -1 and 1.
Example 3. Multiple Linear Regression
We have a response Y and a set of p independnet variables
X 1,X2, . . . ,Xp. Assume we have n
observations and for each i = 1, . . . , n observation, we assume
where Normal (0,).
We can represent n observations simultaneously in matrix form as
The residual can be written as
The residual sum of squares (RSS) is
Minimizing the above RSS gives the usual least -squared estimate for β