Matrix Algebra Review

Definition. An m*n matrix, A_m*n, is a rectangular array of real numbers with m rows and n
columns. Element in the i^th row and the j^th column is denoted by a_ij .

Definition. A vector a of length n is an n*1 matrix with each element denoted by a_i. The i^th
element is called the i^th component of the vector and n is the dimensionality.

Matrix Operations

1. Two matrices A and B of the same dimensions can be added . The sum A + B has (i, j) entry
a_ij + b_ij . So

2. A matrix may also be multiplied by a constant c. The product cA is the matrix that results
from multiplying each element of A by c. Thus

3. The transpose operation A ^T or A' of a matrix changes the columns into rows so that the first
column of A becomes the first row of A^T , the second column becomes second row, and etc.
So the (i,j)^th element in A_m*n becomes the (j,i)^th in the transpose

4. We can define matrix multiplication A B if the number of elements in a row of A is the same
as the number of elements in the columns of B. E.g. when A is (p*k) and B is (k*n). An
element of the new matrix AB is formed by taking the inner product of each row of A with
each column of B. The matrix product AB is

Special Square Matrix

1. A square matrix A is said to be symmetric if A = A^T or a_ij= a_ji for all i and j.

is symmetric,

is not symmetric

2. Diagonal Matrix.

3. Identity Matrix.

The identity matrix is a square matrix with ones on the diagonal and zeros elsewhere . It
follows from the definition of matrix multiplication that the (i, j) enrty of AI is
. So AI = A . Similarly, IA = A .
Therefore matrix I acts like 1 in ordinary multiplication.

The fundamental scalar relation about the existence of an inverse number a^-1 such that
, has the following matrix algebra extension .

then B is called the inverse of A and is denoted by A^-1.

Other Matrix Properties

1. Trace. The sum of the diagonal elements,

2. A square matrix that does not have a matrix inverse is called a singular matrix
The inverse of a 2*2 matrix is given by

3. A matrix is singular if and only if its determinant is 0. The determinant of a matrix A is
denoted as |A| The determinant of a 2*2 matrix is given by

Examples 1. Simultaneous equations

We can rewrite the above three equations as a single matrix equation.

Example 2. Variance/Covariance Matrix

For a vector of random variables, , we can write a matrix containing their variances
and their covariances. Let be the variance of Y_i and let cov_ij be the covariance between Y_i and
Y_j , i < j. Then the variance/covariance matrix for is

Also note that the above can be written as

where is the correlation of Y_i and Y_j . Note that all of these matrices are symmetric. Furthermore,
the terms on the diagonal of the variance/covariance matrix must be positive and terms off
the diagonal of the correlation matrix are bounded by -1 and 1.

Example 3. Multiple Linear Regression

We have a response Y and a set of p independnet variables X ₁,X₂, . . . ,X_p. Assume we have n
observations and for each i = 1, . . . , n observation, we assume