You have two unknowns and two equations. Easily solved.

...

Let x = the 1st number

Let y = the 2nd number

...

x = y - 3

x^2 + y^2 = 89

...

In a system of two equations with two unknowns, we normally rewrite one of the equations so that one of the unknowns (variables) is expressed in terms of the other. But we already have that with our 1st equation. The 2nd step is to substitute that equality into the 2nd equation:

...

So, plug x = y - 3 into:

x^2 + y^2 = 89

(y - 3)^2 + y^2 = 89

...

expand the (y - 3)^2 term to rewrite the whole equation:

y^2 - 6y + 9 + y^2 = 89

...

simplify like terms and subtract 89 from both sides so we end up with a quadratic equation in standard form to solve:

2y^2 - 6y - 80 = 0

...

divide both sides by 2 to simplify further:

y^2 - 3y - 40 = 0

...

factor the quadratic equation above to solve:

(y + 5) (y - 8) = 0

...

y = -5 or y = 8 are possible solutions to this equation.

...

we still need to calculate x and then check our work. Recall our 1st equation:

x = y - 3

...

x = -5 - 3

x = -8, therefore:

x = -8, y = -5

...

or:

x = 8 - 3, 

x = 5, therefore:

x = 5, y = 8