You must mean the square is inscribed in circle. We will give solution to the following problem:

The side of an equilateral triangle is a. A circle is inscribed in the triangle and a square is inscribed in circle. Find the area of the square.

the radius of the circle inscribe is r = a(√3/6).

the diagonal of the square d = 2r

the area of the square is A = d^2/2

A = (1/2)(2a(√3/6))^2

A = (1/6)a^2