The formula that it uses is based on an idea from calculus called Taylor's theorem. The idea is essentially to use properties of something called the derivative to approximate a function by polynomials of different degrees; the theorem then has some statements about how good these approximations are. In the case of cosine and sine, it turns out that this holds:
cos(x) = 1 - x^2/2 + x^4/4! - x^6/6! + ...
sin(x) = x - x^3/3! + x^5/5! - ...
where n! is n factorial = n*(n-1)*...*2*1. Note that these are only in radians; it is possible to do the computation in degrees, but the scale factors just get in the way, so it's easier to convert to radians and do the computations there.
The calculator takes this infinite series out until the next term is less than the decimal precision of the calculator and then stops. In the process (due to a theorem having to do with properties of convergent alternating series) the error from truncating the series is automatically less than the decimal precision of the calculator. (There is a little more error from doing arithmetic in finite precision as well, but that's unavoidable.)
The calculator might also take numbers outside the range [-pi,pi] and put them into that range by removing multiples of 2*pi first.
So to compute, say, sin(0.01):
sin(0.01)=0.01
-0.01^3/6 (~10^-7)
+0.01^5/120 (~10^-12)
-0.01^7/5040 (~10^-17)
and so we're actually done with the term before (the decimal precision of the calculator is typically at most 10^-16). So:
sin(0.01) = 0.01-0.01^3/6+0.01^5/120 - y
where 0 < y < 0.01^7/5040
The "good" formulae for tan(x) turn out to be more complicated than this, but a relatively efficient way to do it would just be to compute sin(x) and cos(x) and divide. This is probably not what calculators actually do, however.
