It depends on where you're going to college, too. I'm going to a very good college (not ivy leagues, but most of the people who get rejected from the ivys come here), and you really need math to do well in physics. The good news is that our department is designed knowing that most people won't have the math skills they think they do, so in your sophomore year you take two math-in-physics courses that are brutal, but catch everybody up with what they need to know for the upper level courses.
I would not agree with everyone that you need to be really good at math when it's taught in math departments to do well in physics. At least here, they're different types of things completely. In the math department, they teach purely from the level of equations and abstract things, and it can involve a lot of annoying memorization... learning formulas (
divergence theorem and
Stokes' theorem and whatnot--but don't understand if you can't understand the wiki. I have had to use these things constantly, and I still can barely tell what they're talking about) and such.
The physics department, on the other hand, teaches math from the standpoint of "you've just gotta be able to do it." As long as you can take the derivatives, or do the integrals, you'll be fine. I honestly don't think I have any more of a "concept" of a derivative of a function than "the slope of the tangent line to the curve at every point," and nothing more about an integral than "the area between the curve and the axis," and I've been fine. You really do have to be good at grinding through the calculations if someone says "take this derivatiive" or "solve this differential equation", though, so I would suggest that you take 2-3 calc courses immediately, to see whether you pick them up better in college. Also: AP Physics B is not even close to a real college physics couse, and AP Physics C is like a watered down college physics course. I would not recomend skipping them unless you're going to a shitty school. In fact, most college courses aren't too much like the HS AP ones.
I would also counter what Abe said about too many physics majors, because it's ridiculous. What
is true is that, when I go from company to company asking for jobs, they tend to say "oh... we're actually just looking for engineers right now." The same thing goes for any natural science major, because they tend to be broader thinkers, but companies just want things done. However, the research opportunities seem to be plentiful. I'm going to be a senior next year, and I got an internship this summer helping with a group of people trying to model the spread of diseases. The fact is that most
research initiatives need people from every discipline (there are physicists, biologists, statisticians, computer scientists, lawyers, public policy majors, etc. My boss is actually a computational quantum chemist) involved in the program, and they kind of fit you wherever your skills are. I have a lot of programing and linux background, and I really like turning simulations into pictures... so I ended up writing visualization software to turn the output from the disease modeling program into an internactive Google Earth presentation. I absolutely love it, haha.
Ive also been told that, if you want to go to grad school for physics, you basically never have to pay for it. Schools will end up paying you because they need TAs and things for their undergrad physics courses... so you go there, teach a class, take a few, and do some research at the same time, and it tends to be for free (especially if you're going for your PhD). I haven't actually looked into this for myself, yet... but I can attest to the fact that the schools where the people I'm helping with the research work seem to be pushing pretty hard to get me to apply there this year. So while industry can't seem to find a use for physicists, academia doesn't seem to have that problem.
I would recommend learning how to program too, though. I think that if you lacked both programming and math, you'd have a heck of a lot harder time with things than I did, because programming is essential if you're running simulations, and math is essential if you're dipping into "higher" forms of physics-stuff.