After taking both chemistry and physics in high school and then again in college (albeit at a higher level), I can say with absolute certainty that physics is harder, especially if it is calculus based physics. Trigonometry based physics vs. chemistry is a little bit more even, but I still say physics is tougher.