I disagree that 'Physics is really Math'.
Physics might be expressed with mathematics, yes, but there is far more to physics than just its mathematical expression. It also requires physical intuition,
which is a kind of sixth sense - a deep gut-feeling - that a physicist develops, and which enables him to understand and think about the physical world in non-verbal and non-mathematical ways.
It is this intuition that guides a physicist in selecting, out of the myriad possible mathematical theories that might describe some universe or other, the small subset of mathematical theories that might describe our universe - i.e. those that are candidates for a description of our reality. The vast majority of theories that might be mathematically consistent are not consistent with physical reality as we know it - a fact that physicists forget at their peril, as many have done in the past. And if all that was required to develop physics was mathematics, then all mathematicians would be good physicists, and we would have no need for the latter. Happily, this is patently not the case.
And the final statement in the question, 'Math is really hard', is just a subjective opinion. Yes, mathematics is a very demanding discipline, intellectually and academically, for most people, and it takes a long time to study and master. But there are many other skills that might be just as challenging to master, and therefore justify being described as 'hard' by someone who is unskilled in them, even if they do not make the same intellectual or academic demands as mathematics. (For example, I would love to be able to play the piano to concert performance standard, but know that it would take years of practice - probably as many as required for a mathematics degree, if not more). So I guess my ending would be....
"...Math is really hard,
But then so are a lot of things in life, so man-up and get on with it!" :o)
In a similar vein to the question, a university friend of mine who was studying chemistry used to belittle my subject, theoretical physics, by saying it was 'just hard sums' [1]. My usual riposte to this was the following:
"Chemists are people who don't know enough mathematics to be physicists, and don't know enough Latin to be biologists!" [2]
[1] 'Sums' is an informal term in British English for mathematics, usually at an elementary school level where it is rarely more than arithmetic, and so the term was being used in a (deeply) derogatory sense by my friend.
[2] Apologies to any chemists on the thread :o) (and to biologists - I know that biology requires more than just a grasp of Latin, just as physics requires more than just a grasp of mathematics).