Simple answer:
If you can get enough food to grow big muscles (they are really energy-intensive), then probably you can feed a family too. Or at least one more women.
The fact that women are better in surviving famines tell a tale about the effectiveness of food-gathering efficiency (lower metabolism and higher body-fat ratio).
The lower famine-resistance of men tell an other tale about the partner selection of women. The better hunters (or later farmers), who can share more food got more women. So those men could fornicate (and proliferate) more who have the least requirement of famine-surviving genetic (it has genetic evidence).
Big muscles means the owner can get many food and strong enough to protect a family. Women want to maximize the survival of their offsprings.
Mating with the strongest increase the number of surviving children.