Simple
First Actor was cast for the first movie.
The first movie flopped, and when the series was rebooted, a new actor was cast.
For Avengers, the same actor as the second movie was supposed to be cast (Edward Norton), but he said he wanted more diversity in his carrier and did not want to repeat the same role again. So, the third actor (Mark Ruffalo) was cast. He is contracted to be the Hulk in i think 5 or 6 more movies.