Most or all of the ones who came from the South. Slavery was the dominant labor system in those states, and the basis of their (primarily agricultural) economy. Washingon, Jefferson, Madison, Monroe, Mason, Henry, and the rest of the Virginia crew were all slaveowners, most of them inheriting slaves from their families. The same applies in the Carolinas, Maryland, and Georgia.
Fewer New Yorkers held slaves—it was in the North primarily a condition of domestic servants rather than agricultural workers, and was on its way out by the late 18th Century.