I am a Georgia Tech alumnus and graduated in 2008 in materials science and engineering. I am very proud of my degree and would not go anywhere else, but you ask for some downsides and here they are:
1. Crime-The location of GT's campus in a urban setting lends itself to some recently well publicized crime. Personally, I think it is a bit overblown as the campus is very safe, but the areas West and South of campus do have some issues. If you are smart about it (i.e. don't be out alone late at night), you shouldn't have any issues.
2. Very difficult/You are on your own- GT is known as a research institute, and the majority of the professors there are to do research, not to teach. This leads to many of the classes being large classes (100+) teaching high end theory with very difficult tests that the professor then curves the results to. Grades in the 30s and 40s can actually be above average. I have never understood this method of teaching. I do appreciate that GT is hard, though, and take pride in the fact that it is not a degree mill like other prestigious universities that are difficult to get into but hard to get out of. There is definitely a weeding out process that takes places the first couple of years. It might not be for you.
Some other reasons you might see out there are the lack of girls (if you are a guy), the social life, and things of that nature. In my opinion, those things are what you make of it and there are more ways to make friends and acquaintances than just your classmates. You are in a major city with tons of career and social opportunities that very few schools can rival.
Feel free to let me know any additional questions you may have. Go Jackets!