I'd say that some of the muscles used mostly are shoulders and back[especially directly under your shoulder
]. Also, make sure that your grip is strong enough in order to hang on to the pole.
That's my thoughts. Maybe try the fitness forum for more help. 
Actually, for the flag you need side leg muscles to keep you legs up, and basically all the muscles on the side, but I guess it's true that you need lots of shoulder and arm muscles.
By the way, if your tall, that doesn't mean it's harder, because usually if your taller you will have more muscle. So it all evens out. You still have the same muscle/weight ratio (unless your comparing to people who work out extremely). Just keep working on getting stronger until you can do it.