My guess is that they, like most traceurs, were so enthusiastic about the discipline that they were happy to share it with you. You will find that most traceurs are kind of a breed apart from most people. There is a real sense of community and sharing; people care less about things like age and gender, and more about helping eachother progress, and sharing in this discipline we all love.
I was self-conscious too, and still am quite often. I am a 32-year-old woman and when I go to jams I am often the only woman (or we are in the distinct minority, say, 2-3 out of 20 or more... at Colorado there were 2 of us out of 70-100 guys), and I am often one of the oldest there. I started a group here in Madison and until recently I was the oldest person, most of our membership is college guys.
If you kept up with the conditioning you are doing fine. And you should definitely come back. As mentioned, you will find that traceurs tend to be an incredibly welcoming and supportive bunch.
As to your physician, hmph. I think she should trust you to know your own body. That said, there are risk factors that increase with age, things like bone density and whatnot, that, while they shouldn't stop you from doing what you enjoy, they should be at least kept in mind as you train. But you are an adult and are used to physical activity, and you don't strike me as an idiot, so I don't see why your doctor should be discouraging you.
Good luck on your journey!