I believed the same thing until I started eating for real. Two years ago I thought I was doing all I could to put on weight and everybody was telling me how much I ate. One day I chose to really commit to a 6-meal per day plan and just stick to it. For two full months, I did not go up a single digit on the scale. Just as I was starting to give up, my weight starting changing and withing about 3-4 months I went from 138 to 150 lbs. I've always been naturally lean, and before that I was also very scrawny. Now I'm not anywhere near big, but my current muscle mass suits my needs and I don't want to increase it any further. You sound like you have a body type similar to mine, so accept the fact that your metabolism needs some time to ''gear in'' for weight gain before you actually start progressing. Just stick to whatever you're doing and let it happen.
Also, I tend to not lose weight even if I'm not eating too well for extended periods of time so I wouldn't worry about weight loss in your case. You can eat what you want and remain lean, but unless you're SEVERELY under eating for many months at a time I don't fear that you will lose much. If you do, it will simply be your muscles and liver losing some stored glycogen and not actual muscle going away.