Thats correct - Only things you allow to annoy you can really be annoying. Its like at work: So many things go wrong every day, you just have to deal with it and stay calm - right?
We've got the Five Precepts, and the fifth of them says, "As all Buddhas refrained from alcohol until the end of their lives, so I too will refrain from alcohol until the end of my life" (in Chinese texts) or "I undertake the training rule to abstain from fermented drink that causes heedlessness" (in Pali texts). So, no alcohol, but cigarettes theoretically are allowed :)