First thing that came to mind was....smoking. Both of my parents were smokers. That's a habit I vowed never to pick up. So I never started.
No offense to people who DO smoke, by the way. :)
One thing that could set me off is....someone invading my space....and they won't get off my back. I'm a solitary person so I don't want someone constantly bugging me. Would drive me insane.