Yes, it is a seriously bad thing, your brother is right(teasing), LOL!
In all seriousness, it could be or not at all, it depends on what we're talking about. I wouldn't want someone rubbing their clothes against my flowering plants, people are disgusting, pet hair, dander, Febreeze, cologne...