Just chill, my nigga. ;) Peaking over someone's fence ain't exactly the same as rippin' a neighbor in the middle of the night. Curiosity is curiosity. They see, smell, what you have and wanted to confirm. He might even want to tell you how beautiful your plants are. Who knows?
Bring him a beer...