Would they concern you more or less than the person walking past your house and looking into your front window? Maybe even standing there (hovering) and looking?
Me personally? I'd say more. Why? Well first of all it's somewhat socially unacceptable -- and therefore uncommon -- for people to stand and look in your front window. Also, one tends to have curtains there. And too, I could see that person.
A drone hovering over my patio? 1) I don't have "sky curtains." 2) They are anonymous and can be controlled from a long distance away. 3) A drone IS a camera. 4) They whine.
Does that make it illegal? Not necessarily (I don't know what the laws are.) Maybe other people would be fine or even happy with drones hovering over their backyard. But someone asked why it would be "worse" than an airplane or helicopter going over and I gave some reasons in my opinion.