Not them again....Ugh?

In this case, if it were me, I would definitely have a heart to heart with the owner. Explain your concerns and let her know that if there is a question, to bring it to you first and foremost with an open mind. Hopefully that will nip any problems with your employ there. As far as other people go, there's always gossip and the horse world is not immune from it. People talk, there's nothing you can do to change it. If someone says something to you, just explain that you are not at liberty to discuss rumors and if they feel the need, they should talk to the barn owner (also let the barn owner know, that any gossip about you will be referred to her). If you don't have a need to be worried, then don't be, just keep the communication open and clear between you and the barn owner. Don't get into unnecessary arguments with people in the barn. Those that know and respect you will give you the benefit of the doubt. Now is a good time for you to see who is really a friend or foe. I call it "cleaning house".