Breaking with social conventions and expectations can be a terrifying thing for some people. The fear that you could be targeted by those who disagree with anyone breaking social mores is valid, however, with poly communities in much of the Western world, you'll not be alone in deciding to live differently to social convention.
What to do about such fear? Well I'd recommend finding a poly community and talking to them about identification issues and safety, and how they deal with it in their local area.
There are poly communities in Australia, the United States, the UK, Canada, South Africa, France, Portugal, Mexico, Denmark, Austria, Germany, Netherlands, the Philippines, Spain, Switzerland and Sweden just to name a few (http://www.polyamorysociety.
Many of the poly people I know don't disclose their lifestyle choices beyond their immediate circle of intimates, some may extend that to family and friends, but not many people publicly identify as being poly. I suspect that this may be because of a lack of good poly role-models, and some of this being due to the whole poly concept being relatively new as far as social movements go.
So, apart from bigotry and in some very conservative communities the risk of losing your employment and children, what else have you to fear?
Some people, and I've been amongst them, are afraid that with the increased number of relationships is the increased potential for break ups and hurt occurs. To put it simply, yes that is true.
But what this simple explanation does not take into account is that with the increased number of relationships, there is an increased amount of support, and that for those who join a poly community, exist another support network, because most of them have been through it too at some point.
Remember, even monogamy is a world of hurt and break up, very few people marry the first person they dated. It is very different being involved in one break up and yet having another deeply happy and loving relationship existing to return to, and to be able to cry on your partner's shoulder about the partner that has left, died, etc.
So there are high highs and low lows, but I think that in this field, polyamory actually comes across better off than monogamy.
This one can be quite scary. As I've suggested before, and certainly have experienced, a monogamous relationship challenges your personal growth to a certain point before comfort, other and self knowledge set in. Not being someone with a long history (or any real history) of multiple monogamous relationships, I suppose I'm not the best qualified to discuss how this works outside my own experience, however polyamory is a real challenge to personal growth.
Its harder to explain away to yourself and your partner quirks, blind spots and baggage when it is obvious to more than two of you. That can be rather scary and hard work. In the end though, it is worth it. I undertook 2 years of good counselling to deal with some of my baggage and blind spots, because I could see that they were getting in the way.
(Actually I recommend good counselling to everyone, monogamous or polyamorous or even asexual. Good counselling is good for you. Very few people have issue free childhoods after all.)
For many of the poly people I know, and for those I am in relationships with, polyamory has increased our self awareness and emotional intelligence and certainly has held up big signs pointing to issues that need to be resolved. Its not all smooth sailing, but the rewards are awesome in the end.
I've certainly found, despite the need to work hard on my own shit, that I'm now a much more secure, confident and powerful individual than when I first entered the world of polyamory. This is my experience and the experience of others I know.
Do the benefits outweigh the costs?
I'm specifically referring to fear here, and really this is something that I can't tell you and you have to judge for yourself. However, if the fear of personal growth and potential hurt is something you are prepared to accept, the next step is dealing with society.
You don't have to tell the world that you are poly, you can live a quiet and unassuming life. You can choose also, to stand up to the world and tell those who disagree with you to go away and leave you alone. In the end, its your choice.