I'm not responsible for the way other people choose to live their lives, or what they choose to believe. Neither are you.
If you really believed that then your neighbour could go out killing, maiming, stealing or performing any other socio- or psychopathic action without any intervention from yourself; sorry, but I am part of society (the communal effort that allows humans to live together) and part of being part of society is intervening when another's actions threaten that society. A failure to act or to speak up contributes, however infinitesimally, to the degradation of society. You claim to be aware of Niemoller's statement yet you seem to regard it as having no bearing upon your own actions, despite the fact it is a clarion call to intervene
before the abusers become so strong that you, and the society you belong to, are destroyed.
You fixate upon the immediate harmful consequences of intervention without considering the long term consequences, possibly preferring to leave it up to any "higher authority" to prevent harm to yourself and others. You claim that you have been subject to ill effects from acting, well whoop-de-do.
Suffering unintended consequences is a part of living. Personally in relation to the subject at hand, I was beaten up by a gang of guys who were attempting to beat up another; I have been assaulted for trying to call the police to stop a man beating his girlfriend; I have stood on the steps of the South African Embassy in Trafalgar Square knowing that the British security services tracked such participants; I complained when a manager called a customer a "fuckin' Paki" behind his back and was nearly fired until the business owner realised the said customer was one of the company's best; I was screamed at when I intervened to stop a mother beating her child in public; I was banned from an atheist MB for complaining about the sexism of some of the commentators.
But the above are only particularly memorable because of those consequences, there have been countless times that I have intervened or spoken up suffering no ill effects. Nearly always there has been a positive result or response - mostly in cases where men were molesting or belittling women.
I am not asking that you act as directly as I tend to do, but I am asking for you and others to support and offer aid both to those do intervene and to those who are affected by prejudice, assault and rape. Additionally I am asking for those who do regard women as more than sex toys to speak up and make it clear such behaviour is not tolerated. Consider it a paradox, some intolerance is required to ensure wider tolerance.