Feminism, according to Webster, is “the belief that men and women should have equal rights and opportunities.” Based on this, one must simply perceive and treat men and women as equals to be kosher with feminism.

To some, however, feminism is all or nothing, and unless you’re a card-carrying member of the club, you’re part of the problem.

During an interview with The Daily Beast, Taylor Swift was asked if she considered herself a feminist, to which she answered, “I don’t really think about things as guys versus girls. I never have. I was raised by parents who brought me up to think if you work as hard as guys, you can go far in life.” While this non-answer failed to address any actual tenets of feminism, Swift didn’t denounce the notion of gender equality. On the contrary, she seemed to take it as a given.

However, some took her refusal to adopt the label as a slight on feminists and women as a whole. Dodai Stewart at Jezebel interpreted the statement, along with Taylor’s entire image, as “patriarchy-friendly,” while Andrea Kiliany Thatcher at The Fashion Spot attributes Swift’s denial to the singer’s lacking backbone. Rebecca Evans at NOW thinks Swift’s statement and lyrics are “insulting to women.”

And the disparagement of non-feminists didn’t end with Swift. In a bigthink.com video, attorney Gloria Allred said, “If you’re not a feminist, then you’re a bigot.” Hanna Rosin regarded Yahoo CEO Marissa Mayer’s assertion that she isn’t a feminist as “boneheaded.” Back in the Jezebel camp, Katy Perry’s admission that she’s not a feminist was deemed ignorant and ridiculous.

Focusing energy on attacking women who don’t, for whatever reason, choose to call themselves feminists is doing the movement—and women—more harm than good. One common feminist stereotype is that they’re control freaks forcing their ideas down others’ throats. Considering all the work feminists have done to further gender equality, it’s sad that infighting about who ought to call themselves what makes it easy for detractors to reference that image.

And that label can be problematic when dealing with people who don’t subscribe to Webster’s (or Allred’s) definition. Since mainstream feminism has historically appealed to government intervention to achieve its goals, many in the liberty movement are skeptical of, if not hostile towards it. Just as people construe varied meanings from the word “Libertarian,” the word “feminist,” justly or not, has fallen to the realm of subjective interpretation. A woman trying to communicate with someone with a different, perhaps negative, frame of reference for feminism might have better luck discussing individual issues rather than throwing out a label and hoping her audience shares and agrees with her definition of the term.

Many women working to break down gendered barriers, including some in the liberty movement, don’t identify as feminists. And many do. At the end of the day, it’s just a label, and adopting or rejecting it says very little about you as a person. We should worry less about labels and more about empowering women, regardless of what they call themselves.