Yes, but not only that. See below ...
>;Should I just stop caring as much as be me where I have to look away at times?
You have this choice. But you'd be forfeiting a "better" version of yourself (improved skills) that you could be ...
> I’m a good active listener, so that’s not a problem.
This is (I humbly think), the key here: Think of eye contact as a way (or, another way, or a complementary way) of *listening with your eyes*, for that is what it (mostly) is: A signifier of attention. You already have in you if you listen well. This is just an extension of that.-
Try to make oneself very rich is a sure way to maintain self confidence. There are other ways but those are mostly born.
Point is, consider if this looking people in the eyes is something really worth investing on: maybe not looking at people is your thing and you're fine that way too? Why you have to be wrong? Maybe it's the others that are doing the uncomfortable thing for others and are wrong.
Eye contact is not an end in itself. If you want an end in itself, be a good hang. Good luck.
Please, do society and yourself a favor and don't act like you perceive VCs to other people, especially people working to serve you as a JOB.
These people you are trying to mimick grew up interacting with people like you trying to get on their good side. That is what give them confidence. Put them inside a football pitch against good players and you will see a bunch of anxious amateurs that will not hold eye contact after being slide-tackled once.
The more you talk to people from different walks of life, the more kinds of interactions you learn to have (and hopefully become comfortable in). You also notice that there's a pretty wide range of "normal" for everything from eye contact to gestures and other body language. You don't have to hold a dead stare for multiple minutes. Doing so can make a lot of people feel uncomfortable, in fact, because like you said it can feel like a "power move" even if they didn't intend for that. Just a few seconds at a time, accompanied by the occasional nod or smile or soft "uh-huh", can be reassuring without being overly intense.
But even then, there will always be some people who don't like receiving (or giving) direct eye contact... it depends on culture, race, in-group vs out-groups, etc. too. Part of maintaining a "safe" amount of eye contact is learning to read the other person's cues and sensing whether it's adding to the interaction or just making them uncomfortable.
Eye contact is helpful when you're trying to show someone you're paying attention and listening to them (or maybe trying to woo/intimidate/dominate them, which hopefully you aren't). But it's best practiced as part of normal everyday conversations (across different topics and kinds of people), and really, it helps you convey genuine emotional reactions to things (like excitement or confusion or boredom).
You don't need to practice "how to stare someone down". It's not supposed to be a power move unless you're in prison or maybe a prison-like corporate boardroom, lol. It's just a subtle "yeah, I'm still here and still listening".
It might help also to analyze your own thoughts and feelings during (or immediately after) you have an interaction with someone where eye contact felt a certain way to you. Aside from the power move VCs, do you ever have interactions where someone else's eye contact just made you feel welcome? Loved? Shunned or judged? Flirted with? Appreciated as an equal? What else are they displaying at the same time in terms of tone of voice, facial expressions, body gestures, physical distance, etc.? It's usually not just a cold-eye serial killer stare, but a combination of things.
What about you yourself, when you give others eye contact... are you just listening? Are you trying to show rapt attention? Submission? Friendliness? Openness? It might help to put it all in context.
IMO what you're trying to communicate is a complete message, like "I'm interested in what you're saying and curious enough to think about it as you're talking", not "I am able to force my eyes open and maintain a gaze at you because I'm more powerful". That's just a dick move, so don't try to do (or practice) that, lol.
And think about this a bit - if you feel defeated after talking to people who stare into your eyes... is that how you want people to feel after talking to you? Maybe it is just as well not to be one of the people with piercing eye contact.