I'm afraid you might be trying to make a niche where there is not much of a niche to fill right now.
5.11 has almost all the bases covered at the moment.
I bought nearly a lifetime supply when Cabela's had their Safari line around 2007.
View attachment 520100
In the 80's, I got several 1st revival Abercrombie & Fitch shirts (which I have now outgrown and gave to my son).
View attachment 520101
When I find sometime I like, I usually go deep because experience has taught me that I will probably never see them again.
Right now, (without shopping around), I would probably go with Columbia Bonehead shirts.
I live in a part of the US that used to be known for making clothing and had several family members and friends involved in the industry (one still is). Paying >$100 for a shirt seems ridicules to people who know that they cost $7.00 to make.
My advice would be to come up with a well-researched and field-tested design, find a superior 100% cotton fabric, and an U.S. sewing company. (PM me - they still exhist).
Customers don't mind paying a more, if they know that Americans made it. If it's made in China with slave-labor, and your trying for 1000% markup at wholesale, you already have way too much competition.
There are a few different approaches to bring your idea to market. I don't mind giving a little advice.
(I've been selling outdoor clothing for nearly 30 years).