I've written a whole series of novels (I'm on the seventh and final one). I know you're not supposed to complete a series unless you can sell the first one, but I quite literally couldn't help myself. The world drew me in.
I just started querying agents on Book One, after going through it with critique groups and beta readers. It's not a shelf novel...it's far from perfect, but I think it's good, and I don't think I'll change my mind about that with time. My problem, besides my inexperience querying agents, is that it's not a broad-audience series. They're decidedly weird books, hard to pigeonhole into an exact genre. I mean, imagine if Hunter S. Thompson came back to life, huffed a bunch of ether and said, "I'm going to write a young adult fantasy novel!" (Except maybe the ether impaired his glorious writing abilities a little. And to be clear here, I don't huff ether).
If I continue having no luck with agents (only queried about 15 so far), would it hurt me so badly to self-publish this series and try to market to my narrow audience? My future books might be more marketable: I'm having very good luck getting my little narrative nonfiction anecdotes published, and wanted to compile them at some point into a memoir. My idea for my next fiction novel seems more mainstream, too.
You think there aren't small publishers out there who appeal to very niche audiences? You haven't done enough research my friend.
Of course, some small publishers are complete and utter disasters, whereas others are chugging along producing beautiful books. Your job is to find out who they are and which side of the line they live on. You can do this instead of writing and it still counts as work! Wheee!
A small publisher can be a whole lot better than going it alone. For starters, you have comrades in arms (other writers on the publisher's list) and that will help you when you need to learn how this
publishing game works. And you have people to ask if it's just you or if everyone's royalty statement is late. Or a bunch of people to contribute toward the PW ad that features all your titles.
Small presses are a great place to find the niche audience you think will be yours.
And having niche work is no barrier when you have wider appeal books in the future. Everyone in publishing understands that there's a limited audience for dinosaur erotica....or was.