I have a pretty high concept novel that is in an overly saturated market. I have had one agent request and reject it. In contests I have entered, the feedback I've gotten is that market is too saturated. I am thinking of hiring a professional editor to possibly smooth out the writing.
However, it's quite expensive, and while I would be willing to spend the money if it really helped me move up to the next level writing wise, I don't want to if it won't get published because of the overly saturated market. If that's the case, it may be better for me to move onto my next novel. I hope that makes sense. Would it be out of line to ask the rejecting agent if he thinks a professional editor would be worth it? I hate to bother agents that are already so busy, and I don't want him to give me an editorial letter. He's the only professional that has read the manuscript.
An "overly saturated market" doesn't have anything to do with the quality of your writing. If the shelf is full, no one is looking to take on more authors in that category.
Hiring an editor to improve your novel for a category that's already full is solving the wrong problem.
On the other hand, you're basing your decision on VERY limited feedback. One agent requesting and rejecting is hardly the basis for making any changes. And I'm not sure what contests you're entering, or who the judges are but unless they are agents who make the choices of what to take on, and what not to take on, I don't think that advice is worth much either.
You need to query much more widely before deciding anything.
And if you're writing in a category, you should be reading that category as well. Front list (ie published THIS year books)
How can you tell if a market is getting full? Look for debut writers in the category. If the category is mostly authors with three or more books, that means there are a lot of authors with on-going series. If there are more than four or five debut authors published THIS year, then there's probably a lot more opportunities.
This is one of the many reasons you need to read widely and deeply in your category. It's very easy to get discouraged by what someone tells you unless you've done you're own reading and know they're full of hogwash.