If an agent read a manuscript that they really liked and were considering taking it on, would they be put off if they found out it took the author 5+ years to write?
If this were Jeopardy and the answer is "The number of years it took to write your novel" the question would be "What is something you don't really want to know?"
That said, I think what you're worried about is the next book. If Book #1 took 5+ years to write, and the publisher has a deadline of a year from now for Book #2, well YIKES.
This is info you discuss with your agent when the time comes. You don't have to disclose it up front.
I assume that my clients spend more time on what will be their debut novel than they will on subsequent novels. For starters, they've done it once, doing it again isn't sailing off to the undiscovered country.
But if it takes you longer, then it takes you longer. Make it worth the wait, and you'll be fine.
And yes, this also falls into the category of things you should not be thinking about now. We're going to need a whole new category: Things You Fret About that Keep You From Writing.