The Dutchman joined Chelsea from PSV as a teenager back in 2007, but despite being highly rated he quickly found himself become a victim of the Blues' infamous loan system.
A series of temporary moves to Coventry ( a team he had never heard of), Newcastle, Leicester, Wigan and Vitesse in his homeland followed, with mixed successs, before he was finally able to end his time at Stamford Bridge with a permanent move to Sunderland in the summer of 2014.
Now speaking to 'Sportsmail', the Holland international has revealed that he was left so down following a disastrous loan spell with the Latics that he was left considering quitting the game.
"I was really thinking about quitting football," he admitted. "I was that upset, I wanted to stop.
"I went on holiday with my missus at the time and said, 'What am I going to do now? No club is going to want me because I haven't played for a year'. I was very close.
"One of the worst loans of my life," he continued. "It was partly the manager, but a little bit me.
"Obviously when you go on loan from Chelsea you expect to play. At the time my mentality was wrong, I'm not going to lie. I thought, 'F*** it, I'm not going to train (properly)'. I didn't want to play in the reserves. It was the wrong attitude.
"I could have trained for myself, so when I got the chance I was ready. I didn't do that. I was just training and going home. That was just a waste of time. That's why Wigan is a closed book."
Eventually, a conversation with Jose Mourinho saw the decision made to put an end to his time at Stamford Bridge, something that left the Dutchman delighted.
"We had a nice conversation and I had a good feeling about that," he recalled. "But I think I made the right move to leave Chelsea behind. They kept sending me on loan to get more experience.
"How long does it take to become more experienced? In your head you're thinking, 'I need stability to prove myself in the Premier League'."
Two and a half years with Sunderland earned the flying full-back a move to Crystal Palace in January of 2016, with the move just feeling right from the very beginning.
"From the moment I stepped inside the dressing room, it felt like home — a family bond," he enthused.
From almost quitting the game when at Stamford Bridge, the short hop across London seems to have done the 28-year-old the world of good, with Van Aanholt now a regular international and a key part of the Eagles' side.