Real Madrid sacked manager Julen Lopetegui on Monday night, with many in the British press speculating that 'Los Blancos' may make another move for Mauricio Pochettino.
And while former Chelsea boss Antonio Conte and interim boss Santiago Solari are favourites to replace Lopetegui, Neville believes the Spurs manager is capable of taking over at one of the biggest clubs in the world.
"What I would say is a bit of a worry are Mauricio Pochettino's comments," he told 'Sky Sports'.
"I said before the game it's the first bricks in the foundations of potentially wanting to leave and during the game tonight the Real Madrid coach has been sacked", Neville said.
Poch admitted that he is currently the unhappiest that he has been during his five-year tenure in North London with Tottenham.
"If you are Real Madrid and you are thinking 'Where can I get a manager, where's the next manager?' He speaks Spanish and has been a coach in La Liga. That is an appealing job for anybody in world football.
"He knew the ground rules before he came in. He knew he wasn't going to get £500-£600m when he went to Tottenham.
"He took the job knowing Daniel Levy, the board at the Tottenham, the restrictions, so my view is that now he probably wants that next level of job because he deserves it as he is a brilliant coach.
"That next level is available at arguably the greatest club in the world of football in Real Madrid.
"That's where the manager might be thinking 'is this now the time for me to step out?' I think Tottenham, if Real Madrid want him, will have a real problem in the next few days.
"I don't think they do, we are just putting two and two together, but we spoke about the timing of his comments, and is the timing because a job has become available?"
Tottenham Hotspur were defeated 0-1 at home by Manchester City on Monday night.