The Norwegian boss is under intense pressure after a 5-0 home thumping by Liverpool and a one-sided 2-0 defeat at the hands of Manchester City on Saturday.
Ferdinand has been one of many ex-United players to voice support for his former team-mate, who led the club to a second-place finish in the league last season.
The ex-England defender said fans started the season full of hope after the high-profile signings of Cristiano Ronaldo, Jadon Sancho and Raphael Varane before results nosedived.
"I sit here now and I don't think we can challenge to win this league," Ferdinand said on his Vibe with Five podcast.
"I look at our team every week and I'm wondering what are we going to do tactically. I don't see any philosophy or an identity in the Man United way of playing, whatever that should be from the management. I sit here confused, looking at the team."
Ferdinand said United had made steady progress during Solskjaer's nearly three years at the helm but now he felt there is a need for a change.
"I was always deep down a bit sceptical," said the 43-year-old. "Could he take us on to be champions? I wasn't sure, I wasn't fully convinced -- I hoped he would be able to do that.
"But the showing with the squad that he accumulated, to the beginning of this season - and what I've seen this season -- I just feel it might be about the time now for the baton to be handed over to somebody else who can take us on now.
"And I think Ole would leave now with his head held high because from when he came in to where he got us to the beginning of the season I think it's positive."
Standout matches today