Speaking in ITV’s new documentary Out Of Their Skin, Cole also revealed how the booing he received during an England game in 2008 made him want to quit internationals because he felt his country 'didn't care' about him.
But he suggested there is a "different agenda" with Sterling, saying the only reason he sees Sterling might be targeted is his skin colour.
Recalling a World Cup qualifier against Kazakhstan in 2008, during which he was repeatedly booed despite England winning 5-1, Cole revealed that the abuse from fans made him want to quit the national team.
"I was like, "Woah, is that really for me? For giving the ball away?",' he said. 'At the time I thought, "I'm done".
"Why am I gonna try to give my all to help a country that didn't care? For me, it wasn't football. That's why I wanted to stop," he said.
"When I left Arsenal, that whole situation, I got the name of "Cashley". And of course with the stuff off the field, in my private life. I was their guy at the time.
But Cole says the booing Sterling receives is due to a different reason.
"I don't see it going on with anyone else apart [from] with Raheem. There's a different agenda to that I feel.
"There's nothing else left apart from his colour. His skin colour."