The Belgian winger netted with a thumping drive in the 27th minute and a 78th-minute glancing header at the Colchester Community Stadium, either side of a deflected long-range shot from Eric Dier.
Colchester, rock bottom of League One, replied in the 80th minute when Ben Davies put through his own goal after Gavin Massey had hit the post, but Chadli immediately teed up Tom Carroll to restore Spurs' three-goal cushion.
Later on Saturday, Arsenal's pursuit of a third successive FA Cup trophy -- a feat last achieved by Blackburn Rovers between 1884 and 1886 -- sees them entertain second-tier Burnley.
League Cup finalists Manchester City and Liverpool are both in action, City visiting the Premier League's bottom club, Aston Villa, and Liverpool hosting West Ham United, who have beaten them twice in the league this season.
Chelsea interim manager Guus Hiddink's quest to repeat his 2009 FA Cup triumph with the west London club continues on Sunday when his team visit second-tier Milton Keynes Dons.
Manchester United were the first team to book a place in the last 16 after winning 3-1 at Championship promotion-chasers Derby County on Friday, with Wayne Rooney, Daley Blind and Juan Mata scoring.