Lorenzo Insigne rattled the crossbar in the first half for Napoli, while Red Star defender Milan Rodic made a vital goal-line clearance to deny Jose Callejon in the Group C encounter.
Home goalkeeper Milan Borjan also produced a sharp stop to keep out an effort from Poland international Arkadiusz Milik, as Red Star were restricted to a late attempt from Richmond Boakye.
Red Star last appeared in Europe's premier club competition in 1991-92, the season after they defeated Marseille in the final on penalties in Bari to lift the trophy.
Red Star players from that title-winning side performed a lap of honour before kick-off at a raucous Stadion Rajko Mitic, with Ancelotti - a three-time European champion as coach - in the opposing dugout.
The Italian left club-record goal-scorer Marek Hamsik and Dries Mertens on the bench, introducing them in the second half, although Napoli were unable to find a way past a Red Star backline that recorded an eighth clean sheet in 10 home matches this season.
Red Star go to Paris Saint-Germain, beaten by a stoppage-time goal in a 3-2 defeat at Liverpool on Tuesday, in their next game on October 3 while Napoli host Jurgen Klopp's side in Italy.