Unai Emery's side demolished Lyon 4-1 last weekend in the Champions Trophy, but PSG were forced to wait until the 73rd minute at the Stade Armand Cesari before Kurzawa volleyed home the decisive goal.
"It wasn't the match I'd hoped for, but the players had warned me of what it would be like to face Bastia," said Emery.
"We didn't play like we did against Lyon but it was a different game. It was a bit like the friendly against West Brom, a very defensive team who play on the counter-attack."
He added: "We were a bit more aggressive (in the second half). We had four, five, six chances...but you have to score, and it's always difficult getting the first goal."
After finishing an incredible 31 points clear at the top last season, Paris are the early league leaders in France as Monaco rallied from two goals down to draw 2-2 at home to Guingamp in their opening game of the campaign.
Hatem Ben Arfa was the only one of five close-season signings named in PSG's starting line-up in Corsica, with Jese Rodriguez and Thomas Meunier both named on the bench and Grzegorz Krychowiak left out altogether after playing for Poland at Euro 2016.
PSG were also again without captain Thiago Silva and Edinson Cavani through injury, while Adrien Rabiot replaced Benjamin Stambouli in midfield with the latter reportedly on his way out of the club.
Neither side produced a chance of note in a gritty opening 45 minutes, while Brazilian winger Lucas crumpled to the ground midway through the second half after he appeared to be struck by an object as he lined up to take a corner.
The referee briefly halted play as stewards restored calm in the stands, and just a few minutes later Kurzawa delivered the vital goal after Jese, put through by Angel Di Maria, had seen his effort blocked by Bastia goalkeeper Jean-Louis Leca.
Marco Verratti and Blaise Matuidi made late appearances off the bench for PSG, with a robust challenge from the latter sparking a fractious finish to the match.
Monaco, who travel to Villarreal on Wednesday for first leg of their Champions League play-off, recovered from 2-0 down against Guingamp to salvage a point at the Stade Louis II.
First-half goals from Moustapha Diallo and Sloan Privat had put Antoine Kombouare's men seemingly in control, but Monaco clawed their way back through a 71st-minute penalty from Fabinho and Bernardo Silva's equaliser six minutes from time.
On Saturday, Saint-Etienne visit Bordeaux while Lyon go to promoted Nancy on Sunday and Marseille host Toulouse in the final game of the weekend.