"It was a really bad day from us, we woke up too late. St Pauli did well, but we're out," Dortmund captain Marco Reus said after the match.
Star striker Erling Haaland gave Dortmund hope by converting a second-half penalty, but the visitors failed to force an equaliser.
The result means neither Dortmund nor Bayern Munich, who suffered a humiliating 5-0 defeat at Borussia Moenchengladbach in the previous round in October, will be in the quarter-final draw for the first time since 2006/07.
St Pauli, currently top of the second Bundesliga table, were ahead after four minutes when Amenyido got in between Dortmund captain Marco Reus and Manuel Akanji to convert a cross.
Amenyido's shocked expression during his celebrations showed he could barely believe his luck while there were ashen faces in the Dortmund defence.
The away side's problems were compounded when they were 2-0 down at half-time after Witsel turned the ball into his own net just before the break while trying to block a cross to Amenyido.
Dortmund grabbed a lifeline when Mats Hummels' pass clipped the outstretched arm of St Pauli defender Jakov Medic in the box and Haaland converted the resulting penalty just before the hour mark.
Dortmund pushed for an equaliser, but Reus and Donyell Malen both squandering chances.
Earlier, Bundesliga club Cologne also crashed out against second-tier opposition after losing a penalty shoot-out 4-3 at home to Hamburg.
In curious scenes, Cologne midfielder Florian Kainz hit the net with his crucial final spot-kick but it was ruled out because he touched the ball twice, sending Hamburg through.
There had already been dramatic scenes in extra time when Cologne striker Anthony Modeste converted a last-gasp penalty to equalise in the 122nd minute and cancel out a Robert Glatzel goal.
Hamburg will be joined in the last eight by fellow second-tier club Karlsruhe after their 1-0 win at third division 1860 Munich.
Bochum won their all-Bundesliga tie at home to Mainz when forward Milos Pantovic netted twice in a 3-1 win.