Kane Hemmings and Kristian Dennis scored their first goals for Notts County as the Magpies were beaten 3-2 on an exhausting night at Rodney Parade.
Hemmings had fired the Magpies into an early lead only for goals from Fraser Franks and Mark Harris to turn the game on its head before the break.
It looked like Notts had rescued a point late on when substitute Dennis turned the ball home from close range.
But, deep into five minutes of stoppage time, Jamille Matt scored from a yard out to win his side the game.
Kevin Nolan made one change to his team following the Friday-night defeat at home to Yeovil Town, Jon Stead replacing the bench-bound Dennis.
Notts started the game brightly and claimed the first shot of the game on two minutes, Nathan Thomas seeing his low effort from the edge of the area comfortably saved by home keeper Joe Day.
And the Magpies soon had the lead after Enzio Boldewijn burst away down the right and supplied the perfect low cross for Hemmings to turn home from close range.
Nolan's men could have had a second when Thomas saw a goalbound effort blocked after nice set-up play involving David Vaughan and Hemmings.
But the match was then turned on its head with two Newport goals in quick succession.
The first came from a corner, Franks nodding home Matty Dolan's delivery, before a misplaced Richard Duffy pass allowed Padraig Amond to supply Harris who kept his composure to stroke his side into the lead two minutes later.
Josh Sheehan's name was taken following a late challenge on Elliott Hewitt, who was OK to continue after receiving treatment, before another barnstorming Boldewijn run came to nothing when the Dutchman could only float his cross over everyone in the middle.
Shortly before the interval, Ross Fitzsimons made an easy save to deny Amond on the turn.
And 2-1 was how it stayed going into the break.
There were no Notts changes at the break but there could easily have been an alteration to the scoreline within two minutes of the restart when Amond glanced Dolan's free-kick towards goal. Fitzsimons somehow kept the ball out.
Thomas was the next Notts man to need treatment after a poor challenge from Bakinson, who was booked, before an edgy period of the match ensued during which both sides struggled to create clear-cut chances.
Nolan's first change came on 62 minutes when Vaughan made way for Tom Crawford, who slotted in alongside Noor Husin.
And Husin was booked shortly afterwards for hauling back ex-Magpie Scot Bennett after the pair had tussled for the ball near halfway.
Dennis was the next man into the fray, replacing the hobbling Stead with 18 minutes to play.
But it was the Newport strikeforce who threatened next, drawing another fine save from Fitzsimons when Amond met Harris' cross with a first-time effort six yards out.
Crawford saw a tame effort easily saved and Andy Kellett replaced Dan Jones as full-time neared.
The move saw Notts switch to three at the back and it bore fruit immediately as they hauled themselves level in the 87th minute.
Boldewijn was key once again, drifting a cross to the back post for Dennis to control before guiding the ball beyond Day in the Newport net.
Five minutes were added on at the end of the second half and the game was now anyone's.
Unfortunately, the home side were the ones to take advantage as substitute Matt turned the ball home on the goalline after a cross into the box had been nodded back across goal.
The referee's whistle blew seconds after the restart, sending the Magpies home empty-handed.