But having forced the Canaries to extra-time and penalties, the Lancashire club suffered heartache at the final hurdle as they bowed out within touching distance of a place in the third round.
Glenn Murray gave the home side a deserved ninth minute lead, but golden oldie Dion Dublin levelled for City early in the second half.
Both sides had chances to kill off the tie, especially minnows Rochdale as they looked to cause another Carling Cup upset having dumped out Stoke City in this year's first round.
But neither side could take advantage in either period of extra-time and when penalties came along, it was the visitors who made their experience count to make sure of place in Thursday's third round draw.
Tom Kennedy, substitute striker Adam Le Fondre and Chris Dagnall all scored their spot-kicks, but sub Rory Prendergast and Gary Jones saw their penalties saved by keeper David Marshall as Norwich won with a penalty spare.
Sub Chris Brown, Simon Lappin, Chris Martin and Darren Huckerby all scored as Norwich kept cool to keep a 100 per cent run in the shootout.
Earlier, Norwich nearly got on the scoresheet in the first minute, but Lappin's strike was parried by former Stockport County shot-stopper James Spencer.
And although the ball rebounded straight back to the former St Mirren midfielder, Spencer denied him for a second time.
The visitors' early chance then sparked Dale into action. John Doolan went agonisingly close with a right-foot rasper which flew inches wide, before the impressive Dagnall fired over the bar from close range.
Goalscorer Murray then glanced a header wide for the home side the Canaries found themselves on the back foot.
The pressure told a couple of minutes later as Murray took advantage of slip by his namesake Ian, bursting through City's defence to fire past Marshall and into the bottom left-hander corner.
And Rochdale nearly doubled their lead a minute later through Dagnall, but Marshall spared his side's blushes with an excellent save.
Huckerby, playing only his second game after recovering from injury, went close with a powerful shot, but Spencer saved well as Norwich battled to get back in the game.
Spencer made two excellent saves midway through the half, first blocking Gary Doherty's fine header, before fending away a header from Darel Russell.
Adam Rundle had a shot saved for Rochdale and Murray went close to netting his second, while Dagnall and John Doolan also had good first-half chances for the home team.
But having weathered the storm, Norwich responded in perfect fashion in the second half, veteran striker Dublin finding the target to level matters.
Dagnall had a shot blocked at the other end and Guy Branston went close with a header, before Norwich started to dominate.
Huckerby unleashed a powerful right-foot shot in 56th minute, but it flew wide of the target, while Frenchman Julien Brellier's right-foot shot from 35 yards out forced Spencer to make an excellent save.
Martin had a shot saved and Huckerby had an effort charged down, before Rory McArdle header over for the home side.
Dublin had a volley blocked and Martin had two good chances, with McArdle and Kennedy going close.
But the match went into extra time and, although it was Rochdale who had the better chances, it was Norwich who held on for a nail-biting win.