Brandon Carney struck a 90th minute winner - his second of the game - to secure Boro's place in the next round after Needham had fought from 3-1 down to draw 3-3 in the second half.
The game got off to the perfect start for Stevenage when Jack Storer buried a header from Carney's left-wing cross within a minute.
Robbie Sweeney fired over from close range before the visitors fought back with an equaliser from the spot on 17 minutes through Adam Mills. The scores were level for barely two minutes though as Boro retook the lead when Jamie Silvestre superbly swept home Ross Millard's cross from the left.
Defender Jack Jones saw a low effort parried by visiting 'keeper James Bradbrook as Stevenage kept up the pressure, playing some nice attractive football along the way. Needham, however, were staying in the game and improved as the tie went on, playing some neat passages of play of their own and showing the kind of form that got them through three qualifying rounds to get to this stage.
Stevenage took the goal advantage into the break but within minutes of the restart it was 3-1 when Bradbrook's clearance hit Carney and flew into the back of the net.
With Robbie Sweeney looking lively up top and Ben Kennedy marshalling the midfield Boro looked comfortable, but back came Needham with two goals in quick succession. Firstly Mills scored his second of the game from close range after Boro failed to clear a 64th minute corner, before a curling effort from Taylor Bannister drew the scores level and sent the visiting supporters wild.
Both sides pushed forward in the closing stages but with extra time looming there was to be one final goal in injury time. Much of the credit goes to sub Louis Yamfam for good work down the left. He beat his man and whipped in a beautiful cross which Carney nodded down and into the back of the net to spark scenes of celebration on the home bench.
Needham Market can leave tonight with their heads held high after a superb, battling performance, but it's Boro who advance to a second round home tie against Wealdstone.
Stevenage: Reading, Casey, Butaka (Jackson 85), Jones, Millard, Kerr, Kennedy, Sweeney, Storer, Carney, Sylvestre (Yamfam 78)
Subs not used: Guerfi, Patel, Bruce-de-Rouche