Joe Jacobson got the opening goal as he fired in from the penalty spot after Matt Bloomfield was tripped by Sonny Bradley 15 minutes in.
The hosts had the opportunity to extend their lead just after the hour mark when they got a second spot-kick following a handball from James Collins, but James Shea guessed correctly to keep out Jacobson's second attempt from 12-yards.
That sparked the Hatters into life and they got the equaliser through a Matty Pearson header.
Pelly-Ruddock Mpanzu, Elliot Lee and Eunan O'Kane all went close late on, but Allsop and the post denied the Hatters a stoppage time winner.
Hart said: "Lots happened, it was very eventful. But I think we certainly deserved the draw, if not more. It was difficult for us to see the first penalty, Sonny said it was nothing. But it was given. This is where we're suffering at the moment. If we took all the first halves out of it, we'd probably be in the top four!
"We're suffering for our little mistakes at the minute and it's tough for us, but I don't think there's any doubt in the quality of our play we try and produce and the courage that they show. To come back as many times as we do, it's phenomenal and we have just got to try and minimise these comebacks and start taking the lead.
"It was a great counter late on, unfortunate there, we hit the post a couple of times. We created chances, we've just got to make sure we're a bit tighter in the first half. We're playing up a division and we've got to learn, but we've got to learn quickly."