Andre Ayew scored the only goal of the game with eight minutes to go as he bundled the ball in from close range, ending Town’s two-game winning run at home.
The hosts had a number of chances throughout the 90 minutes but were ultimately left disappointed as they remain 21st in the Sky Bet Championship.
Graeme Jones made three changes to the side that lost 2-1 at Preston North End last week, with Simon Sluga, Andrew Shinnie and Harry Cornick all coming back into the starting line-up.
The Hatters controlled play early on and could have gone ahead within the first 11 minutes if it wasn’t for a last-ditch block in the penalty area.
Pelly Ruddock-Mpanzu done well on the left to get the ball into the box and his delivery found Berry, who flicked it towards Cornick, but as the forward was about to fire towards goal George Byers produced a timely interception, with referee David Webb waving away appeals for a penalty for handball.
Ayew went closest for the away side in the first half after darting in from the right, but he only found the side netting.
Town had a presentable opportunity to take the lead five minutes before half-time. Cornick played the ball across to Shinnie and his fiercely struck effort stung the gloves of Freddie Woodman from 25 yards out.
The resulting corner also found its way to the midfielder on the edge of the box, but he drove the ball back in and Mpanzu was inches away from diverting it towards goal.
On the stroke of half-time Borja Baston went close with an acrobatic effort and Swansea remained on the front foot at the start of the second half, as Baston had a clear sight of goal before Matty Pearson blocked well.
The Hatters quickly grew into the half as Luke Bolton’s deep cross went all the way through to Cornick at the far post, but his left-footed shot flew wide of the post.
Swansea then had three threatening corners in succession, the first of which Sluga did extremely well to save Ayew’s header.
But Town responded yet again and had two good chances to score. Substitute Ryan Tunnicliffe played a lovely ball over the top for Cornick to run onto, he rounded Woodman and laid the ball back to George Moncur, but the midfielder’s effort was cleared away.
Less than a minute later James Collins provided a brilliant cross for Shinnie at the far post, but his diving header went over the bar.
The end-to-end encounter went the way of Swansea as they found a way through in the 82nd minute courtesy of Ayew, as he chested in substitute Jay Fulton’s heavily deflected cross, which looped up off Tunnicliffe's boot, to give the away side the lead.
The Hatters continued right to the last whistle but couldn’t find an equaliser, even as Sonny Bradley went close and Moncur was denied from the edge of the box.
Town are back in action at Kenilworth Road on Boxing Day as they face Fulham, while Swansea - who moved up to sixth - face Brentford.
TOWN: Sluga, Bolton, Pearson, Bradley (C), Bree, Butterfield (Moncur 61), Mpanzu, Shinnie, Berry (Tunnicliffe 61), Cornick (McManaman 74), Collins. Subs not used: Stech, LuaLua, Jones, Sheehan.
Yellows: Collins
SWANSEA: Woodman, Roberts, Wilmot, Van der Hoorn, Bidwell, Grimes (C), Byers, Carroll (Fulton 75), Ayew, Celina (Naughton 87), Baston (Routledge 75) Subs not used: Nordfeldt, McKay, Kalulu, Cooper.
Goals: Ayew (82)
Yellows: Bidwell, Byers
Referee: David Webb
Attendance: 10,062 (1,032 away)
