The shadow England team surged into a 17-point lead with tries from Henry Trinder, Ugo Monye and James Simpson-Daniel but the Barbarians staged a comeback either side of the interval.
Tim Visser, George Smith and Joe Van Niekerk added to the first-half tries from Benoit Baby and Frederic Michalak as the Barbarians took control.
England thought they had escaped with the win with a late try from the Exeter flanker Tom Johnson, but the Barbarians sealed the victory with a second effort from Visser.
Lancaster said: "I was proud of our performance in terms of the inexperience of our side in relation to theirs. We put ourselves in a position at 32-31 where we could have won it. I guess at the end their experience and quality told.
"As a lead into the Churchill Cup and for players to get the experience of playing at Twickenham and benchmark themselves against World Cup players, it is valuable."