Centre Harry Sloan, lock Elliott Stook and wing Henry Purdy all touched down for England while fly half Henry Slade added nine points with the boot.
South Africa started with a bang, scoring with just two minutes on the clock through wing Seabelo Senatla, who displayed his explosive pace to touch down in the corner. Fly half Robert Du Preez added the extras from the difficult angle.
Du Preez added three quick-fire penalties to take South Africa's lead to 16-0 with 16 on the clock while Slade missed his first chance at goal with a penalty from just inside the South African half.
Senatla was gifted his second try after 25 minutes, intercepting a loose pass to dive over but this time Du Preez couldn't convert to leave the score at 21-0.
There was some joy for England in the first period when Sloan went over before half time after an excellent team move and Slade knocked over the extras to make it 21-7 at the break.
England started the second half as South Africa had the first, scoring within the first two minutes. A strong break from captain Jack Clifford freed up space out wide and the imposing Stooke was on hand to dive over in the corner. Slade added the conversion to bring his side to within seven.
But South Africa hit straight back though through No. 7 Jacques Du Plessis, a defensive lapse letting the Baby Boks in and the flanker made no mistake.
It almost got better for the Boks when Senatla raced through but play was brought back for a high tackle and instead England's Slade kicked three points to make it 28-17.
A Du Preez penalty then made it 31-17 before England grabbed their third try through replacement Purdy, and Slade was again on hand to add the extras.
At 31-24 down England had the momentum and had chances to draw level but couldn't find the breakthrough.
Walshe said: "We're massively disappointed with the result, but the second half performance was pleasing. We gave ourselves a mountain to climb in the first half and at 21-0 down we knew it would be tough, but credit to the guys - they dug deep and came very close to pulling it back.
"We're not out of the competition, we will have to rely on a couple of results going our way but we all we can do is focus on playing well against USA on Thursday."
Points: Sloan 1T, Stooke 1T, Purdy 1T; Slade 3C, 1P
England Under 20 starting XV against South Africa
15 Anthony Watson (Bath Rugby)
14 Ben Howard (Worcester Warriors)
13 Harry Sloan (Harlequins)
12 Sam Hill (Exeter Chiefs)
11 Jack Nowell (Exeter Chiefs)
10 Henry Slade (Exeter Chiefs)
9 Alex Day (Northampton Saints)
1 Danny Hobbs-Awoyemi (Northampton Saints)
2 Luke Cowan Dickie (Exeter Chiefs)
3 Scott Wilson (Newcastle Falcons)
4 Elliott Stooke (Gloucester Rugby)
5 Dominic Barrow (Leeds Carnegie)
6 Ross Moriarty (Gloucester Rugby)
7 Matt Hankin (Saracens)
8 Jack Clifford (Harlequins) (c)
16 Scott Spurling (Saracens)
17 Alec Hepburn (London Wasps) on for Danny Hobbs-Awoyemi 58 mins
18 Tom Smallbone (London Irish)
19 Tom Price (Leicester Tigers) on for Dom Barrow 58 mins
20 Joel Conlon (Exeter Chiefs) on for Matt Hankin 65 mins
21 Callum Braley (Bristol Rugby) on for Alex Day 44 mins
22 Ollie Devoto (Bath Rugby)
23 Henry Purdy (Leicester Tigers) on for Anthony Watson 40 mins