Its own innings lacked direction. Tammy Beaumont made a gritty 62 from 97 balls at the top of the order but England relied on wicketkeeper Amy Jones, who made 53, to spark the innings with her running between wickets.
England was never on top in the run chase because of Wolvaardt's innings.
When she was gone, Kapp's superb leadership kept South Africa on top. Kapp was out with 10 runs needed from 10 balls and Trisha Chetty and Shabnim Ismail were easily able to see South Africa to victory.
"We're obviously pretty gutted with the situation," England captain Heather Knight said. "It's obviously not through lack of effort. We're in the situation we are and obviously we've got another match fairly quickly so we're going to have to turn things around and believe there's a small chance to make it through."
Earlier, Bangladesh pulled off its first win at a Women's Cricket World Cup when Pakistan lost five wickets for five runs to lose by nine runs in a thrilling run chase.
Sidra Ameen was among those dismissed during the collapse when she was run out for 104 after posting the first century by a Pakistan batter at a World Cup.
Pakistan was 183-2 after 42 overs and seemingly well placed to surpass Bangladesh's total of 234-7 with Ameen in command. But leg-spinner Fahima Khatun sparked the Pakistan collapse with three wickets. Two run-outs added to the chaotic ending to the innings.
Khatun finished with figures of 3-38 from eight overs and was voted player of the match.
"I wasn't nervous because I'm always ready to play for my team," she said. "I just bowled in the right areas, according to my plan and knew success would come.
"It was a very sporty wicket and if bowlers bowled in the right area then they would be successful."
Pakistan captain Bismah Maroof made 31 from 48 balls to boost her team. But after her dismissal the Pakistan innings began to unravel because of poor shot selection against accurate spin bowling.
"It's a hard loss to digest," Maroof said. "Sidra (Ameen) will be disappointed. She was playing really well and will be disappointed she didn't finish the game.
"We were in the game, we should have chased it down but the poor shot selection cost us."
Bangladesh now has one win from three starts in New Zealand while Pakistan has no wins from four matches.
"I can't describe this in words because this is our first-ever win in a World Cup and this is history we've made today," Bangladesh captain Nigar Sultana said. "Now we're looking to carry this win through the tournament."