A passer-by saw the boy's activities around the device and contacted police, who called in a bomb squad to carry out a controlled explosion.
It is not known whether the unnamed boy assembled the device himself.
It was made from gunpowder in a jam jar, with nails and screws glued to the inside.
Police investigators said the explosive material used had been created out of the ingredients of fireworks and sparklers and was flammable but not explosive.
The child, who was born in the city in 2004, was arrested the same day and is in juvenile detention. However, because he is not yet 13, under German law he cannot face a criminal conviction.
Chancellor Angela Merkel's spokesman said "a report like this is frightening" but declined further comment, saying the matter was in the hands of prosecutors.
Stefan Biehl from the federal prosecution service said: "I can confirm that we have started an investigation based on the discovery of a nail bomb in Ludwigshafen."