Milk has long been recommended by doctors and nutritionists for boosting calcium intake and helping to keep bones strong.
But new research suggests that it does little to strengthen bones and can double the risk of an early death.
A study which tracked 61,000 women and 45,000 men for 20 years found there was no reduction in broken bones for those who consumed the most milk.
For women it was associated with an increased chance of suffering a fracture. Those who drank three glasses or more a day (680ml) were twice as likely to die early than those who consumed less than one.
The NHS recommends milk to help with osteoporosis and says that a pint (about 550ml) provides a healthy amount of calcium for the day.
Until 1971 all children under seven received one third of a pint of milk each school day, However, Margaret Thatcher scrapped the allowance to save money when she was education secretary under Sir Edward Heath. It earned her the nickname "Milk Snatcher".
The study's lead author Professor Karl Michaelsson, of Uppsala University in Sweden, said: "Our results may question the validity of recommendations to consume high amounts of milk to prevent fragility fractures. A higher consumption of milk in women and men is not accompanied by a lower risk of fracture and instead may be associated with a higher rate of death."
Almost three million people in Britain are thought to suffer from osteoporosis. Half of women will suffer a fracture after the age of 50, and one in five men. Bone is living tissue that is constantly broken down and built up. In healthy individuals, bone production exceeds bone destruction up to about the age of 30 when the skeleton gradually starts to deteriorate.
Calcium is needed for bone building, which would suggest milk should be beneficial. But researchers believe the fat in milk cancels out the positive effects of calcium, triggering inflammation and increasing the risk of heart attacks.
However, low fat dairy products, such as cheese and yoghurt, had a beneficial effect, reducing early death and promoting bone health. British experts said the research should be treated with caution because the milk in Sweden is fortified with vitamin A which could have an impact on the findings.
Prof Sue Lanham-New, head of the Department of Nutritional Sciences at the University of Surrey, said: "We know that low calcium intakes (less than 400mg per day) is a risk factor for osteoporosis. Individuals should still be encouraged to consume a balanced diet from the five key food groups of which milk and dairy are key."
Gaynor Bussell, a public health nutritionist, said: "There may be another factor causing the increased mortality and fracture rate in women. Milk is a convenient source of calcium as well as many other vitamins and minerals. One such study is insufficient to base public health decisions on."
Public Health England said other studies showed milk protected against heart disease and stroke and may help prevent diabetes. The health body said it would not be changing guidelines.
Dr Louis Levy, of Public Health England, said: "The authors advise caution in interpreting the results and are not recommending that anyone stops drinking milk or eating dairy products." The research was published in the British Medical Journal.