For the study, the team looked at two large groups of people in the US: more than 340,600 people in the National Health Interview Survey and more than 93,600 treated as outpatients at Veterans Administration clinics.
They evaluated heart disease and cancer risk among the individuals, whose ages ranged between 18 and 85 and were tracked from 1997 to 2009.
Researchers found those who had one or two drinks four or more times per week were 20 percent more likely to die prematurely.
This was compared to those who consumed drinks three or fewer times per week - and was consistent across all age groups.
Findings also showed that drinking did protect against heart disease, but not if you had a daily glass.
"Consuming one or two drinks about four days per week seemed to protect against cardiovascular disease - but drinking every day eliminated those benefits," said first author Dr Sarah Hartz, an assistant professor of psychiatry at Washington University School of Medicine.
"With regard to cancer risk, any drinking at all was detrimental."
Those who live in one of the world's five "Blue Zones" - geographic areas with low rates of chronic disease and home to some of the world's oldest people - may disagree.
Residents of Icaria, Greece; Sardinia, Italy; and the Nicoya Peninsula, Costa Rica all follow a Mediterranean diet with a glass or two of red wine a day.
People who live in Okinawa, Japan, drink a wine made from rice. The Seventh-Day Adventists in Loma Linda, California, are the only Blue Zone to not consume alcohol.
The National Institutes of Health recommends women have no more than seven drinks per week and men have no more than 14.
One drink is equivalent to 12 ounces of beer, five ounces of wine, eight ounces of malt liquor and 1.5 ounces of 80-proof distilled spirits or liquor.
But Dr Hartz believes the current recommended limit is too much.
"A 20 percent increase in risk of death is a much bigger deal in older people who already are at a higher risk," she said.
"Relatively few people die in their 20s, so a 20-percent increase in mortality is small but still significant.
"As people age, their risk of death from any cause also increases, so a 20 percent risk increase at age 75 translates into many more deaths than it does at age 25."
This new study comes on the heels of research from Washington University in Seattle, which looked at all types of drinking from light to binging.
It found one drink per day increases the risk of developing one of 23 alcohol-related health problems by 0.5 percent, compared with not drinking at all.
This rose to seven percent for people who consume two drinks per day and spiked to 37 percent for those who down five drinks.
Any protection against heart disease, stroke and diabetes turned out to be not "statistically significant".
Dr Hartz predicts that as medicine becomes more personalised, doctors may suggest people with a family history of heart problems have the occasional drink.
But in people with a family history of cancer, physicians may recommend abstinence.
"If you tailor medical recommendations to an individual person, there may be situations under which you would think that occasional drinking potentially could be helpful," said Dr Hartz.
"But overall, I do think people should no longer consider a glass of wine a day to somehow be healthy."
