Lead author Dr Linda Chang said: "The APOEe4 carriers show the strongest effects, with negative influences on brain structures and cognition mirroring those in elderly participants and patients with Alzheimer's disease."
There are currently 850,000 people living with dementia in Britain, which is expected to rise to one million by 2025 and two million by 2050.
The most common kind of dementia is Alzheimer's disease. But scientists are still struggling to determine what causes the condition, and despite 412 drug trials, nothing has been shown to combat the disease so far. The researchers think the APOEe4 gene slows brain development and could be a target for future medication. APOEe4 is also known to make people susceptible to disease so some scientists believe Alzheimer's could be triggered by an infection and may be fought with antibiotics.
Ian Le Guillou, research officer at Alzheimer's Society, said: "These interesting findings suggest that people with the APOEe4 gene - which increases the risk of developing Alzheimer's disease - have differences in their brains from childhood.
"However, we need to be cautious in interpreting these results as, although this study involved over 1,000 children, there were less than 30 in the highest-risk group. We would need to see these results replicated in a larger group, as well as having longer term follow-ups to better understand how the changes in the brain progress with age.
He added: "Although people with the gene are at an increased risk of dementia, there are still things they can do to lower their chances of developing the condition. This includes taking regular exercise, not smoking and keeping their blood pressure in check."
The research was published in the journal Neurology.