Human beings are creatures of habit, especially when it comes to bedtime. We sleep on the same side of the bed, have the window open just so, the bedroom door ajar. Some friends swear by hot baths, herbal tea or a glass of wine. One acquaintance lets Melvyn Bragg's In Our Time lull him to sleep (it "never fails").
Mostly I read until my eyes won't stay open, which usually isn't very long. All of us are amateurs, however, compared with one light-sleeping couple I know who wear Tempur-Pedic eye masks and earplugs, and surround themselves with three white-noise machines and a fan. When they moved in together, they each kept their own beds and pushed them together, so neither need disturb the other when they roll over in the night.
The more you think about sleep, the odder it seems that it should be a problem at all, let alone be surrounded by so much superstition and anxiety. We need it like we need water, but it's not a resource we have to go out and find - or at least it shouldn't be. As a culture, though, we are obsessed with improving what scientists describe as our "sleep efficiency" - increasing the proportion of the time we spend in pursuit of sleep actually being asleep - so that our days can be more wakeful, more productive. There's an entire sleep industry offering technology and drugs, but to what extent are we just the victims of our own anxiety? Does our need fuel the industry, or is it the other way round?
Like many people, I take great care to follow the same routine on going to bed. But none of it ensures that I stay asleep. For the past three nights I have woken up repeatedly from the early hours, and felt more tired on waking than I did when I went to bed. There's no reason - no small children waking me up, no extra stress. It just happens, with predictable unpredictability, several nights a week. When I do sleep through I feel as though I could run a marathon, but those nights are in the minority.
I am far from unusual. According to a study carried out by researchers at Keele University in 2007, 37 per cent of us have one or more symptoms of insomnia on most nights, whether that means having trouble falling or staying asleep, waking frequently or waking up feeling worn out. Just over half wake several times some nights and a further 26 per cent most nights. Staying in bed longer doesn't make it better, as studies prove that, with sleep, quality bests quantity. That doesn't stop armies of us attempting to catch up at the weekend, bingeing and purging, never fixing the underlying problem.
According to some experts, sleepiness is the least of the insomniac's worries. During the night we cycle between different stages of sleep, including deep (or "delta") sleep towards the beginning of the night, and rapid eye movement (REM), when we dream, and it's the former type that is of most concern among doctors and researchers. In a recent article for The New York Times, the psychiatrist Vatsal G. Thakkar posited the theory that many cases of attention deficit hyperactivity disorder may in fact be misdiagnosed sleep disorders, and warned of more consequences of our ignorance. "Chronic delta sleep deficits in humans are implicated in many diseases, including depression, heart disease, hypertension, obesity, chronic pain, diabetes and cancer, not to mention thousands of fatigue-related car accidents each year," he wrote.
"Sleep disorders are so prevalent that every [doctor], paediatrician and psychiatrist should routinely screen for them. We need far more research. Every year billions of dollars are poured into researching cancer, depression and heart disease, but how much money goes into sleep?"
No wonder we're too anxious to sleep at night. We are constantly peppered with advice on how to tell whether we have a problem, and what to do about it. We're told that if we need an alarm clock to wake us up, that's a sign we're not sleeping enough, that the amount we sleep on holiday is the amount we naturally need. We're told that we're sleeping less than ever, that all the blue light from iPads and television screens is disrupting the natural release of melatonin, the hormone that tells our brains it's night-time. But all of this is contentious - as is the very concept that there is an optimal amount of sleep.
"The idea that adults need eight hours of sleep is nonsense - it's like saying all women should fit into a size-eight dress," says Prof Jim Horne, the founder of the Sleep Research Centre at Loughborough University. "Our research shows that, on average, actual sleep for the adult population [in Britain] is only about seven hours. And there's very little evidence for the idea that previous generations slept longer, that we're chronically sleep-deprived. We've never had it so good: the average worker 100 years ago shared his bed not only with his partner, but children, fleas, bedbugs, dampness and noise. Our bedrooms today are warm, we have duvets and spring mattresses. Lamentations about poor sleep are nothing new - the Romans complained about it. And just because we sleep more at weekends doesn't necessarily mean we need it."
As for the idea that blue light is disruptive, "it might be true, but we must remember that part of sleep is settling down to relax, and if you're looking at iPads and playing computer games, it gets you excited - I think both interfere with sleep".
Horne also questions the quality of many of the studies that purport to prove connections between health problems and a lack of sleep. With obesity, for instance, there is only a true link for people who sleep less than five hours a night, "which is really abnormal. There's no evidence that people who sleep six to seven hours are any fatter than those who sleep seven to eight," he says. "And if you really are overweight and a 'short sleeper', it might be better to do 15 minutes of brisk walking a day than spend an extra hour sleeping."
Still, the fact remains that many of us don't feel rested when we wake up, so it's no wonder we'll do and spend anything to fix the problem. For $205, Amazon will sell you "earthing sheets" that claim to connect your skin to "the rhythms stored in the earth [that] are able to reset the wake-sleep clock". They're a bargain compared with what a duvet can cost: a Siberian goose-down duvet at ABC Carpet & Home in New York is on sale for US$2500 [$2915]. If I wanted to be wired up to a polysomnogram, which records brain-waves and other vital signs to find out how well you sleep, I could check into one of the many sleep clinics within a short drive of my home, or I can buy a DIY device such as the Jawbone UP24 bracelet, which tells how long and deeply we sleep. Later this year you'll be able to buy an alarm clock that consists of a large pad that measures sleep cycles and a light that wakes you at just the right moment.
Increasingly, we're also turning to chemical interventions: in Britain one in 10 take sleeping pills, compared with one in four Americans. (Once, when I was due to catch a plane home to London, an entire dinner table of New Yorkers offered me Xanax and Valium from their personal stashes.) There are also new medications such as Modafinil, a stimulant intended for sufferers of narcolepsy that health officials warn "is not a replacement for getting enough sleep", although that doesn't seem to be deterring its recreational fans. For those who prefer natural remedies, there are myriad options at health stores, from the valerian tea used by those sleepless Romans, to CherryActive, a product made with Montmorency cherries - which, according to the manufacturers, "are high in naturally occurring melatonin". And then, of course, there is what we use to counter the effects of poor sleep: one report last year showed the British coffee-shop market was growing seven to eight times faster than the economy.
But while we focus on the effects of not getting the kind of sleep we need, what we tend to overlook is sleep itself. For most of us, says Prof Horne, "sleep is to stop you being sleepy - in the same way that we eat to stop us being hungry. But isn't there a bit more to eating than that?"
In the new book The Secret Life Of Sleep, an overview both of scientific discoveries about and cultural attitudes to sleep, the American author and mental-health counsellor Kat Duff argues that since as far back as Aristotle, Western cultures have thought of sleep "as the absence, or debasement, of consciousness ... The demands and attractions of our 24/7 global economy are squeezing the hours out of our nights. We are losing the knack - and taste - for rest." At an extreme is the recent trend for "sleep hacking", whose proponents aim to push their waking hours to the limit, and swap tips such as taking ice baths and removing blinds so you're woken at dawn.
While sleep medicine is a relatively new specialism in the West, Duff tells me that she was surprised to learn that in the East "the Hindus and Buddhists had been studying sleep for millennia, and were able to define the stages of sleep that were confirmed by neuroscientists in the last 20 years". That made her realise that our culture "demeans" sleep. She likes the Taoist approach: "going consciously into sleep rather than letting it be a collapse from wakefulness. It's the notion of respecting it as a valuable part of our life experience, not just for how it can improve our productivity."
For all the melatonin supplements and sleeping pills, we still don't fully understand how sleep works.
"There must be about 100 mechanisms in the brain which regulate and affect our sleep," says Horne. "We can't get to the level of the cell to decide what's going on in sleep to make us feel better. We don't know the answer."
As for its purpose, while we build bone and muscles and strengthen our immunity during deep sleep, the purpose of REM sleep is more contentious. In her 2010 book The Twenty-Four Hour Mind, American psychologist Rosalind Cartwright sets out a case that dreams are a way of "regulating emotion", and help us "update" our sense of identity each night. Horne, meanwhile, argues that REM sleep is "dispensable": certain medicines such as antidepressants suppress it and you can survive, even thrive, without it.
And even the idea that we should naturally sleep through the night, as I so often don't, is in dispute. According to the historian A. Roger Ekirch, in pre-industrial times it was common to have a "first" or "deep sleep" and a "second" or "morning sleep", with an hour or more of wakefulness between, sometimes referred to as "the watch". Duff cites a 1992 study in Maryland that simulated the 14-hour dark nights of mid-winter. The participants settled into a rhythm of two sleeps, with a few hours between.
But there are other reasons why we wake up. "Alcohol tends to help people get to sleep but commonly disrupts sleep in the second half of the night, causing people to feel quite shabby on waking," says Dr John Briffa, the author of A Great Day At The Office, a guide to managing energy levels. He also advises avoiding starchy or sugary foods at dinner-time to stave off blood-sugar falls in the night, which "the body will seek to correct by [activating] the stress response. This can cause individuals to wake in the middle of the night in an alert state that can make it difficult to drop off again."
For me, the trick that most often helps me drop off is to try to keep my eyes open. In other words, trying not to sleep. If I wasn't too tired, I might be able to see the irony.