"For many people their work lives are so busy with very long hours that they just go back with a sense that 'this is the way life is and we just have to get on with it'."
Most people who struggled returning to work were probably already dealing with additional "life problems", Mr Skinner said.
Gloomy days in the middle of winter seemed more of a problem for people feeling down than post-summer holidays, he added.
Mental Health Foundation chief executive Judi Clements said returning to work could have a profound effect on people, particularly those who had a busy and stressful break.
She recommended easing back into work.
"Try thinking about when your next break is so you're not contemplating an endless work period without any break.
"For some it might be a good time to think, 'Well, if coming back to work feels so gloomy and awful, is this really the right place for me?"'
It is important that people look after themselves by getting enough sleep, eating well and exercising.
The blues was a fleeting feeling for most, but those who were still anxious or down after a few weeks should consider talking to their GP.
"It might not just be back-to-work blues - it might be depression or an anxiety state that needs some help."
Auckland University workplace psychology expert Helena Cooper Thomas said holidays were good for recovering but the effects were "relatively short-lived".
People returning to work should continue to make the most of the good weather and make time for activities they enjoyed, such as exercise, hobbies, gardening or music.
In order to feel engaged at work, people needed to be mentally and physically available, feel safe in the work environment and consider their work meaningful, Dr Cooper-Thomas said.
Some people might not want to return to work because of anxieties about childcare, or safety issues, like an unpleasant workplace environment.
"So they have to consider these factors and they might need to change what they're doing. But otherwise, just try to focus on the good."
Signs of back-to-work blues
Feeling disoriented and taking a "go-slow" attitude
Little interest in work or focusing on your next holiday
Feeling irritable, in a bad mood or suffering headaches
What can you do to reduce it?
Organise your work space and give your desk a personal touch
Think about learning some new skills this year
Arrange a job review to discuss ways to stay fulfilled and challenged
Ensure you have at least 15 minutes a day of personal time
Arrange after-work activities to look forward to on evenings or weekends
Look after yourself - get more sleep, exercise, and eat better
Source: Mental Health Foundation