The 24-year-old from Middlesborough was seen at London nightclub Mahiki before leaving with several girls and some critics fear their late nights can affect performance.
But James hit back: “I have been working very hard and doing things constant trials.
“We went to a comedy gala and then I went to have a drink with J Union for literally an hour. Was not even drunk to be fair.
“It’s frustrating if people think that I’m constantly partying but I try to ignore it.
“While a good performance and put the work in that is what matters.
“We are in a bubble here and under pressure so sometimes we need a release.”
James also insisted that there was no chance of collapsing again. At the beginning of the live stage of the competition in October suffered a severe asthma attack caused by stress and X Factor bosses had to call an ambulance for studies.
Speaking at a concert TalkTalk X Factor, James said: “I seem to have gone from strength to strength since.
“I think he was trying to cover too much advice and taken too.
“I was overwhelmed and very intense.
“Now I trust my own judgment again and stayed true to myself.”