Jacob Rees-Mogg has accused civil servants of using flexible working to skive off as he demands an audit of the practice across Whitehall.

In one of his last acts as government efficiencies minister before a new prime minister takes office, Rees-Mogg has warned departments against the “overuse” of flexi-time and called for detailed lists of how widely it is used.

He is threatening to sell off the office space of departments that fail to reduce working from home as figures show some Whitehall areas “nearly empty”.

New polling, however, indicates that about half of staff across the country work from home some of the time, saying it has improved productivity as well as work-life balance.

Rees-Mogg, who is expected to stay in the cabinet under Liz Truss and could become business secretary, has repeatedly clashed with unions as he tries to drive staff back to Whitehall.

Allies say he is also concerned about “flexi-time” arrangements that allow civil servants to work irregular hours, such as putting in longer days through the week to take a day off. They say he fears that civil servants are “using the system combined with mass working from home to just not work”.

He suspects they are going shopping or to the gym during working hours. Rees-Mogg wrote to all cabinet ministers on Thursday to say that he was “concerned about . . . the overuse of ‘Flexi-Time Deals’ across the civil service”. He asked them to review use of the practice.

He wants to know how many staff use such deals, including those on “informal flexible working arrangements”. He asks for “details of any controls . . . to ensure civil servants working flexibly are, in fact, working their contractual hours” by the end of the month.

With latest figures showing fewer than half the desks in some departments being used, Rees-Mogg says he is “disappointed by how we still have buildings that are underused and some nearly empty”.

He warns: “Where departments are not able to achieve consistently good levels of attendance across every day of the working week maximising the usage of building space, especially in expensive locations like London, I will now ask the Government Property Agency to reduce the building space sublet to that department.”

Dave Penman, head of the FDA civil service union, accused Rees-Mogg of an “ideological obsession that any flexibility for civil servants is somehow a skive”. He said flexi-time has been used since the 1960s.

“It means civil servants work extra hours when the demand is high and take time off when it can be more easily accommodated,” Penman said.

Polling commissioned by the union from the consultancy Public First found that 49 per cent of adults say they work from home some of the time, including 15 per cent who say they never go to an office. Among those who work from home, 66 per cent said it was easier to focus. Many said, however, that it harmed relations with colleagues.

Penman said: “Our polling shows almost half of workers have some opportunity to work flexibly from home, demonstrating that this quiet industrial revolution has been embraced across the economy. Rather than standing in the way of progress like Luddites, ministers should embrace the opportunities this provides to save taxpayers’ money on office accommodation.”