Five Labour mayors have urged Rishi Sunak not to scrap, delay or scale back HS2 as it would “leave swathes of the north with Victorian transport infrastructure that is unfit for purpose”.

Sadiq Khan, Andy Burnham, Tracy Brabin, Oliver Coppard and Steve Rotheram say they have been “inundated” with concerns from constituents about the potential “economic damage that will result from any decision not to proceed with HS2 and Northern Powerhouse Rail (NPR) in full”.

The regional mayors issued a shared statement to express dismay at the prospect of the UK government scrapping the rail project’s northern leg, ahead of a collective meeting on Wednesday.

The cabinet minister Lucy Frazer, when asked if they would listen to the mayors’ plea not to cut the rail project further, said the prime minister and chancellor “listen to a wide variety of voices”.

Frazer, the culture secretary, told Sky News: “I’m sure the prime minister and the chancellor listen to a wide variety of voices. But as you will know, it’s the responsibility of the government to keep all projects under consideration. And that’s what the chancellor is doing. He is, as he does on all matters which are spending billions of pounds of taxpayers’ funding, looking at a whole range of projects to make sure that they are value for money.”

Asked whether HS2 would run to Manchester, she said: “Well, that is a decision, as you know, for the chancellor, not for me.”

The mayors’ joint statement said: “Investment in transport infrastructure is a huge driver of economic growth – creating jobs, increasing productivity and opening up new business opportunities. HS2 and NPR will deliver this right across our regions.

“This government has said repeatedly that it is committed to levelling up in the midlands and north. Failure to deliver HS2 and NPR will leave swathes of the north with Victorian transport infrastructure that is unfit for purpose and cause huge economic damage in London and the south, where construction of the line has already begun.”

The five mayors urged the Northern Powerhouse Rail project to be delivered in full to ensure “not only north-south but west-east connectivity between Liverpool and Hull, via Manchester airport”, which they say must be non-negotiable.

The five-way statement added: “The UK does not need a new line that only goes from Birmingham to Old Oak Common, which is six miles from central London.

“This does nothing for the north of England. The full Y-shaped HS2 plan was designed to deliver economic benefit right across the country not only between the north and London but between Leeds, Sheffield, Manchester and Birmingham. All of these gains look set to be lost if media reports this week are to be believed.”

Sunak has faced a political backlash over reports he is considering axing the Birmingham to Manchester leg of HS2 amid soaring costs.

Government sources briefed the Times on Monday that the prime minister may offer to fund an underground rail station in Manchester as part of a package of transport investment in the north aimed at winning the support of Burnham, the Labour mayor of Greater Manchester.

Such a compromise would mean phase 2 – taking in a Birmingham to Manchester leg – would be delayed by up to seven years.

On Tuesday, John Stevenson, chair of the Northern Research Group of Conservative MPs, has signalled they may be open to a compromise in which the second phase of HS2 would be delayed for several years.

“At the end of the day, we think HS2 is important for the country. But our east-west connectivity, I think, would be a higher priority,” he said.

A number of senior Tories, including the former chancellor George Osborne and the ex-deputy prime minister Lord Heseltine, have said scrapping the Manchester leg of HS2 would be a “gross act of vandalism” and an abandonment of the north and Midlands.

Before Wednesday’s meeting, Khan said: “Over recent days we have seen a justifiably horrified reaction from businesses and communities across our regions concerned about the economic damage that a decision not to proceed with HS2 and Northern Powerhouse Rail in full will cause.”