Private hire drivers are taking legal action against London Mayor Sadiq Khan over the congestion charge.
The group, which includes Uber drivers, says the charge is discriminatory as 94% of them are from black, Asian and other minority ethnic backgrounds.
London's "black cab" licensed taxis will remain exempt.
The mayor's office says it is not prepared to ignore the damaging impact the rise in private hire vehicles is having on congestion and air pollution.
From 8 April, private hire vehicle drivers will have to pay the £11.50 daily congestion charge to drive in central London, under rules introduced by the mayor.
Uber driver Abdurzak Hadi says that as he drives in central London from Monday to Friday, he will be almost £60 a week worse off.
"I will be punished for coming to work. This is a tax on poor drivers," says Mr Hadi.
Most drivers, such as those working for Uber, will have to pay the charge themselves and cannot pass it on to passengers, because it is the company that sets the rates for fares.
London has roughly 114,000 private hire (PHV) drivers, who are overwhelmingly from black, Asian and other minority ethnic backgrounds, and this is what has led to a legal challenge.
The percentage figure comes from a report to the mayor entitled "Changes to the Congestion Charge", which followed a consultation which Transport for London says received 10,000 responses.
The report says: "As the majority of PHV drivers (about 94%) are from black, Asian and minority ethnic backgrounds (BAME) and many are from deprived areas, there is a disproportionate impact on these groups."
However, it assesses the impact as being "minor adverse".
The report also includes analysis showing that a majority of black cab drivers are white British.
The Independent Workers' Union of Great Britain (IWGB), which represents private hire drivers, is seeking a judicial review of the mayor's decision on the basis that it indirectly discriminates against BAME PHV drivers.
On Friday, the union began that process by writing a pre-action letter to the mayor.
Indirect discrimination is covered by the Equality Act 2010. It occurs where there is a practice, policy or rule that is applied generally to a large group but a sub group that possess a particular 'protected' characteristic ends up being treated less favourably.
Those characteristics include race, age, disability, sex or sexual orientation, religion or belief, gender reassignment, maternity and pregnancy, marriage or civil partnership.
IWGB general secretary Dr Jason Moyer-Lee calls the plan "regressive" and "both discriminatory and fundamentally unfair".
"We would urge the mayor to adopt one of the many alternative policies which would actually address congestion, instead of just penalising low-paid ethnic minority workers," says Dr Moyer-Lee.
He argues that if the minimum wage was paid to all private hire drivers, companies would control the number of drivers because they would not want cars circulating without paying passengers.
TfL figures show licensed private hire drivers in the capital have almost doubled in less than a decade, from 59,000 in 2009-10 to 114,000 in 2017-18, while black cab drivers have fallen from 25,000 to just under 24,000.
Last summer, New York capped its total number of private hire vehicles, and London's mayor is pressing ministers to give him similar powers to control their numbers in the city.
The government has lost a number of legal challenges by environmental group ClientEarth over harmful levels of air pollution, and councils are under pressure to address the problem, with Birmingham and Manchester looking at imposing congestion charges.
In a statement, the mayor's office said: "The number of private hire vehicles entering the congestion charge zone has shot up from 4,000 a day in 2003, when it first came into operation, to more than 18,000 now.
"Sadiq simply isn't prepared to ignore the damaging impact this has on congestion and increasing air pollution.
"Congestion has a crippling impact on businesses across the capital.
"At the same time, our toxic air in London is a major public health crisis that is stunting the lung development of our children, leading to thousands of premature deaths, and increases the risk of asthma and dementia.
"Removing the congestion charge exemption for private hire vehicles is a key part of our plans to both reduce congestion and to protect Londoners from harmful emissions from polluting vehicles."
However, the mayor's own assessment says overall traffic will only be reduced by 1%, whereas private hire traffic will be reduced by 6%.
Dr Moyer-Lee says this shows "the biggest change envisaged by the mayor is not a real reduction in overall traffic but rather a shift away from minicabs to other vehicles".
The mayor's office points out that only around a third of PHV drivers enter the congestion charge zone, so the majority will not be affected by the changes.
It assesses the annual cost of congestion in London at around £5.5bn and predicts that, without action, by 2041 it could take more than an hour to travel 10km by road in central London, 15 minutes longer than today.