More than 100 residents in Yopougon, a neighbourhood of the commercial capital Abidjan, started pulling apart the hanger-like structure on Sunday and built barricades out of burning tyres nearby.
A crowd returned on Monday, throwing rocks at police who dispersed it with tear gas.
"They want to kill us. We don't want this centre here," said protester Joel Blehi as he sheltered by a pharmacy after a gas canister was fired in his direction.
Police said the hostility arose from a misunderstanding that patients with COVID-19 would be treated at the centre.
"There's been a lack of communication. It's more like a testing centre for residents," police spokesman Charlemagne Bleu said.
The violence is the first sign of community resistance to the coronavirus response in Ivory Coast, where authorities have closed schools, places of worship and most shops and imposed a night-time curfew. The country has registered 261 confirmed cases and three deaths.
The centre is one of several being built in Abidjan for voluntary mass coronavirus testing, the health ministry said.No protesters or police were harmed in the clashes, but four arrests were made, Bleu said.