The seven-story steel and concrete structure in the coastal town of Sihanoukville, west of the capital Phnom Penh, was a Chinese-owned project.
At least 23 people were injured and many workers were still missing and feared trapped inside, according to the office of the spokesman for the local province of Preah Sihanouk.
"The steel structure has collapsed on itself and we don't dare move it," the spokesman Oar Saroeun told Reuters. "We can only wait and listen for any signs of life...
"We are afraid more of it will collapse on them.... We will work through the night to remove the steel."
A statement issued later on Saturday by Preah Sihanouk Province officials said 36 workers had been rescued from the debris. It was not clear how many more were missing.
Photos of the scene shared on social media showed groups of rescuers working their way through a crumpled heap of steel girders and concrete.
Preah Sihanouk province and its largest town, Sihanoukville, has seen a rush of investment in recent years from China, especially into the casino, property and tourism sectors.
Home to Cambodia's largest port and a Chinese Special Economic Zone connected to Beijing's Belt and Road initiative, the town is also undergoing a construction boom to serve those growing crowds of Chinese tourists and investors.
Police have detained four people, including three construction supervisors, for questioning in relation to the accident, according to a statement from the province.