Some 3,000 police have been deployed to monitor the situation and arrest those who might resist having their property dismantled or burned down.
Those with nowhere to go were taken to a holding location outside the city.
Police warned residents not to interfere with the operation, dubbed "Restore Order." But the razings still sparked riots in some townships late Thursday.
Since last week, police have detained some 17,000 people in Harare who operate unlicensed street stalls and flea markets.
Thousands of Zimbabweans lost their jobs during the country's economic recession, and have turned to the underground market to survive. Seventy percent of the population is unemployed and the country regularly faces food and gas shortages.
Some information for this report provided by Robertson and AP.