Other things being equal (and, to a good first approximation, they are), when one man marries two women, some other man ends up marrying no woman. When one man marries three women, two other men don't marry. When one man marries four women, three other men don't marry. Monogamy gives everyone a shot at marriage. Polygamy, by contrast, is a zero-sum game that skews the marriage market so that some men marry at the expense of others.
For the individuals affected, losing the opportunity to marry is a grave, even devastating, deprivation. But the effects are still worse at the social level. Sexual imbalance in the marriage market has bad social consequences and many grim ones at that.
Two political scientists, Valerie M. Hudson and Andrea M. den Boer, pondered those consequences in their 2004 book Bare Branches: Security Implications of Asia's Surplus Male Population. Summarizing their findings in a Washington Post article, they wrote: "Scarcity of women leads to a situation in which men with advantages — money, skills, education — will marry, but men without such advantages — poor, unskilled, illiterate — will not. A permanent subclass of bare branches (unmarriageable men) from the lowest socioeconomic classes is created.” unquote
In China and India, for example, by the year 2020 ‘bare branches’ will make up 12 to 15 percent of the young adult male population."
The problem in China and India is sex-selective abortion (and sometimes infanticide), not polygamy; where the marriage market is concerned, however, the two are functional equivalents. In their book, Hudson and den Boer note that "bare branches are more likely than other males to turn to vice and violence." To get ahead, they "may turn to appropriation of resources, using force if necessary." Such men are ripe for recruitment by gangs, and in groups they "exhibit even more exaggerated risky and violent behavior." The result is "a significant increase in societal, and possibly inter-societal, violence."
The problem is that those who favour legalized polygamy only see the effects on the partners. It does not "harm" the partners, therefore it's fine.
That it has social ramifications far beyond the home is considered irrelevant.