updated March 5 with Case 0
Ryan Avent (among others) comments on Ed Glaeser's piece on Skyscrapers: Density and Skyscrapers and seems to support allowing more high density development in cities, which are restricted by local government regulation.
The issue that does not really come out (though is mentioned in the comments) is the problem of zoning vs. externalities. The reasons for the lower zoning densities are many, but the rationale is externalities (and the desire to move nuisance law from courts to regulation to avoid the large transaction costs and uncertainties of the judicial system). If you increase density, as the neighbors know, you increase local externalities which they bear (through longer travel times, etc.).
Since no one owns the right to uncongested roads, developers (and planners who support them in their quest for the highest and best use) think they can just offload these costs to local neighborhood streets and it is okay. But what they do is take time from other people by increasing congestion. The neighbors think (presumably by custom, status quo, or some other logic) that they have the right to prevent these externalities, and they do so with restrictive zoning. Zoning regulates negative externalities that are not currently governed by Pigou or Coase.
Neither side is right, the problem at its core is undefined property rights and untolled roads. It is just two sides of selfishness: greedy developers vs. greedy NIMBYs.
I had a similar conversation with Jonathan Levine at the November Regional Science Conference. He asked if a developer could come and build a high-rise in an existing single-family residential area. I replied "good luck", suggesting the NIMBY's hold the cards. I also suggested this was a relatively rare case, which I think is because the zoning largely reflects the market in most places. I.e. there is not much market for high rise housing in largely single-family neighborhoods. There are of course edge cases. Systematically, there are three four cases when a developer is considering building somewhere:
(0) The zoning is not binding. In this case the zoning exceeds market demand, but there are negative externalities to development which the neighbors want to avoid.
(1) The zoning is not binding. In this case the zoning exceeds market demand, but the negative externalities are small.
(2) The zoning is binding, but the developer through either request or lobbying (perhaps at some monetary expense) gets an area rezoned. The lobbying allows political decision-makers to collect rents from restrictive zoning, or neighbors to achieve side-payments. These side-payments compensate the community for the negative externalities that will be received upon a change in the status quo. The political rents are a problem for the political system and how we finance campaigns or administer bribery laws.
(3) The zoning remains binding, but the land is not rezoned because the developer's payments were, or would be, insufficient to persuade the opponents or decision-makers.
Case 0 is a problem for the community, which must now pay the developer not to develop. We see this when communities purchase development rights (e.g. agricultural reserve areas). Here the Coasian right to develop resides with the developer One can see difficulties with Case 0 scenarios. Developers can threaten to develop, and get paid not to develop, even though they were not serious. Bluffing is very easy. Hence the desire for communities to control the development rights through restrictive zoning. Residents must act through government because of the asymmetry and coordination costs, 1000 neighbors all harmed slightly do not have the ability or incentive individually to give a side payment to a developer not to develop at all (or as much), because some residents will want to free-ride on the side payments of others. Only through the government or government-like organization can this be achieved.
Case 1 is not a problem for developers or the community, and the development proceeds without hitch.
Case 2 costs the developers money, but in the end if they choose to build, it still must generate above "normal profits", otherwise it would be better for the developer to keep their money in a bank account. This results in a transfer of money, but is at least neutral and probably win-win.
Case 3 might result in some social loss (especially if there are, in fact, economies of agglomeration), but what is happening is that the loss perceived by the opponents outweighs the benefits perceived by the developer. There may be of course miscalculations about the opponents willingness to pay or willingness to accept, but in the end the potential gains did not outweigh the potential losses, and the project may not have been as good as claimed.
Whether the zoning reflects the wishes of developers or neighbors depends on context, and as density increases, we would expect the neighbors to become relatively more powerful, if only because the negative externalities of development become apparent to more and more people.
As I implied above, there is not a moral right in retaining the status quo, but from a Coasian perspective, zoning creates the property right in a given level of externalities which rezoning proposes to change. There should still be an approximately efficient outcome in the end, if not for political rent-seeking.