At least since the dawn of agriculture, when this instinct really took off and dug in, so to speak, we've been quite territorial as a general tendency. Not in the sense of being unwilling to share space with anything else; on the contrary, we fill space with just about as much noise and bustle as we can pack into it, just so long as that noise and bustle is organized to serve whomever has the power to alter the environment and benefit from that alteration. Herein lies the territoriality.
Once it was a dominion over the land one claimed, a claim usually backed by some kind of organized political power. Now it lies largely in a sense of proprietorship over whatever one purchases, including the temporarily bought labor of other human beings, which is often tied to the land, things built on the land, and things which interact with the land to produce the things we want.
The ability then to affect change in one's living environment is a coveted one that is often hard won with social permissions, and on the backside of these social permissions is the threat of sanction. This is especially the case the denser arrangement one enters into, as typically there is a larger cultural, economic, and political harmony that one must harmonize with and at the same time be accepted by to interact with it in a meaningful way, or else it is altered with enormous expenditures of energy, which require social power.
After all, there is a wide range of possible aesthetic expressions, manifestations, and functions in the modern industrial - and globalized - world, and these things take social power and resources to permit, and once they are expressed and manifested, they must be frozen into place and maintained with all of the wrath of the legal and enforcement mechanisms backing them.
There are many ways to lay down a wall, or carve out and pave a pathway or road, or to put up a house, or to arrange a garden or even landscape, and an equal number of ways for these things to visually appear, and then the ways in which these things eventually come to pass are also dependent on available traditions, techniques, skills, materials, and social impetuses for organization.
One steps foot in the typical modern American city for example and finds it to be essentially a home for cars, and all of the infrastructure bends around this fact, a state of affairs that is owed to a particular form of culture, political economy, and lineage of historical events with their own ascents of particular interests. Which work just fine for certain classes and individuals, and for others, there is profound alienation and dispossession. Some wiggle room certainly, but how much, and where, and when?