Hahaha! Whats wrong with nudists?
To pick up your metapher.
The mentioned people never have payed for any access granted - they swelled in, as one little boy unauthorized sneaked in and got hold of the key of the entrance door and shared this key to everyone else. Some people knew, that it was against the law to break into a house and get the key. Nethertheless they ignored this fact and used their own key to go in to settle down, because they always liked to live in such a house, but never had enough skill to build one themself. (later people just didn't know of what the very first boy did and just found the house open to them). Not only did they settle down, no, they changed the rooms to their likings, using weird or nice stuff to fill it, again without asking. But there were still parts in the house, they could not reach.
The owners, as you said, decided to live in the house as before, renewing and repairing it, ignoring the strange, impatiently, anyway mostly nice people. They couldn't just lock the people out again, as most of them just really didn't know, what they were in fact doing wrong - and the owners weren't moneybegging lawers anyway.
Occationally, the owners had to paint a room, which was used by the people. And the people disliked it of course, because it got uncomfortable for them. But they just got around that problem shortly after anyway, were able to paint their room again and all was well. Also the people complained, that the owners didn't want to let them paint all of the house, including the non accessable rooms. The owners tried to explain, that he thought, not all of their paint was a good match for the house and that these were the very private rooms. The people didn't really understand, but couldn't help it.
One day the owners bought a dog (the people had already a few dogs in the house and liked them, but the owners didn't like them as they were wild mixes). They build a new hut for it, right beside their house, including a small door and a lock, so he was save. It was made that way, because it was the best way not to force such a dog to live in the house.
The people recognised, that their key didn't seem to fit well into the lock of the hut and one of them started arguing, what could be inside and demanded an open hut. Why? And what would he want to do with the dog anyway? Petting?
Anyway, the other people didn't care much for long, as they still could live well in the house and do what they wanted to do in their rooms.
End of the story.