That's correct!
There's a skill you can develop that lets you push people around more effectively if they live in worse lodgings than you do.
Otherwise, only color. They fit into play only when you're doing things in them.
There are a few experiences that poor lodgings might keep you from marking, but if you don't care to mark those experiences, and you don't mind your neighbors, you're right, there's no reward for moving up.
-Vincent