Different buildings have different footprint sizes.
Housing is 4x4, with the building itself taking 2x2 and the "yard" or "stage" taking up the rest of that area. The "visitor's track" (taken by the canned animations of the tax collector and the man with a sack looking for a house) can be set to the west or the south, which will advance 2 spaces and then return.
7x5 All Baths
5x5 All Courts
5x4 All Academy, Town Restaurant, Royal Workshop
4x4 All Housing, All Towers, All Farms, All Pastures, All Market, Royal Sanctuary, Blacksmiths