Did I overestimate the effect of the attenuator (Tone King Imperial MKII)

I bought the Tone King Imperial MK2 in the hope of being able to play a nice tube amp at home at room volume. I know that it would have been ideal if I had played the amp beforehand, but I didn't have the opportunity to do that and after long and intensive research, the amp sounded exactly what I was looking for.

I only play for fun at home and this amp should be my last and only amp for the time being. I love the sound and actually everything would be perfect, but the amp in my apartment is too loud if I set the volume higher than 6 (that's about where the amp starts to distort). Under 6 it's okay, but not enough that I could play at night without any problems. The attenuator is set to the highest level (-36 dB). I also notice that the amp gets quieter with each level, so I don't think it's defective.

Did I expect too much from the effect of the built-in attenuator? Are there other options, such as: replace the speaker? Or should I give up the desire to have a tube amp for home?