The easiest work around would be to purchase headphones of high impeadance that are diffacult to drive. That is why headphone amps are a thing so in hifi circles there are headphone amps that start around 1.5w for normal use that go up to 50w per channel when driving diffacult headphones and about 80-100w rms into 4ohms for speakers.Anyway, it is speaker output, not a headphones output, so signal is too strong. That might be a reason of poor performance.
Use some ATT before connecting headphones.