The keying circuit rectifies a small sample of the radio's RF carrier to turn on the keying transistor, and then the relay.
If there's not enough RF voltage, there won't be enough current through the coil of the relay, and it drops back to receive position.
What's happening is that you DO have enough RF voltage on the amplifier's input socket at first before the relay closes, but not enough when the relay is activated. Causes the relay to drop, but now you have enough RF voltage again and the relay closes again. Lather, rinse, repeat.
The amplifier's input circuit needs to fool the radio into thinking that it's driving a proper 50-ohm antenna.
Sounds like the amplifier's input circuit is out of whack, causing the radio's RF voltage into the keying circuit to drop when the relay connects the radio to that input circuit.
Placing a SWR meter and a second coax jumper between the radio and the amplifier will reveal how bad the amplifier's input impedance match really is. You'll need to key it with the switch, set the meter to 'cal' and then read the SWR between the radio and the amplifier with the radio keyed. I predict you'll see a really high SWR reading.
If there is a slug-tuned coil in this amplifier's input circuit, you may be able to adjust it and bring down the input-side SWR.
And if you can't cure it with an adjustment alone this suggests that the input circuit has been messed up and needs to be fixed.
73