Beetle hit the nail on the head. From page 21 of the operation manual:
If your transceiver is configured for “RF Gain” use, rotating
this control fully clockwise in the SSB/CW/Digital modes
will provide best sensitivity. To reduce the receiver’s RF
Gain somewhat, rotate this control counter-clockwise
slightly.
You will observe an increasing number of bars on
the S-meter as you rotate the RF Gain control counter-clockwise;
this indicates increasing AGC voltage, which is causing
the front-end gain to be reduced. In the FM and Packet
modes, this control will automatically be set to “Squelch,”
even though the setting of Menu Mode No-080 [SQL/RF
GAIN] is “RF Gain.”
If this control is configured for “SQL” operation, the FT-
897’s RF Gain will be set for maximum sensitivity in all
modes, and the SQL/RF Gain control will function solely
as a Squelch control. In this case, rotate the SQL/RF Gain
control to the point where the background noise is just silenced;
this will provide the best sensitivity to weak signals,
while keeping the receiver quiet when no signal is received.
The LED just above the Main Dial will glow Green when
the squelch is opened by an incoming signal or noise.
The bold text talks about an increased number of bars on the S meter as you decrease RF gain, but if you increase the rf gain (reduce AGC voltage) the bars on the S meter will decrease.