Diagnosis
The 'Assistant volume' is set independently from the media playback volume. They are separate controls that get out of sync.
How to Fix It
While the speaker is idle, say "Hey Google, volume 7" or use the touch controls on top to adjust the volume. This sets the *Assistant* volume. Then, ask a question to test it. You can also adjust this baseline in the Google Home app under Device Settings > Audio > Assistant volume.
If That Didn't Work
Check for 'Night Mode' in the device's Audio settings in the Home app. If enabled, it lowers the assistant volume during set hours. Disable it to test. Also, ensure 'Audio ducking' (lowers media when assistant speaks) is turned on for clarity.
Heads Up
Volume adjustments via your phone's physical buttons or slider while media is playing only affect media volume, not the Assistant's baseline level. You have to set the Assistant volume separately when no media is active.