Diagnosis
This is a context bug. Your iPhone thinks you want to control its internal sound, not the HomePod you're playing to. It loses the handoff connection.
How to Fix It
The surefire way is to re-establish control from the source. Open the app you're playing audio from (like Music or Podcasts). Tap the AirPlay icon (the triangle with circles). In the list of speakers, tap your HomePod Mini. Once the audio is actively routing to it, the physical volume buttons on your iPhone should control the HomePod's volume (you'll see a HomePod icon next to the volume slider).
If That Didn't Work
Force quit the audio app and the Home app, then reopen them. Also, check that your iPhone and HomePod are on the same Wi-Fi network. If you have multiple Homes set up in the Home app, make sure the HomePod is in the same 'Home' as your iPhone's current location.
Heads Up
This glitch happens most often if you use Siri on the HomePod to start music, then try to adjust volume from your phone. Starting playback directly from your iPhone is usually more reliable for maintaining volume control.