DIAGNOSIS This is a Bluetooth pairing handshake failure. The app uses Bluetooth to initially communicate with the speaker before connecting it to Wi-Fi.
FIX Force quit the Google Home app completely on your phone. Turn your phone's Bluetooth off and on again. Reopen the Home app and start the setup process anew. Ensure the Nest Mini is pulsing blue (indicating it's in setup mode) and that your phone is within 3 feet of it.
IF THAT DIDN'T WORK Your phone's OS or the Home app is outdated. Update the Google Home app from your app store. Also, check for any pending system updates on your iPhone or Android phone. Restart your phone after updates and try again.
HEADS UP During setup, make sure your phone is connected to the 2.4GHz band of your Wi-Fi network, not just the 5GHz. The Nest Mini needs to join the same band your phone is using for the initial configuration.