Diagnosis
This is almost always a router compatibility or signal strength issue, not a speaker hardware fault. The speaker is losing its stable connection.
How to Fix It
Move your router and speaker closer together, ideally within clear line of sight. In your router settings, ensure the 2.4 GHz band is enabled (Nest Audio needs it) and try assigning a static IP address to the speaker to prevent address conflicts.
If That Didn't Work
The speaker may be conflicting with other smart home devices on a crowded Wi-Fi channel. Log into your router and change the 2.4 GHz channel to 1, 6, or 11. Also, in the Google Home app, try disabling 'IPv6' in the Nest Audio's device settings under 'Technical information'.
Heads Up
Nest Audio has known issues with certain mesh network systems and routers that use band steering. If possible, temporarily test it on a different, simpler network to confirm the problem is environmental.