Diagnosis
You're likely not holding the factory reset button long enough, or you're pressing it while the device is in a weird state. The button is recessed for a reason—it requires a deliberate, long press.
How to Fix It
Here's the surefire method. 1) With the router powered on, straighten out a paperclip. 2) Gently press the tip into the recessed reset hole on the bottom until you feel the button click. 3) Keep holding it down. You'll see the light on top turn off, then start pulsing yellow. 4) Keep holding until the light pulses white. This can take 10-15 seconds total. Once it pulses white, you can let go. It's now reset and will appear as a new device in the Home app.
If That Didn't Work
If the light never changes, try the reset sequence while the device is powered off. Plug it in while continuously holding the reset button with your paperclip, and keep holding for 15 seconds after the light comes on.
Heads Up
A factory reset wipes everything: your network name, password, and all settings. You'll have to set it up again from scratch in the Google Home app, just like when you first bought it. Make sure you know your internet service details before you start.