DIAGNOSIS This is typically interference, either from other Bluetooth devices or from physical obstacles between the remote and the box. The remote uses Bluetooth, which can get choppy in crowded wireless environments.
FIX First, reduce interference. 1) Power down any other nearby Bluetooth devices like headphones, speakers, or keyboards. 2) Make sure there's a clear line of sight between you and the Apple TV. The box's front-right corner is where the antenna is. 3) Re-pair the remote. Go to Settings > Remotes and Devices > Remote and choose 'Unpair Remote,' then pair it again using the close-proximity method (hold TV button and Volume Down).
IF THAT DIDN'T WORK There might be a software glitch. Restart the remote itself. Go to Settings > Remotes and Devices > Remote and select 'Restart Remote.' Also, restart the Apple TV via Settings > System > Restart. This clears any system lag affecting input.
HEADS UP If the lag only happens in one specific app (like Netflix or Hulu), the problem is that app, not your remote or Apple TV. Try force-quitting the app (double-press the TV button, swipe up on the app preview) and reopening it. Check the App Store for updates to that app.