Diagnosis
The remote's motion sensor/gyro is having communication issues with the TV, or there's interference from other devices (like LED lights or soundbars).
How to Fix It
First, re-pair the remote to the TV. Point the remote at the TV and hold down the Back button (the arrow pointing left) and the Home button (the house) together for at least 5 seconds. You'll see a pairing message on the screen. Next, check for interference. Turn off any bright LED lights, especially strip lights behind the TV, and see if the responsiveness improves. Also, go to your TV's Settings > General > System > Location and turn OFF "Set Automatically" and any permission for apps to use location. This can free up Bluetooth bandwidth.
If That Didn't Work
Reset the remote's internal sensor. On a flat, stable surface, slowly rotate the remote clockwise three full rotations, then counter-clockwise three rotations. Then, press and hold the (Settings) and (Mute) buttons on the remote together for 3 seconds. This recalibrates the gyro.
Heads Up
The magic remote works on a combination of Bluetooth and infrared. Make sure the front of the remote (the IR blaster) has a clear line of sight to the bottom-right corner of the TV screen. Obstructions, even a coffee table book, can block the signal for button presses, though not the pointer.