Diagnosis
The chuck is not seated properly on the spindle taper. Dirt or a burr on the mating surfaces is causing misalignment.
How to Fix It
Remove the chuck. Open the jaws fully. Insert the long end of a hex key into the chuck. Strike the hex key sharply with a hammer to unthread the chuck counter-clockwise. Clean the spindle and chuck threads with a wire brush. Reinstall by hand-tightening, then give it one final snug turn with the tool in drill mode.
If That Didn't Work
The spindle itself may be bent from a heavy side load. Inspect the spindle for visible runout without the chuck. A bent spindle means a gearcase replacement.
Heads Up
Do not use the tool with a wobbly chuck. It will damage drive bits and cause unsafe operation.