Hmmmmm, unusual for sure. Some random thoughts:

Memory steer on these is usually caused by a stiff knuckle joint. I'll assume that's not the cause as loostening the ball joints cures the symptom. Have you driven it with the hubs locked to see if it improves?... Login to read more.