Good question and one that researchers at the University of Queensland in Australia are actively studying. They have done a series of studies comparing patients with neck pain to normal subjects without neck pain.
They are looking at two groups of muscles. The first group are deep and attach to the bones of the neck. These are called the deep cervical flexors and include the longus colli and the longus capitus muscles. These muscles support the spine and keep it from buckling under the weight of the head.
Using computer models and electromyographic (EMG) studies they are finding out which muscles contract in normal adults versus patients with neck pain. They are also looking at when neck movement occurs based on which muscles are firing. The small motion required during nodding occurs because of the deep cervical flexors. When they stop firing as they should, other muscles take over instead.
The body tries to keep normal motion going any way it can. The result may be pain and loss of motion as the muscles stop working normally. The head nodding exercise is a powerful tool used to restore normal muscle activity. It's actually a retraining exercise. You are helping the deep flexors return to normal.