The Reward Circuit - a brief overview

My current research project makes use of positive feedback as a form of reward or incentive in learning. This is actually exciting as it combines motor learning processes and positive reinforcement. Reinforcement and reward-based learning implicate the limbic network, in which the basal ganglia, BG, is one of the most important members. Specifically, it is the ventral portion of the striatum called the nucleus accumbens (NAcc) and underlies a series of behavior e.g., motivation, and emotion. Initially thought as purely motoric, BG evolved to engage more diverse behavior such as executive, and then the limbic functions. The inclusion of the limbic region as a part of the BG has also been proven anatomically (Nauta, et al., 1978; Mogenson et al., 1980; Haimer et al., 1986). More importantly, the BG limbic circuit does not work in isolation. This summary will be based on an excellent review by Haber and Knutson (2010) and a few other relevant good stuffs will be provided as references at the end.

Neuroanatomy of Reward
Prefrontal Reward Regions
The involvement of prefrontal cortex in reward comes naturally. Scientists have been interested in studying the role of the frontal lobe in cognition in which reward processing is one of the main features. The main reward-based prefrontal regions traditionally include the anterior cingulate cortex (ACC) that includes BA 24, 25, and 32 and the orbitofrontal cortex (OFC) that includes BA 11, 12, 13, and 14. Unlike sensorimotor cortex, the prefrontal cortex is diverse in terms of cytoarhitectonic features and functions. The recent paper in PNAS (Neubert FX, et al, 2015) scrupulously describes similarities and differences between primate and human prefrontal cortex associated with reward. In general, the human prefrontal cortex is sub-divided into a few areas:
  (a) Sensory region: the orbital part of the brain linked to the olfactory bulb and the insula.
  (b) Ventromedial prefrontal cortex (vmPFC) that includes BA 10, 11, and 32.
  (c) Rostral OFC that covers BA 11, 12, and 13.
  (d) Dorsal ACC or dACC that is BA 24.

Reinforcement-based motor learning presumably implicates basal ganglia (Schultz et al., 1997; Graybiel, 2005). In fact, reward-based action may involve more complex neuronal processes beyond the traditional basal ganglia and sensorimotor loops. For example, it is possible that reward-based decision-making is also involved (Rushworth et al., 2004), such that during learning rewards may influence the production of subsequent movements. Prior studies show that regions in the prefrontal cortex are involved in this type of activity (Shima and Tanji, 1998). Using fMRI in gambling tasks (Daw et al., 2006), it has been shown that the intraparietal sulcus and frontopolar cortex are preferentially active during exploration. In contrast, regions of the striatum and ventromedial prefrontal cortex are involved in exploitative decision making to accumulate more rewards. The vmPFC is a region in which activity is associated with stimulus-reward value, selecting actions that are more rewarding (O'Doherty et al., 2003; Rushworth et al., 2004; Daw et al., 2006) and encoding the value of performed decisions (Knutson et al., 2001; Smith et al., 2010).

The Ventral Striatum
In 1954, Olds and Milner discovered how a tiny structure called the NAcc that lies ventral to the sensorimotor striatum is linked to reward behavior in rats. In 1978, Heimer described the link between NAcc and olfactory tubercle in rats. Ventral striatum was later regarded as the reward center and thought to be the interface between the limbic and motor systems (Mogenson et al., 1980). In recent years, the traditional boundaries of ventral striatum have expanded beyond NAcc which includes the ventral portion of the striatum and the ventral caudate. Thus, the name ventral striatum (VS) is legit to contrast this with the more dorsal, sensorimotor-related striatum. The NAcc structure has an outer shell and a core, each contains different neuronal cell types and functions.

(1) Afferent projections
Like the dorsal sensorimotor striatum, VS receives massive topographic glutamatergic inputs from the cerebral cortex, from the thalamus, and the brain stem. The word topographic deserves an emphasis, refer to Fig-2. From the prefrontal reward regions, vmPFC sends projections mainly to the NAcc. From NAcc, we go more dorsally and laterally to regions that cover ventral caudate nucleus and ventral putamen. Principally, these two areas receive projections from the OFC. The dACC also projects to a more central and lateral portion of caudate and putamen. Lastly, the DPFC (dorsal prefrontal cortex) terminates more diffusely along the rostro-caudal striatum, particularly the head of the caudate. A critical difference with the dorsal striatum, VS also receives projections from the amygdala and hippocampus. The afferent axons are concentrated within the NAcc. Amygdala is known to play a role in processing reward, e.g. an emotional aspect of reward, associating stimuls and reward or punishment. Thalamus is the final link connecting VS with the rest of the brain regions. In the thalamus, reward processing is managed by one of the largest nuclei there called the medial-dorsal nucleus (MD) that carries bidirectional wiring between the frontal lobe and VS, amygdala, and hippocampus.

(2) Efferent projections
Like the dorsal sensorimotor striatum, VS sends efferent axons to two main targets. First, the more specific efferent outputs to the ventral pallidum, not GPe or GPi. Second, more broad midbrain regions such as the ventral tegmental area (VTA) and middle substantia nigra (SNpc) which are rich of the dopaminergic neurons. The pathway between midbrain and VS is known as the mesolimbic pathway and is known to mediate reward processing. Other efferent projections are to the pedunculopontine nucleus (PPT) and nucleus basalis (NB, the principal source of cholinergic fibers to the cortex and amygdala). In particular, PPT carries broad functions such as arousal, attention, motivation, and voluntary limb movements.

Midbrain dopamine neurons
The involvement of dopamine in reward is perhaps first shown elegantly by the work of W. Schultz since 1970s. The VTA and SNpc, the substantia nigra pars compacta, are two important components and their neurons can be identified using a certain phenotypic marker, e.g. calbindin for dorsal SNpc and VTA (the ventral SNpc, however, is calbindin-negative).

(1) Afferent projections
The principal inputs to VTA/SNpc come from the striatum through both the GPe and VP. Other inputs come from other parts of the brain stem, mainly PPT. Anatomically, the largest source of projection comes from the ventral striatum.

(2) Efferent projections
Midbrain dopamine neurons send massive projections out back to the striatum through the mesolimbic pathway mentioned above. It has been found that there is a medio-lateral and an inverse dorso-ventral topography arrangement. Thus, the ventral portion of SNpc projects to the dorsal striatum, the dorsal portion projects to the ventral striatum. It has also been observed that there is differential efferent projections back to the striatum. In other words, VS receives the least number of projections while the sensorimotor striatum receives the most. This striato-nigro-striatal network is consistent with the idea that the limbic system is able to influence sensorimotor behavior through the interface situated in the striatum. Apart from the striatum, dopamine neurons from the midbrain send rather diffuse projections to the frontal lobe through the mesocortical pathway, so it is wrong to say that BG is the only target of dopamine in the brain.


Further Readings
[1]  SN Haber, Knutson B. (2010). "The reward circuit: linking primate anatomy and human imaging". Neuropsychopharmacology.
[2]  FX Neubert, et al. (2015). "Connectivity reveals relationship of brain areas for reward-guided learning and decision making in human and monkey frontal cortex". Proc. Nat Acad. Science.

More on Basal Ganglia & Cerebellum

The themes concerning basal ganglia and cerebellum have been summarized in my previous blog post. Both structures have been traditionally known to play dominant roles in voluntary movements. However, as time goes by, such roles have developed into a wider perspective related to non-motor functions such as general learning, executive functions, and emotion.

Functional topography of Basal Ganglia
There is a convergence of cortical information in the striatum from the cortex. This means that axons of cortical neurons terminate on a far smaller number of striatal neurons, Similarly, the number of neurons in the pallidum and substantia nigra is smaller than that in the striatum, allowing further convergence along the direct and indirect pathways. There also seems to be a somatotopic organization with the cortico-striatal-thalamic pathways. Evidence mainly comes from animal studies using tracers systematically injected into the monkey brain. This includes projections from GPi to the M1, SMA, and premotor cortex (Hoover & Strick, 1993; Akkal, Dum, & Strick, 2007), to Area 7b in the parietal lobe (Clower, Dum, and Strick, 2007).

Important evidence showing how BG is involved in cognitive processes has been shown anatomically through the work of Middleton and Strick (2002). The authors injected a few different prefrontal areas such as Area 9m /9l, Area 46v/46d, and Area 12l ['m' and 'l', 'v' and 'd', 'l' stands for medial/lateral, ventral/dorsal, and lateral respectively]. They found that labels associated with these areas comprised almost 1/3 of the GPi, indicating the importance of BG for executive tasks.

Non-motor Aspects of Basal Ganglia
Although in the early days, scientists thought that BG is important only for sensorimotor functions, there has been considerable acceptance of the four parallel divisions or systems in BG. They include circuits responsible for the skeletal/sensorimotor system, oculomotor system, executive functions, and limbic functions (Alexander et al, 1990). The oculomotor system engages the frontal eye field (FEF) and the supplementary eye field (SEF) anterior to the dorsal premotor cortex (PMd). From these two areas, projections go into the caudate area of the striatum and in turn controlling the SNpr/GPi via the direct pathway before going out to the superior colliculus (brain stem). The oculomotor system is very important for saccadic eye movements and memory-guided saccades.  Refer to the figure below.

Fig-1:  Four different basal ganglia functional divisions (adapted from Kandel, 5e)

The prefrontal cortex subserves higher-order behavior such as cognitive control, reasoning, problem-solving, general attention, working memory, which are collectively known as the executive functions. Two areas, the dorsolateral prefrontal cortex (DLPFC) and lateral orbitofrontal cortex (LOFC), project to the dorsal caudate of the striatum. DLPFC is important for organizing behavioral responses to complex problems and using verbal skills in problem-solving. LOFC, on the other hand, controls empathy and socially-appropriate behavior. The limbic circuit [Nauta, 1986 for review] begins with projections from the anterior cingulate cortex (ACC) and ventromedial prefrontal cortex (MPFC) to the ventral striatum, which also receives input from the memory structures: hippocampus, amygdala, and entorhinal cortices. The ACC/MPFC complex is important for motivating behavior and reinforcement learning.

Patients with cerebellar damage
When a person performs skilled and goal-directed movements, the cerebellum maintains accurate and timely limb movements. One influential idea says that the cerebellum is crucial for error-based motor adaptation. Prior evidence that point to this idea comes from studies involving visuomotor and force-field adaptation (e.g. the works from Amy Bastian or Reza Shadmehr). In 1970s, Marr and Albus independently suggested that the cerebellum may be involved in motor learning. This is associated with the concept of plasticity between the Purkinje cells and parallel fibers inside the cerebellar cortex. Ito later proposed the role of complex spikes from the climbing fibers as the learning signal. Experiments using motor adaptation have shown that people with cerebellar damage impacting climbing fibers aren't able to adapt to both force-field and visuomotor perturbation.

The output gates of the cerebellum are located in the deep cerebellar nuclei in the white matter. These nuclei send projections back to the cortex via the thalamus, and they are strongly excitatory necessary to maintain muscle tone. This, however, isn't related to strength but more on the timing. Lesions of the interposed nucleus reduce the accuracy of reaching movements because of errors in timing, in direction and extent, in straightness due to poor joint coordination. The hand oscillates irregularly around the target. Neurological examinations show that damage to cerebrocerebellar path delays movement timing (Holmes G., 1939). A complex movement can be decomposed into a sequence of movement components. In healthy people, this decomposition is not obvious as the movements are performed smoothly and timely.

Experiments with primates help to elucidate the role in the motor disruption. When a monkey is trying to keep its arm in a fixed position, the application of a force to extend the elbow triggers a stretch reflex in the biceps. This will pull the arm rapidly and precisely to its initial location. The contraction of the extensor triceps plays a dominant role in keeping this precision, preventing the elbow from overshooting after the biceps contract. This appears as an anticipatory mechanism, i.e. feed-forward control. When the two deep cerebellar nuclei, dentate and interposed nuclei are deactivated, the arm oscillates instead of firmly going back to the original position. There seems to be an excessive, yet inaccurate, feedback correction (Vilis, Hore, Flament, 1984 & 1986).

Different Cerebellar Lobules
Like the cerebral cortex, the cerebellar cortex can be divided into four functional divisions: the vermis, intermediate zone or paravermis, lateral hemispheres, and flocculonodular lobe. Animal studies, in particular, in rats and cats have been used as models for anatomical studies. Perhaps, the most popular and classic human cerebellar anatomy comes from the works by Larsell & Jansen (1970). Refer to the Fig-2 below. The lateral hemisphere is especially important because of its massive connections with the cerebral cortex. Both vermis and the lateral hemisphere ('H') are further divided into nine subdivisions or lobules, each assigned a Roman number I to IX.

The earlier attempts to localize cerebellar functional organization were done as a result of lesion studies in both higher-order mammals and humans. Like the sensorimotor cortex, the existence localization and cerebellar somatotopic map do not represent the bodily extent, but rather, the functional demand. A series of the different somatotopic maps were released by Bolk (1904), Adrian (1943), and Snider & Stowell (1944). The progress continued with the help of tracers and, more recently, neuroimaging. Using fMRI, Grodd et al. (2001) did an elegant study producing sensorimotor topography of the cerebellum. The study was based on 46 human subjects performing a series of motor tasks such as opening/closing of right/left fist, extending right/left arm and leg, and moving the lips. I also recommend another article by Manni & Petrosini (2004) in Nature Rev. Neurosci. for a more elaborated summary with historical contexts. The following diagrams are taken from their paper.

Fig-2: Gross anatomy of the cerebellum (right, after Larsell & Jansen) and functional map (left, after Grodd et al.). Both vermis and the lateral hemisphere ('H') are subdivided into lobules. Of utmost interest is the left somatotopic map. The arms are represented by Lobule V-VI. In particular, fine motor control of the hand and fingers are by Lobule VI and VIII of the lateral hemisphere. Note that the connections to bodily parts are ipsilateral.



The involvement of non-motor functions are shown by the neuroimaging work by Stoodley et al (2012), They found right-handed finger-tapping activated right cerebellar lobules IV–V and VIII. Verb generation engaged right cerebellar lobules VI and Crus I and a second cluster in lobules VIIB–VIIIA. Furthermore, mental rotation activated medial left cerebellar lobule VII (Crus II). Lastly, 2-back working memory task activated bilateral regions of lobules VI–VII.

Important Anatomical Connections
Studies elucidating specific connections between the cortex and cerebellum have been done using anatomical tracers in animals. For example, retrograde transneuronal transport with herpes simplex virus, HSV1, was injected into the cerebral cortex of Cebus monkeys to label neurons in the dentate nucleus (Dum & Strick, 2003, Akkal, Dum, & Strick, 2007). With sufficient survival time, it was enough to reveal a set of first-order neurons in the thalamus, and second-order neurons in the dentate nucleus. Dentate nucleus was targeted by researchers at the time as the shape is the largest and easily recognized. The study showed different dentate output channels to the motor areas (M1 and SMA) and non-motor regions in pre-SMA, Area 7b, Area 46, and Area 9L.

Another important study adopting both retrograde and anterograde tracers is by Kelly & Strick (2003) that reveals the existence of cortico-cerebellar loop in both motor and non-motor domains. In the motor loop, neurons in M1 project to cerebellar lobules V, VI, and HVIIB and HVIII, and project back to the same regions of cortex via dorsal parts of the dentate nucleus and the motor thalamus. In the prefrontal loop, however, Area 46 projects to lobule HVIIA/B (mainly to Crus II) via the pontine nuclei, and returns back to the same areas of the prefrontal cortex via ventral parts of the cerebellar dentate nucleus and prefrontal thalamus. This segregated domain also confirms the neuroimaging works mentioned above, separating motor and non-motor areas in the cerebellum. Lobules that are connected to prefrontal regions are considered non-motoric. Refer to Fig-3.

Fig-3:  Anatomical projections from the cerebral cortex and cerebellum forms loop (from Kelly & Strick, 2003)

Basal ganglia and cerebellum receive inputs from cortical motor areas and send projections back to the same areas. These multisynaptic links suggest the extent of influence of both structures to movement production. Most anatomical connectivity has been revealed by tracing studies done by Strick and his group. For example: injections of retrograde tracing to a specific area in M1 or SMA (e.g. arm) with right survival time have revealed how neurons in the dentate nucleus and internal segment of the globus pallidus are labeled. In fact, there seems to be clear functional segregation (e.g. arm, face, digits) in both dentate nucleus and GPi. This suggests that neurons in both the cerebellum and basal ganglia do project to the motor cortex. However, there is no direct connectivity between basal ganglia and cerebellum.

Basal Ganglia & Cerebellum - a brief overview

The previous blog post talked about the "hierarchy" of the sensorimotor system, ranging from the highest order of the brain (cerebral cortex) down to the brainstem and the spinal cord. There are two more parts of the CNS that are essential in motor control and we'll discuss them here.

Fig-1: The position of the cerebral cortex, basal ganglia, brainstem, and cerebellum in the motor hierarchy with ways how they influence each other. Both the corticostriatal loop and the corticopontine-cerebellar go through the thalamus.

Outputs from the motor cortex are connected to several subcortical nuclei called the basal ganglia (BG) and the cerebellum. These two compartments are known to influence and modulate movement performance such as accuracy and timely execution. Basal ganglia are especially known to give Go/No-Go signal to the cortex. The role thalamus is particularly essential to convey the information back to the cerebral cortex from the corticostriatal and the corticopontine-cerebellar loop. To complete the topic, it is worthwhile to look at the left diagram.

What are Basal Ganglia?
They are essentially a group of subcortical motor nuclei that include:
(1) The striatum, comprising the caudate nucleus & putamen. They form input channels to BG.
(2) The globus pallidum structures: external (GPe) and internal (GPi) segments.
(3) Substantia nigra at the base of the midbrain.
(4) Subthalamic nucleus (STN), located below or ventral to the thalamus.

The striatum is located at the base of the forebrain lateral to the thalamus and is separated by the internal capsule into the caudate and putamen internally. The structure is 95% composed of a specific type of GABAergic neurons called the medium spiny neurons. Depending on the location, these neurons express different dopaminergic receptors (D1 or D2). Substantia nigra contains two distinct divisions. First, the more medial pars compacta (SNpc) contains dopaminergic neurons that send projections to the striatum. Second, the more lateral pars reticulata (SNpr) contains GABAergic neurons. Together with GPi, SNpr is considered the BG output channels.

Refer to the diagram shown in Fig-2. The concept of Go/No-Go comes from the fact that basal ganglia are able to enhance or inhibit a movement through the influence of two major pathways: the direct (solid lines) and indirect pathways (dotted lines). Specifically:

• Excitatory inputs from the cortical areas increase the inhibitory signal to GPe, causing a decrease in inhibitory activity to STN. Therefore, GPi will be more excited and then send more inhibition to the thalamus. In turn, cortical areas receive less excitation. This is the indirect pathway.
• D1 and D2, two dopaminergic receptors in the striatum are able to maintain balance so that the level of excitation is not out of order. In Parkinson's Disease patients, these receptors are disrupted by the damage to the substantia nigra pars compacta. As a result, there will be no inhibitory control for both GPi and GPe. An overproduction of inhibition causes thalamus to produce less excitatory signals back to the cortex. Muscles and posture become more rigid. 
• These excitatory input signals from the cortex are able to inhibit GPi through the direct pathway. As a result, it brings less inhibition to the thalamus, bringing more excitation to the cortex. Thus, projections back to the cortex have to go through the thalamus. Outputs from the GPi also go to the brain stem. 
• More recently, scientists have shown direct projection from the frontal cortex to STN that bypasses the striatum called the hyperdirect pathway. Thus, this pathway is also independent of D1/D2 influence. The exact function is still debatable.

Fig-2: The cortico-striatal loop showing indirect (dotted lines), direct pathway (solid lines), and hyper direct pathway. 

The BG receive most of their inputs from the cerebral cortex, notably the primary motor cortex, M1. And as mentioned, they also receive dopaminergic inputs from the SNpc. This has a strong implication; it means that the cortical information coming from the cerebral cortex can be modulated by the dopaminergic system. The way BG influence movement can be seen by studying two main motor pathways, i.e. direct and indirect pathways. Both pathways carry projections out through the GPi (globus pallidus interna) as the exit channels. Neurons from these output channels project back to the cerebral cortex and the brainstem through the ventromedial thalamus. The increased activity in one pathway will reduce movement execution, while the other, increase it. Probably, insights on the role of BG to movements come primarily from the studies with Parkinson's Disease patients.

Fig-3: The anatomical divisions of Basal Ganglia and Cerebellum, taken from [1, 2].

What is Cerebellum?
The cerebellum is located posterior to the brainstem. It is smaller than the 'big brain', but contains almost double the number of neurons in it. The cerebellum exerts influence on the ipsilateral side of the body but contralateral to the motor cortex. Cross-sectionally, the cerebellum can be divided into:
(1) Outer gray matter: contains a repeated pattern of circuitry; see below.
(2) The inner white matter with 4 pairs of output nuclei.
(3) Cerebellar peduncles: like highways connecting the cerebellum and the rest of the brain.

White matter nuclei of the cerebellum are the fastigial, the anterior and posterior interposed, and the dentate nuclei. The inputs to the cerebellum are conveyed primarily through the inferior and middle cerebellar peduncles, whereas the outputs are conveyed primarily through the superior cerebellar peduncle. Like the cerebral cortex with its four lobes, we traditionally divide the cerebellum into four divisions: the vermis, intermediate zone, lateral hemisphere, and the flocculonodular zone; the areas best seen if we unroll the cerebellum.

The cerebellum is intimately connected with the motor nuclei in the brainstem, cortical areas, and the thalamus. Frontal motor and parietal lobules are the principal cortical inputs to the cerebellum (Glickstein, 1980s). Refer to my diagram below. Based on the functional circuitry, there are 3 different divisions of the cerebellum:

1. The vestibulocerebellum regulates balance during stance and gait and controls eye movements. It has afferent connections to the vestibular system.
2. The spinocerebellum regulates movement execution of the body and limbs. This is achieved based on somatosensory afferents from distal and proximal limbs, trunk, and other body parts from the spinal ascending tracts. It also receives sensory information from the audio-visual and face areas. Adrian & Snider (c. 1940) found that there is a somatotopic map represented in the area of the vermis and paravermis. Outputs from interposed nuclei are connected to the lateral descending tracts, whereas fastigial nuclei to the more ventromedial tracts. Spinocerebellar circuits modulate movements by monitoring incoming sensory inputs and perform the online adjustment. 
3. The cerebrocerebellum is involved in skilled, timely, and accurate movement planning. It is thought that this area involves cognitive functions. Consistent with this notion, it is the most extensive area in humans, more than that in monkeys. It has direct connections with the motor cortex through pontine nuclei that loop back to the cerebral cortex through the thalamus. 

Fig-4: Summary of 3 functional pathways of the cerebellum shown with their INPUT and OUTPUT connections. 

Cerebellar synaptic organization
The cerebellar gray matter layer contains an intricate, repeated pattern of 'computational' circuitry that is shown in Fig-5. There are two main axons of cerebellar inputs: mossy fibers and climbing fibers. These two types of fibers provide a strong excitatory influence on the cerebellum.

1. Mossy fibers carry information through two pathways. First, direct projections from the spinal cord such as somatic information from skin and muscles. The projections are collectively known as spinocerebellar tracts. Second, indirect projections by going through different brainstem nuclei, e.g. pontine nucleus and reticular formations. Mossy fibers influence Purkinje cells indirectly through inhibitory cerebellar interneurons.
2. Climbing fibers carry information strictly from the inferior olive nucleus located at the midbrain. This structure, in turn, receives axons from the vestibular nuclei, tectum, dorsal column nuclei, and cortex. Climbing fibers make direct contact with the Purkinje cells. Unlike mossy fibers, climbing fibers have the capability to elicit complex spikes. Scientists believe that these spikes occur mostly when there is sensory prediction error e.g. during learning or adaptation.

Purkinje cells are the only output from the cerebellar cortex to deep white matter. They have massive dendritic branching, allowing multiple synaptic inputs, and carry post-synaptic inhibitory (GABAergic) outputs to the cerebellar nuclei.

Deep cerebellar nuclei always yield excitatory outputs to the thalamus before sending the flow back to the cerebral cortex. However, this excitatory state is modulated by two things: direct influence from mossy/climbing fibers and from the Purkinje cells.



Fig-5 Synaptic organization within the cerebellum. There are multiple loops within the circuitry. The dentate nucleus is the largest cerebellar nuclei that influence the motor cortex via the thalamus. Thus, the cerebellum is able to influence voluntary movement.



References
[1]  Chapter 42 - Cerebellum. In Kandel E.R. et. al. (2000). Principles of Neural Science 4e, McGraw-Hill.
[2]  Chapter 43 - Basal Ganglia. In Kandel E.R. et. al. (2000). Principles of Neural Science 4e, McGraw-Hill.
[3]  Chapter 8 - Control of Movement. In Carlson, Neil R. (2013). Physiology of Behavior 11e. Pearson Education.