Replies: 3 comments 1 reply
-
Systems-Level FrameworkMotivated / Goal-Driven (BOA / vmPFC / VS etc)First, the motor action system is strongly driven by the goal system, represented by the BOA model, where "limbic" ventral / medial areas provide critical inputs to motor systems that significantly influence which actions are taken, and when etc, in accordance with the "spiral" connectivity patterns identified by Haber et al, as shown in figure below. This is the outer loop of motor action.
Figure 1: BOA goal-driven framework. As one specific example that I tracked down in detail, #294 (reply in thread) InagakiChenRidderEtAl22 in Svoboda's lab showed that PPN (pedunculopontine nucleus) glutamatergic neurons drove a "Go" signal to initiate a licking response in their task. These neurons are under inhibitory control from a region of the SNr that is in the ventral lateral striatum (VLS) circuit, which is interconnected with the anterior insular cortex (aIC) (along with other orofacial motor areas). It is likely (though as yet unproven) that it is this aIC area that is integrating the auditory Go cue with the motivated plan to perform the action, and initiating a cascade of activity that ultimately results in the licking behavior. Key point: once we recognize that much of the actual decision making is taking place in the Ventral / Medial Goal system, the Dorsal / Lateral Motor system's job is significantly constrained: it mainly needs to deal with how and when to act, but not so much what to do. Overview Figure
Figure 2: The primary areas and circuits discussed below, all in rodent brain terms (primate translations below), arranged in rough anatomical locations (rostral / anterior to the left, caudal / posterior to the right). Frontal Cortex (all interconnected, not shown):
Striatum (input layer of the BG):
SNr (substantia nigra pars reticulata):
Thalamus:
Cerebellum:
Brainstem:
See ArberCosta22 #294 (reply in thread) for more details on complexity of these brainstem systems: they are highly intertwined and thus gross anatomical labels are not so useful here. Brainstem Motor ActionsThere are various levels of motor control areas in the Midbrain (e.g., superior colliculus (SC); pedunculopontine (PPN)) that provide descending control to the Pons, Medulla, and down to the Spinal cord (ArberCosta22). These motor control centers provide coordinated, relatively high level motor control "knobs" for higher areas to manipulate. Each of these areas sends efferent copy to various higher-level areas, typically including the deep cerebellar nuclei (DCN), which then projects to the VL / VA thalamus and the IL (intralaminar) thalamus, specifically the CL (central-lateral) nucleus which is a rostral area of the IL thalamus (IchinoheMoriShoumura00). This efferent copy is critical for providing a plus phase training signal into all higher areas (BG, cortex) about the nature of the action taken. In the specific and unusual case of the PPN, there is a glutamatergic projection into the FSI (PV) fast spiking interneurons of the striatum (AssousDautanTepperEtAl19), which results in the inhibition of striatal MSNs -- perhaps this provides a mechanism for suppressing other actions while locomoting. Thalamus (the central hub)Thalamus provides the central hub of connectivity for all motor areas, coordinating and integrating signals from disparate areas. There are three major types of thalamus of relevance to motor areas, providing three distinct types of signals, the first two of which correspond to the core vs. matrix distinction of Jones07, and the third is the intralaminar nuclei. See KuramotoOhnoFurutaEtAl15 #294 (comment) and ShepherdYamawaki21 #294 (reply in thread) and RovoUlbertAcsady12 (below). Unfortunately the anatomy is not "pure" according to the core vs. matrix distinction, but we nevertheless adopt a simplified terminology:
In reality, parts of VM are core-like while parts of VL are matrix-like, and VA is mixed, and varies across species. In general, BG gated thalamus is more anterior, especially in primates, as shown in RovoUlbertAcsady12 below. Core VL = Cerebellar specific motor parametersThe core type "VL" thalamus has relatively specific, focal projections into motor cortex, targeting the middle lamina (around layer 4), similar to sensory thalamic inputs into primary sensory areas. It gets excitatory (glutamatergic) inputs from the DCN (cerebellum), which drive IT (intra-telencephalic) and PT_upper (layer 5 upper) neurons during motor preparation to activate a detailed motor plan that contains information about all manner of motor control variables. See EconomoViswanathanTasicEtAl18 and GaoDavisThomasEtAl18 in #294 (reply in thread) The IT and PT_upper neurons have extensive reciprocal connectivity with VL / VA thalamus, and these recurrent loops are critical for the maintenance of motor plans (GuoYamawakiSvobodaEtAl18), as is continued DCN input GaoDavisThomasEtAl18. In summary: VL is critical for subcortical training of detailed motor cortex representations of action parameters, via temporal difference between motor plan and actual motor action taken, and for generating these motor plans prior to actual action. Matrix VM = BG gated timingThe matrix type thalamus has very broad, diffuse projections into motor cortex, targeting the superficial lamina (layer 1), preferentially hitting the tufted branching dendrites of layer 5 PT_lower neurons. These PT_lower neurons directly project to the brainstem motor areas, largely bypassing the thalamus, and are clearly responsible for directly driving motor actions. The VM thalamus gets direct inhibitory inputs from the BG output nuclei (SNr / GPi), and the classical disinhibition thereof then activates the VM thalamocortical projection, activating PT_lower neurons that then drive the motor action. This simple, classic motor output gating story is well supported by all of the work from Svoboda's lab. Because the matrix-style connectivity is so broad and diffuse, it can only specify timing information for when something should happen, and this is supported by the nature of the neural activity in motor areas around the "Go" cue window -- it is not action or sensory specific. Intralaminar (IL) = Corollary discharge / efferent copyThere are two different groups of IL nuclei: PF / CM (posterior) and CL (anterior), which notably only project to motor areas of striatum, not ventral striatum (though VS does have its own version of these). PF / CM = BG gating corollary back to StriatumThe posterior IL nuclei of PF (parafasciular) and CM (centromedian, which is only defined in primates, analogous to a part of PF in rats) receive a wide range of excitatory inputs from all the usual suspects (cortex, DCN, SC, PPN) and inhibitory inputs from the SNr / GPi (MandelbaumTarandaHaynesEtAl19, GrillnerRobertsonKotaleski20). The PF / CM then provides nearly the same amount of synaptic input to MSNs (and other striatum types) as the cortex does, along closed loop pathways such that each striatal area that drives a corresponding part of SNr then receives the net output of that SNr back through the PF projection (FosterBarryKorobkovaEtAl21). This then provides a corollary discharge signal back to the striatum, such that each MSN can know the net effect of the whole population for a given type of motor command. The PF projection targets the dendritic shafts, not spines, and is thus likely to be of a more modulatory, broad signal, consistent with the fact that it also targets the CINs. "The NMDA/AMPA ratio is around 0.5 for CL synapses, but 2.5 for PF synapses, suggesting that PF synapses are involved in long-term plasticity and facilitating plateau potentials, more so than CL synapses." In the model, the PF plays a critical role in the learning rule #326 (comment), determining the subset of neurons that receive the pure "credit assignment" component of learning, directly proportional to the sending and receiving activity. Thus, only the neurons that were involved in the actual action that was ultimately generated by the brainstem, as reflected in the ascending inputs into the PF along with the SNr output of the BG, receive direct credit or blame about the consequences of the action. CL = Cerebellum corollary and efferent copyThe most direct (disynaptic) pathway connecting the two major motor areas of Cerebellum and BG is via the CL (centrolateral) IL thalamus. This can provide a "plus phase" training signal to the BG striatum neurons for the actual motor parameters that were driven in the final motor command as executed by the brainstem and spinal cord. Specifically IchinoheMoriShoumura00 showed that the lateral cerebellar nucleus (LCN -- equivalent to the dentate in primates) projects into DL which then project into the spines (not shafts) of MSNs, specifically in the DLS striatum (i.e, primary motor striatum), precisely where a motor-specific training signal would be needed. CL drives MSNs strongly and effectively. Because LCN also receives significant efferent copy signals from lower brainstem / spinal areas, this CL pathway is also capable of driving plus phase motor signals based on an efferent copy of what actually went out to the muscles. This same information is also projected via LCN -> VL to drive a plus phase representation in M1, so it can learn the language of the brainstem. In the learning rule #326 (comment), the "delta" plus-minus phase difference is influenced by these projections. This is learned by all active MSN neurons independent of the PF gating, so they can learn and update their motor action coding even when they were not directly involved in the final action output. A key hypothesis is that this "exploratory" learning enables more to be learned in parallel from the final outcome, relative to just the core credit assignment component. Basal Ganglia: Descending Modulatory Control and Cortical GatingThree factors strongly suggest that the primary function of the BG is gating:
Putting all these together, it paints a picture where the BG serves as a broad, disinhibitory "gating" system on motor function, rather than a system that exerts fine-grained control over detailed motor parameters. Classically, this gating function has been described in terms of action selection, where the BG chooses one action from a "menu" of options, and in terms of action initiation, where the BG selects when to activate a motor action, the parameters of which are specified elsewhere (e.g., Mink, 1996). However, flying in the face of this logic, striatal MSN neurons do appear to encode lots of detailed motor parameters in their patterns of firing, leading many to conclude that in fact the BG does play a role in fine-grained motor control, particularly in its descending projection pathways into the brainstem, consistent with effects of lesions and other manipulations of various parts of the BG circuit (e.g., Grillner et al, 2020; Park, Coddington & Dudman, 2020, Arber & Costa, 2022). To resolve this critical question, it would make sense to actually look at what the SNr output neurons seem to be encoding. Somewhat surprisingly, in contrast to the large numbers of studies recording from striatal neurons, there are only a few reports on motor correlates of SNr activity (likely due in part to the relative difficulty in recording, due to the small size and depth of the SNr). One set of very influential papers from Hikosaka et al shows that SNr activity seems to act as a selection gate on saccade initiation. However, a more recent paper from Barter et al. (2015) shows fine-grained opponent-process gain control signals in SNr in relation to body movement parameters. When combined with anatomical data showing that different subsets of SNr neurons project to different brainstem motor nuclei, this later data suggests that the BG can indeed drive more specific motor control signals in its descending pathway. The opposing direct and indirect pathways from the striatum can provide a graded inhibitory and disinhibitory modulation of excitatory motor activity generated by other excitatory projections into the motor brainstem (from cerebellum and motor cortex). Thus, a "modern synthesis" of BG function suggests that it plays two separable roles: modulatory motor control via descending projections from the SNr / GPi, and broad modulatory gating of the motor cortex via ascending VM thalamus projections. It is also likely that the self-feedback into the striatum via the PF thalamus is similarly broadly-tuned. Critically, this more "hands on" view of BG function via the descending pathway helps resolve logical puzzles about the notion of "action selection". For example, if the BG is just selecting actions from a menu, how and where is the menu represented, and how would something like broadly-tuned VM thalamus connectivity enable selecting one action over another? If instead the BG is directly modulating action parameters via established motor control "knobs", then there is no need for a "menu" to be generated for each situation: the BG just learns to optimize its knobs, which can operate in parallel and continuously over time. The large numbers of striatal neurons in comparison to the lower-dimensionality of these control knobs (and recognizing also that ~40k such knobs across the SNr / GPi is still quite a large space) can be useful for allowing learning to encode various different specific sensory-motor situations and goal contexts for driving appropriate motor control signals. Computationally, there is a major advantage for learning in a systematic, high dimensional space where each possible action and parameterization thereof is distinctly represented redundantly, with each such parameterization getting different sensory and contextual inputs. Ultimately, the net effect of all the BG inhibitory output signals may well often serve to select certain actions (and action parameters) over others, but the critical point is that it is not dependent on any other system to present a small "menu" of possible actions -- it can learn to "drive" all on its own. This is consistent with the evolutionarily ancient nature of the BG circuitry, going back 500 million years (Grillner & Robertson, 2016), predating the neocortex. The broad nature of the thalamic pathways out of the BG (VM, PF) suggests that the cortex only gets a very general signal from the BG, which appears to be important for enabling the descending motor control signals in deep layer 5 to be "ungated" and actually sent, after whatever period of prior preparation might have taken place. Thus, the BG can help coordinate the timing of its own descending motor control signals with those of the cortex, via these broad gating projections. In many widely-studied lab tasks, this timing may be the main contribution that the BG provides, but in more real-world action contexts, the detailed descending BG control signals are likely critical. Finally, the actual motor activity is sent back up to the cortex (and BG) via ascending spinal cord and brainstem pathways, which provides a way in which the BG can help train the cortex to generate better motor commands in the future. Critically, this arrives along with input from the cerebellum through a common ascending pathway to the VL thalamus, instead of requiring a separate BG-only pathway for this detailed fine-grained information. The detailed circuitry of BG, in particular the updated "PCore" framework based on GPe as the central core of BG function #324, supports the idea that it is optimized for balancing Go vs. No factors in deciding whether and when to activate a motor action. Critically, as noted at the outset, much of the true initiation-level decision making happens in goal-driven ventral areas, so the dorsal motor areas are mostly about making things work well once the plan has been enaged. Cerebellum: Motor Program EditingThe DCN of the cerebellum are a major integration hub of ascending and descending motor signals, and are glutamatergic, excitatory neurons, projecting strong driving inputs to VL / VA thalamus analogous to retinal input into the LGN thalamus. These properties so clearly contrast with those of the BG, and support a division of labor between these motor areas: the cerebellum provides specific, detailed input to cortex, and brainstem motor areas, in an excitatory manner that can drive specific actions and action parameters. By contrast, the BG output is inhibitory, so that it depends on other areas for generating the raw material for actions, which it can then sculpt. There is also considerable evidence that the cerebellum is important for shaping motor actions on the order of 10s to 100s of milliseconds, while the BG is important for longer timescales. Despite the excitatory nature of cerebellar output, the purkinje cells in the cerebellar cortex provide a tonic inhibition of DCN neurons, and thus the function of the cerebellar cortex is also disinhibitory, like the BG. However, it is now increasingly clear that the DCN themselves are a critical locus of learning and function, independent of the cerebellar cortex (BroersenAlbergariaCarulliEtAl23), with the cortex providing a bidirectional modulation of DCN activity (DeZeeuwI21) that integrates across a much wider sensory-motor contextual scope to make motor actions better coordinated across time and across different effector groups. Also, unlike the BG SNr, there is not a massive downscaling of the main pathway signal in the cerebellum: the DCN neurons are essentially a huge fiber bundle of ascending and descending signals, and the purkinje cells have a highly focal relationship with specific such neurons, rather than having a broader gating-like dynamic across large numbers of them. Thus, purkinje cells are in a position to provide a precise and detailed modulation or "editing" of the motor signals coursing through the DCN pathway. The fact that cerebellar output goes directly into cortex and striatum (via VL / VA and CL thalamus) enables the edits made by cerebellum to shape learning in these other areas. Thus, cerebellum is also a teacher pathway, consistent with the general finding that its role is primarily during learning and early development more generally, and recent data from ChenFremontArteaga-BrachoEtAl14 shows that it can directly modulate learning in the BG. Within the DCN, the ascending efferent-copy signals and descending motor-control signals are organized into repeated connectivity motifs, involving the IO (inferior olive) pathways, which have been demonstrated to be able to learn to anticipate a subsequent motor control signal, in the classic eyeblink conditioning paradigm (BroersenAlbergariaCarulliEtAl23). In computational terms, a primary function of the cerebellum (both DCN and cortex) is thus to perform error-driven learning where the ascending efferent-copy signal serves as the plus phase teaching signal for all the sensory-motor inputs, so that a later, corrective motor action can instead be activated earlier in time. A synaptic tag-like delay in the granule cell synapses can provide this time-shifting mechanism, per our implemented model Verduzco-FloresOReilly15. This version of supervised learning in the cerebellum avoids the biggest problem with extant models: how does the IO (inferior olive) know when an error has occurred? In this new version, IO just provides a learning rate modulation but the raw learning signals are all via the DCN. This is a critical refinement of the widely-discussed Marr / Albus framing of cerebellum function, which put the IO entirely in charge of providing the error signal driving learning (via the climbing fibers), without explaining very clearly exactly how the IO signal could be computed. In terms of the model: the hypothesis is that if we stick to coarse-grained motor control at the level of discrete action choices, we can ignore the huge cerebellar cortex, which is more important for fine-grained, fluid, efficient control, and focus instead on the much smaller and more tractable DCN. Motor Cortex: Dual Ported, Open-ended LearningFinally, motor cortex, organized hierarchically as M1 (primary) and progressively higher premotor / supplementary motor and prefrontal motor areas, is in a position to synthesize more elaborate and detailed motor plans than any of the other areas. From the experiments on decerebrate cats (see GrillnerRobertsonKotaleski20) we know that cortex is not necessary for basic motor function. But nevertheless, other experiments show that finer-grained, more complex / detailed motor plans do require cortex. Cortex is "dual ported" in the sense of receiving both VL (focal, specific) and VM (broad, modulatory) thalamic projections, and has distinct subtypes of neurons that can: a) formulate motor plans via extensive bidirectional connectivity and specific thalamic input (IT, PT_upper neurons), and: b) directly drive motor actions triggered by BG "Go" gating signals, via VM "matrix" projections. Thus, in classic subsumption architecture style (Brooks86), the cortex fully integrates the two distinct motor subsystems of the cerebellum and BG, and is in a position to provide its own direct motor action commands at multiple levels of abstraction along the descending motor pathway. In terms of learning, the corticothalamic layer 6 CT -> VL pathway provides a pulvinar-like predictive learning mechanism, with ascending DCN -> VL projections providing the strong driver "ground truth" signal against which predictions are compared, using the same temporal-difference learning mechanism used throughout axon. This can extend across multiple hierarchical layers, mirroring the perceptual-side hierarchical predictive learning mechanisms, with higher layers predicting sequences of lower-level action steps. |
Beta Was this translation helpful? Give feedback.
-
RovoUlbertAcsady12This is the best map of thalamic organization, based on types of driver inputs (in macaque, but similar organization also in rodents). In general, thalamus serves as a kind of checklist for everything that the cortex does with respect to the rest of the brain -- and now our overall framework checks off a basic function for most (but not all!) of these areas.
|
Beta Was this translation helpful? Give feedback.
-
SPN / MSN DLS Learning RuleFour primary objectives:
Integrated learning rule:
Where:
Overall, this integrates a simple delta-based supervised learning signal for motor program learning and counterfactual logic, with standard three-factor BG DA modulated learning:
|
Beta Was this translation helpful? Give feedback.
-
Simplest Test Case: Sequence Learning
|
Beta Was this translation helpful? Give feedback.
-
This discussion is an attempt to take all the data reviewed in #294 and #324 and synthesize a complete motor action & learning system to be implemented in Axon.
Bits of attempted synthesis are present in those discussions, but this one is pure theory.
Beta Was this translation helpful? Give feedback.
All reactions