Rubicon Choose Results

[rcoreilly]

Results from the Rubicon choose model

This thread contains first-pass notes on results from the model.

Overall the model results show that it learns to accurately predict differential outcomes, cued by the CSs associated with each arm, and driven by the US outcomes experienced. The PFC layers, BLA, and VS Matrix show differential activity consistent with the learned expectations, and drive the goal-driven behavior of the model to make choices in favor of the good vs. bad outcomes.

"Bad" in most of the cases in this model just means "less good", as a result of lower expected value, but not actually negative outcomes. Thus, we do not expect the model to never select the "bad" option, and more discriminating choice behavior will require a mechanism for directly comparing options.

[rcoreilly]

June, 2024

All of the runs have 6 arms, each with a unique CS, organized into 2 groups with the same 3 US outcomes.

• 3 of which are "Good" (100% probability of reward of standard 1.0 magnitude, 1.0 effort factor).
• 3 of which are "Bad", by virtue of probability, magnitude, or effort manipulations

The comparison condition is ore00082 with "Bad" arms = 0.99 probability -- this has almost no effect, but causes the stats to mark the Bad arms as such. There is a small but significant bias in the default case in favor of choosing the 1st, left-hand group of arms, for reasons not yet fully understood (perhaps related to the zoolander property that the model only goes left). Therefore, we run the models with this first group as the bad group, to better reveal the learning effects against this small bias in favor of choosing the bad arms.

N=25 repetitions, run in parallel, to provide reliable results.

Bad US probability = 0.8

• ore00094 has 80% chance of reward
• ore00095 has 90%, for comparison

Behavior: Probability of selecting the lower-probability arm

Robust effect from a relatively small probability difference. 0.9 is actually relatively similar. This is the strongest of the manipulations, because the 0 outcome drives extinction learning in the BLA, which then inhibits the positive BLA. This can be titrated by the level of the inhibition, with these results with inhib = 1.

BLA Ext

The BLAPosAcq shows positive CS activity, which is the same between the Good and Bad cases, but the Ext line shows no extinction learning in the control case, but increasing learning in the p=0.8 case.

VS Mtx Go - No

The VS Matrix Go vs. No pathways are ultimately responsible for engaging the plan to approach an arm. This plot shows the activity in the Go pathway minus the No pathway, for the Good vs. Bad arms, also showing the ore00095 90% case, which has intermediate activity.

The difference is mostly in the Go pathway, although later in learning the No pathway does learn to oppose the Bad choices.

Learned Estimates of PVpos Outcomes

A critical feature of the model is that the PFC areas (OFCpos, ILpos) learn to predict the subsequent US outcome, at the time of the CS. The PVposEst layer receives from these areas, and is trained at the time of the US outcome with the actual positive value of the US.

The following graph shows this PVposEst value, at the start of the arm when first seeing the CS. It clearly learns to predict the difference in outcomes.

We also record the spread of activity over the PVposEst layer, as a measure of the expected variance in predicted outcomes. The following plot shows this variance (again at the time of the CS onset), showing that the p=0.8 and 0.9 cases have a higher variance relative to the 100% case.

Active Maintenance in PFC layers

The PFC layers show differential activity for the control vs. p=0.8 case, across all relevant layers.

OFCpos:

ILpos:

PL:

[rcoreilly]

US Magnitude = 0.2, 0.5

Overall, similar results are found with the manipulation of US magnitude.

• ore00088 = magnitude = 0.5
• ore00093 = magnitude = 0.2

Behavior

The key difference here is that the initial difference in behavioral choice weakens over time, because unlike the US omission case, the model learns to expect the reliable difference in reward outcome, and thus the RPE dopamine signal between conditions converges over time, reducing the overall difference among conditions.

DA RPE

This plot shows the decrease in DA over time, due to reward prediction:

BLA

The BLA exhibits the same differentiation among conditions, without any differences in extinction activity (not shown).

VS Matrix Go - No

Likewise differences in VS matrix activity:

PFC Activity

Similar differentiation in PFC activity:

OFC:

PL:

[rcoreilly]

Effort Cost

The effort cost manipulation shows similar effects. The effort costs are compressed so large effects are required to see behavioral effects:

• ore00097: Effort = 6x
• ore00098: Effort = 8x

Behavior

The increased effort cost causes the model to avoid the bad arms, as expected:

Prediction of Negative PV outcomes (effort)

The model learns to predict the increased effort cost for the bad arms:

Along with the variance in negative value:

VS Matrix Go - No

Similar differences:

[rcoreilly]

BLA vs. direct VS Matrix Go v. No

The model can make choices based on two main neural signals, at the time of the CS onset:

• BLAposAcq activity which indicates how strongly the CS is associated with a positive US outcome.
• PFC activity predicting the nature of the outcome (in OFCpos, ILpos etc).

Both of these feed into the VS Matrix Go and No pathways, which compete to make the choice about whether to approach a given arm.

In the probabilistic US condition, we can vary the strength of the BLAposExt -> BLAposAcq inhibition to control how much the BLA factor contributes to avoiding the bad outcomes. Inhibition strength is:

• ore00094: 1.0
• ore00099: 0.5
• ore000100: 0.2
• ore000101: 0.1

The behavioral effects show that the BLA makes an important contribution:

And here is the direct effect on BLAposAcq activity, when it is less and less inhibited:

and the effects of BLA on VS Matrix Go vs. No:

Thus, overall, it is clear that both the PFC and BLA effects contribute in roughly equal proportion to overall choice performance.