ARA Poll: Are you making "one-hop" treats predictions based on foundtional treats refactor predicates?

[mbrush]

I pose the following question for "rule-based" ARAs (BTE, Unsecret, ARAGORN) as a discussion thread to collect answers and debate solutions. @andrewsu @colleenXu @webyrd @kaiwenho @cbizon please respond in a comment.

After the treats refactor, is your ARA implementing rules/templates to perform "one-hop" inference of treats edges based on the new "foundational" predicates defined as part of this refactor (e.g. in clinical trials for, applied to treat, studied to treat, treats or studied or applied to treat)?.

For example, the inference that X treats or studied or applied to treat - Y (support path/edge) ----> X - treats - Y (predicted edge)

WE ASK THAT EACH ARA CREATE A COMMENT BELOW TO ANSWER THE QUESTION, AND IF RELEVANT PROVIDE DETAILS ABOUT WHAT ONE-HOP PREDICTIONS THEY ARE MAKING,. THANKS YOU!

---

TL;DR version:

• As background, recall that one pillar of the 'treats' refactor is that KPs will replace many treats edges they had been creating with edges that use a more "foundational" predicate to capture more precisely what a knowledge source is actually reporting to be true. e.g. that a chemical/drug is in clinical trials for a condition, or applied to treat a condition, or beneficial in models of a condition - in cases where the evidence used by these sources is not strong/direct enough to warrant a treats assertion edge.
• However, the CQS will allow these KPs to define simple templates that will 'bring back' treats edges as predictions, with the more foundational edge as a one-hop support path.
• We collectively made (but perhaps failed to adequately circulate/document) the decision that these so called "one-hop" predictions would be the remit of the CQS to make.
• It has recently come to the attention of the TAQA team that some ARAs may have implemented rules/templates in their own systems to make these one hop predictions as well - so that their ARA returns 'treats' prediction edges that the CQS is also returning.
• We have identified one ARA that is doing this, and are trying to figure out if there are others.

The ask for each ARA here is to report here if they are generating any one hop predictions based on the new predicates introduced by the treat refactor ('in clinical trials for', 'applied to treat', 'studied to treat', 'treats or studied or applied to treat').

We need to get a sense of this to decide how to move forward w.r.t. if we want this to happen, and if so, how to mitigate some of the downstream consequences of this happening. Thanks!

[kaiwenho]

Unsecret is generating such one-hop predictions based on the refactored treat predicates. At this stage, the system only produces a single type of inference: - X treats or studied or applied to treat - Y (support path/edge) ----> X - treats - Y (predicted edge). I had planned to extend this capability to other refactored "treat" predicates as we incorporate more updated data from KPs. However, since you mentioned

We collectively made the decision that these so called "one-hop" predictions would be the remit of the CQS to make.

I will pause this plan for now and wait for the final decision before determining whether to cancel or continue.

[andrewsu]

We collectively made the decision that these so called "one-hop" predictions would be the remit of the CQS to make.

I don't recall this decision being made. And if we did make it (my memory is horrible these days, so entirely plausible), I think we didn't completely think through the implications for the automated testing. BTE does make these assertions, and they are a key part of passing the "Top Answer" tests. If you wanted BTE to not use the "foundational one-hops" to create the direct X - treats - Y predictions, we almost certainly would not pass the automated test thresholds, and I don't see a realistic alternate to getting to a passing state. I suspect that would apply to most other ARAs as well.

[cbizon]

Aragorn is using treats_or_... but not on its own to infer edges. It is using it as a component of a bigger rule. So for instance we might have a somewhat weak rule that goes (a)-(whatever)-(b) that on its own is just suggestive but in parallel with (a)-treats_or_...-(b) is a lot stronger than either on its own. In the UI these two paths, which are really part of one rule, get pulled apart and shown independently.

FWIW, I agree with @andrewsu - I might have misunderstood / misremembered but I don't see a problem with ARAs also using these rules; I thought the CQS was kind of a backstop rather than sole source.

[mbrush]

My takeaway from the 9-3-24 Architecture Call is that we want to allow for ARAs to generate one hop treats predictions if they wish. One reason for this is that awareness of these predictions will allow ARAs to more accurately score treats prediction results that may also be supported by multi-hop rules or other prediction methods.

Details of this approach:

• One-hop 'treats' inference logic is defined in a CQS 'template' by the KP that brought the use cases, with input from the CQS team and other KPs/ARAs who may also want to generate or use the prediction
• The CQS template repo provides a central location to document/share one-hop inference logic
• The CQS would continue to generate one-hop 'treats' predictions - acting as a backstop to ensure that all useful predictions get made and returned to users
• ARAs would be able to replicate these predictions in their implementations, but would be required to follow the same inference logic that is encoded in the CQS templates.
• Should an ARA find something they want to change in the inference logic, a ticket should be filed in the CQS repo and a broader discussion should happen to try and align.
• End users will see the CQS and ARA(s) as sources of the one-hop predictions in the UI.

A couple concerns raised about this approach:

• Users looking as sources of one-hop 'treats' predictions will see the CQS and ARA(s) - which each executed the same reasoning template to produce the result. This appearance of multiple independent sources may lead to users to think this is a stronger result than is warranted. Also, users have to do the work now of exploring wiki pages for many ARA and CQS sources if they want to see how each reasoner made them . . . only to discover (if they are able to) that they are actually all based on the same reasoning/evidence/method.
• Similarly, the final O&O scoring may be skewed too high due to the duplication - as they may see multiple ARAs/CQS generating the same prediction, where each is based on the same underlying support/method.

Alternate approaches to the one above that we might consider:

• Alternative 1: make it so that the CQS merely acts as a shared ‘source of truth’ repository for defining logic for one-hop prediction specifications, but does not actually implement this logic to generate treats predicted edges. Here, we would need to systematically ensure that each 'treats' prediction specified by an existing template is executed by at least one ARA - so the predictions are not dropped from Translator.
• Alternative 2: Make it so that the CQS is the only place where these one-hop treats predictions get generated, and then have ARAs retrieve / access these predictions dynamically at scoring time so they are considered when scoring ARA-generated 'treats' predictions. This would avoid duplicate predictions being made in the first place, and mitigate the concerns outlined above. But require changes to ARA scoring tools.
• Alternative 3: reconsider the argument that ARAs benefit from access to the one-hope 'treats' predictions when scoring a Result - given that they will have access to the foundational one-hop edge supporting the prediction in their KG, and this more foundational edge should provide more fine grained information than they have before the treats refactor should they tune their scoring tools to use it. (Recall that for any 'treats' edge they had access to before the refactor, they will have access to the more foundational edge that replaced it after the refactor. This edge provides more detailed/specific information that could be even more useful/discriminating in helping to score a result, should the ARA scoring tools be tuned to allow for this.)

One we settle on an approach, the EPC/CQS team can draft some guidance documentation to be sure the rules we settle on are clear and followed.

[rhubal]

Fwiw, was swayed by Matt's observation that duplicated results may suggest to the user a more powerful result than might o/w be warranted.
Only to say however implemented the UI and user's expectations should be considered.

[andrewsu]

we will draft some guidance / conventions to ensure that ARAs consult and replicate one-hop reasoning logic encoded in CQS templates

Probably I should just wait until that guidance is written, but I also want to be sure I understand the general implications here. If I look at the CQS MVP1 templates, I see a number of query templates, both multi-hop and one-hop. Is the idea that ARAs should preserve the priority order listed for these templates (e.g., mvp1-template1-clinical-kps before mvp1-template2-connections-hypothesis before mvp1-template3-openpredict) while potentially adding additional ARA-specific templates interspersed therein?

[mbrush]

When you say "consult and replicate", do you mean that an ARA is envisioned to run a workflow query that mirrors the CQS? Or just query the CQS and return their results mixed in with the ARA's? . . .

Sorry if 'consult and replicate' was unclear. What was meant is that if an ARA wants to create its own one-hop 'treats' predictions based on in clinical trials for` edges, it should look at the template defined for this type of one-hop prediction in the CQS repo, and follow the same logic in its own implementation of the inference (e.g. w.r.t. constraints on the trial phases that warrant this prediction).

"Source of truth" makes me think "lookup" and so the Multiomics KP (Clinical Trials and Drug Approvals and the like). Is that what is intended here?

I meant 'source of truth' here as in the authoritative specification of a rule for a one-hop prediction - not 'source of truth' as in a provider of established facts.

Is the idea that ARAs should preserve the priority order listed for these templates (e.g., mvp1-template1-clinical-kps before mvp1-template2-connections-hypothesis before mvp1-template3-openpredict) while potentially adding additional ARA-specific templates interspersed therein?

The numbering of these MVP1 templates simply reflects the order in which they were create. If BTE wants to implement its own one-hop 'treats' rules/templates, it should look to see if the inference they want to make is already represented in the repo. If it is, their internal implementation should follow the same logic. If it is not, then they should add a new template to the CQS repo that specifies the inference logic, and implement their own rule/template in the BTE ARA system that follows the same logic. In this way, the CQS can still serve as a single place where all such one-hop treats predictions are specified - so any ARA that wants to duplicate it can replicate the same logic in their implementation. Not sure this answers your question though @andrewsu

[andrewsu]

If BTE wants to implement its own one-hop 'treats' rules/templates, it should look to see if the inference they want to make is already represented in the repo. If it is, their internal implementation should follow the same logic.

So as an example, suppose BTE wants to infer "treats" edges from aeolus. I would consult mvp1-template4-service-provider-aeolus.json, which would tell me that I should filter for biolink:evidence_count > 20 because the CQS team has determined that is optimal (or at least we're being consistent among all teams who use aeolus). Do I have that right? If yes, then is there specific logic we should try to infer from mvp1-template3-openpredict.json? That seems to be a one-hop query on openpredict without any additional filtering. Thanks!

[mbrush]

@andrewsu re: your first question concerning how BTE might consult and replicate one-hop CQS templates - yes, I think you have this right. To clarify though, CQS is not the one deciding the optimal prediction logic. This is a collaboration between the CQS and the KP that brought the use cases, and other KPs/ARAs that may also want to generate or use the prediction. This central/collaborative process is one of the points of the CQS. In the case of the aeolus one-hop prediction, it was your team that provided this use case, and we decided based on your input the appropriate logic / count constraints should be for this prediction.

As for what can be learned from the openpredict template - this is a special case. OpenPredict is generating their own treats predictions through some sophisticated ML approach, so the treats prediction from open predict is essentially a lookup of a predicted edge. The CQS is not executing any path inference logic for them, as in the other templates. They are just using the CQS to surface their predictions, and create the appropriate support into. (t,b,h, I don't think they really need to be using the CQS at all - but that's another matter).

Finally, I'll say that once we settle on what role the CQS will play in the area of informing/generating one-hop treats predictions, and what ARAs need from it, we can build better metadata/documentation to guide this interaction.