Synonymization/generality problems

[dkoslicki]

For test 7 asset 79, the ARAX result has progesterone as result 2 of 103. But the asset asks for progestin (the synthetic version of progesterone, iirc).

For test 12, asset 309, pegfilgrastim is asked for, but any filgrastim or G-CSF should be accepted.

For test 13, asset 362, the goal is to get ACE inhibitors and ARAX returns Angiotensin-converting enzyme inhibitors (i.e. the full name)

For test case 15, asset 318, Ivacaftor is asked for and ARAX returns Ivacaftor / texacaftor and ivacaftor /lumacaftor as results 2 and 3. Do we want to do the testing post-conflation?

Test 24, asset 396 Butyrate derivatives is asked for, but butyrate and butyric acid aren't accepted as answers.

Test 29, asset 504, thrombin is asked for, but alpha-thrombin is not accepted as an answer (similarly for test 30, asset 595)

Test 35 asset 665, KRASG12D inhibitor MRTX1133 is asked for, but MRTX1133 (CHEMBL.COMPOUND:CHEMBL5081048) is not accepted

[dkoslicki]

Similarly Test 9 asset 623, Rabeprazole is asked for, but Rabeprazole sodium is not accepted as correct. Again with the post/pre-conflation issue

[colleenXu]

Note this issue for Asset 362 / ACE inhibitors: #76.

There's a little more going on with this test. I also link it to the "too general" issue you opened.

[colleenXu]

We noted the same issues for some tests:

• asset 396 Butyrate Derivatives UMLS:C0581820: BTE returned entities that seem related like butyric acid CHEBI:30772, D-alpha-aminobutyric acid CHEBI:28797, and phenylbutyrate CHEBI:41500
• asset 665: there's another NodeNorm entity for MRTX-1133: UNII:DU32DM9CHD. I got that ID from Wikipedia

[colleenXu]

Here's some more tests with the same kind of problem (I'm closing their separate issues and moving the info here):

asset 32 Lactase `CHEMBL.COMPOUND:CHEMBL2108505`

(old issue)

• the NodeNorm entry for this ID seems odd: it has no other equivalent IDs (Dev NodeNorm).
• There are other lactase entities with more equivalent IDs that show up in BTE's results with higher scores/ranks: MESH:D043322 (ChemicalEntity) and UMLS:C0083183 (Protein)
• I've noted before that tools tended to fail this test. I haven't done this comparison lately though

asset 355 fosinopril `PUBCHEM.COMPOUND:55891`

PUBCHEM.COMPOUND:55891
(old issue)

• there's a different NodeNorm entity for FOSINOPRIL SODIUM (PUBCHEM.COMPOUND:73415812) with more IDs mapped to it. BTE gives this sodium-variant a higher score/rank (although it's still not in the top 10%).
• very similar to the rabeprazole example above

asset 397 hemin `PUBCHEM.COMPOUND:26945`

hemin
(old issue)

• the NodeNorm labels for this entity's IDs are "protoheme", not hemin
• Instead, I see "hemin" in the NodeNorm conflated-ID labels for hematin PUBCHEM.COMPOUND:455658. BTE consistently has hematin in its result set (often too low for a TopAnswer, but still).
• It's not clear to me if the conflation of hematin and hemin in NodeNorm is okay or not. Wikipedia notes the difference in ions and also has hemin IDs that map to other NodeNorm entities
• think all tools have been consistently failing this test, which may be in part due to the answer ID choice: Tests that no one passed #69

asset 595 thrombin/F2 `UMLS:C0040018`

(old issue)

• While UMLS:C0040018 is a protein ID, it doesn't have any equivalent IDs.
• Instead, I wonder about using the same thrombin IDs that I proposed in Asset 503: bad answer ID for thrombin/F2 gene #83 : UniProtKB:P00734 or NCBIGene:2147 (primary since we're doing gene-protein conflation). BTE has thrombin/F2 as a top result in its result sets using those IDs.
• See Asset 504: bad answer ID for thrombin, duplicate of 503? #84 (comment) and Synonymization/generality problems #62 for notes on different thrombin NodeNorm entities.

[maximusunc]

So I think this gets to a root question: The Test is asking for a superclass identifier (in most cases) and an ARA gives back a subclass (more specific). Is the ARA expected to also return the superclass? If not, why not?

[dkoslicki]

From the subclass reasoning perspective (if a general concept is asked for, also return more specific instances), my impression was that precision is preferred to generality. But I guess it hasn't really been codified one way or another. My $0.02 is that a subclass being should "count" for the test asking for a superclass. Open to other opinions though

[maximusunc]

I agree that more specific instances are preferred, but does that mean that the general concept that was asked for should not be returned?

If we were to support subclass answers in the tests, how exactly would that work? Is there an easy way to determine if something is a subclass of the expected output?

[maximusunc]

This general topic will be discussed during the testing session of the upcoming Relay.

[sandrine-muller-research]

wrt "wrong output IDs": I guess not all tests are from me but the purpose of the test assets is to test the UI so when the UI returns something wrong (e.g. superclasses) the ID given by the UI is reported.

wrt synonymization/generalization:

• since the testing is testing the UI, the testing is done "post conflation" if the creative query returned by ARA does the conflation. At least that's how I made the tests.
• When a super set family was outputed the question I was asking myself was "are all BioLink entities in that superclass" passing/failing. If not then it is "too general" and is a failure.
• for the thrombin case: I have not idea why alpha-thrombin is not accepted because I did not write a fail test for that one.

[colleenXu]

Asset 79 also has this issue. It's been running in sprints 5-6.
The answer is "progestin" MESH:D011372, but this is a drug class with ~ 60 individual chemicals. Some of those individual chemicals (including progesterone itself) are high-ranking ARA results and should probably pass...

No one has been passing this test is sprints 5-6 (also noted previously).

This test is also flagged in #93