Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Improving gives high scores to some results, though visible (pubs) evidence is low and clinical score is zero #52

Open
sharatisrani opened this issue Apr 25, 2024 · 2 comments
Assignees
Labels
critical important to fix

Comments

@sharatisrani
Copy link
Collaborator

See these 3 PKs. In all cases, Improving gives scores of 1.0, though the pub evidence is low, and (thus?) no other ARA returns the result.

Cases with 1 piece of evidence and Sugeno Score it 5.0:

  1. https://ui.test.transltr.io/main/results?l=Autism&i=MONDO:0005260&t=0&r=0&q=eaf6e3f6-ce4e-4f65-8a91-e8e389c61e2e
    PK eaf6e3f6-ce4e-4f65-8a91-e8e389c61e2e
    Pregnenolone Succinate

  2. https://ui.test.transltr.io/main/results?l=Diabetes%20Mellitus&i=MONDO:0005015&t=0&r=0&q=f152d851-c03f-40d3-a4f1-54d68812e36a
    PK f152d851-c03f-40d3-a4f1-54d68812e36a
    Guar Gum - Spoke KP
    Indeglitazar - Improve Agent

Case where the top 3 results have 5.0 scores all have 2 pieces of evidence:

  1. https://ui.test.transltr.io/main/results?l=TFPI%20(Human)&i=NCBIGene:7035&t=1&r=0&q=dbcc0ce7-f6fd-4764-9b93-8fd3fc8d582e
    PK dbcc0ce7-f6fd-4764-9b93-8fd3fc8d582e
    (2e,3z)-2,3-bis[amino-(2-aminophenyl)sulfanylmethylidene]butanedinitrile
    U0126
    Sdccgsbi-0051184.p002

We need to understand on what basis Improving is scoring these so high, when there is no clinical evidence score for these either. If the basis is not changed, we will have to do something at the ARS scoring level.

@brettasmi @Rosinaweber @suihuang-ISB @sierra-moxon

@sharatisrani sharatisrani added the critical important to fix label Apr 25, 2024
@sharatisrani sharatisrani changed the title Imrproving gives high scores to some results, though visible (pubs) evidence is low. Imrproving gives high scores to some results, though visible (pubs) evidence is low and clinical score is zero Apr 25, 2024
@suihuang-ISB
Copy link

Hi Sharat - will look into that this weekend. For now, a possible cause and some context: Our score is based on empirical clinical evidence. Where such is lacking, which led to "no spore" and hence was a problem for the Translator, we have now a computed score based on network topology - but without using node weights learned from clinical data. This led to numbers that are by orders of magnitude lower than those based on clinical evidence. Brett then implemented a simple correction (rescaling) to make these two classes of answer be in the same range of score . One can suspect that this led to this error.

On another note - for terminological accuracy - as discussed once earlier: You seem to indicate that mere existence of a publication is "evidence". I know from some outside perspective his is what people think. But in medcine, there mere existence of a paper is far from evidence. And imProving does not check existence of papers for evidence. Even preclinical evidence (cells, animals) experiments is NOT considered evidence in clinical medicine - somehow we need to make the notion uniform. Not sure if we have done that.

@sharatisrani
Copy link
Collaborator Author

sharatisrani commented Apr 28, 2024

Look forward to the result of @brettasmi 's change.

I hear you on the definition of "evidence." I'm just conveniently referring to evidence generically, as the UI does.

image

@sharatisrani sharatisrani changed the title Imrproving gives high scores to some results, though visible (pubs) evidence is low and clinical score is zero Improving gives high scores to some results, though visible (pubs) evidence is low and clinical score is zero Jul 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
critical important to fix
Projects
None yet
Development

No branches or pull requests

2 participants