From 45e38415bb3af40566c13e8f6f3312d885326655 Mon Sep 17 00:00:00 2001 From: Katy Sadowski Date: Sat, 29 Jun 2024 16:46:29 -0400 Subject: [PATCH] add withinvisitdates, fix typos --- _pkgdown.yml | 2 + docs/404.html | 3 + docs/LICENSE-text.html | 3 + docs/articles/AddNewCheck.html | 5 +- docs/articles/CheckStatusDefinitions.html | 5 +- docs/articles/CheckTypeDescriptions.html | 5 +- docs/articles/DataQualityDashboard.html | 5 +- docs/articles/DqdForCohorts.html | 5 +- docs/articles/SqlOnly.html | 5 +- docs/articles/Thresholds.html | 5 +- docs/articles/checkIndex.html | 7 +- docs/articles/checks/cdmDatatype.html | 5 +- docs/articles/checks/cdmField.html | 5 +- docs/articles/checks/cdmTable.html | 5 +- docs/articles/checks/fkClass.html | 5 +- docs/articles/checks/fkDomain.html | 5 +- docs/articles/checks/isForeignKey.html | 5 +- docs/articles/checks/isPrimaryKey.html | 5 +- docs/articles/checks/isRequired.html | 5 +- .../checks/isStandardValidConcept.html | 36 +++--- .../measureConditionEraCompleteness.html | 5 +- .../checks/measurePersonCompleteness.html | 43 ++++--- .../checks/measureValueCompleteness.html | 17 ++- docs/articles/checks/plausibleAfterBirth.html | 5 +- .../articles/checks/plausibleBeforeDeath.html | 5 +- .../checks/plausibleGenderUseDescendants.html | 5 +- .../checks/plausibleStartBeforeEnd.html | 5 +- .../checks/plausibleTemporalAfter.html | 5 +- .../checks/plausibleUnitConceptIds.html | 5 +- docs/articles/checks/plausibleValueHigh.html | 5 +- docs/articles/checks/plausibleValueLow.html | 5 +- .../sourceConceptRecordCompleteness.html | 24 ++-- .../checks/sourceValueCompleteness.html | 10 +- .../standardConceptRecordCompleteness.html | 22 ++-- docs/articles/checks/withinVisitDates.html | 107 ++++++++++++++++-- docs/articles/index.html | 3 + docs/authors.html | 3 + docs/index.html | 3 + docs/news/index.html | 3 + docs/pkgdown.yml | 2 +- .../reference/convertJsonResultsFileCase.html | 3 + docs/reference/dot-evaluateThresholds.html | 3 + docs/reference/dot-getCheckId.html | 3 + docs/reference/dot-processCheck.html | 3 + docs/reference/dot-recordResult.html | 3 + docs/reference/dot-runCheck.html | 3 + docs/reference/dot-summarizeResults.html | 3 + docs/reference/dot-writeResultsToCsv.html | 3 + docs/reference/dot-writeResultsToJson.html | 3 + docs/reference/dot-writeResultsToTable.html | 3 + docs/reference/executeDqChecks.html | 3 + docs/reference/index.html | 3 + docs/reference/listDqChecks.html | 3 + docs/reference/reEvaluateThresholds.html | 3 + docs/reference/viewDqDashboard.html | 3 + docs/reference/writeDBResultsToJson.html | 3 + docs/reference/writeJsonResultsToCsv.html | 3 + docs/reference/writeJsonResultsToTable.html | 3 + inst/csv/OMOP_CDMv5.2_Check_Descriptions.csv | 34 +++--- inst/csv/OMOP_CDMv5.3_Check_Descriptions.csv | 34 +++--- inst/csv/OMOP_CDMv5.4_Check_Descriptions.csv | 34 +++--- vignettes/checkIndex.Rmd | 2 +- vignettes/checks/isStandardValidConcept.Rmd | 9 +- .../checks/measurePersonCompleteness.Rmd | 17 +-- vignettes/checks/measureValueCompleteness.Rmd | 8 +- .../sourceConceptRecordCompleteness.Rmd | 10 +- vignettes/checks/sourceValueCompleteness.Rmd | 4 +- .../standardConceptRecordCompleteness.Rmd | 4 +- vignettes/checks/withinVisitDates.Rmd | 39 +++++-- 69 files changed, 464 insertions(+), 193 deletions(-) diff --git a/_pkgdown.yml b/_pkgdown.yml index acbe616b..822c5d5f 100644 --- a/_pkgdown.yml +++ b/_pkgdown.yml @@ -90,6 +90,8 @@ navbar: href: articles/checks/plausibleValueHigh.html - text: plausibleValueLow href: articles/checks/plausibleValueLow.html + - text: withinVisitDates + href: articles/checks/withinVisitDates.html hades: text: hadesLogo href: https://ohdsi.github.io/Hades diff --git a/docs/404.html b/docs/404.html index 75158e0f..7c89b76d 100644 --- a/docs/404.html +++ b/docs/404.html @@ -144,6 +144,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • diff --git a/docs/LICENSE-text.html b/docs/LICENSE-text.html index d734fc77..aaecd811 100644 --- a/docs/LICENSE-text.html +++ b/docs/LICENSE-text.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/articles/AddNewCheck.html b/docs/articles/AddNewCheck.html index 11920ea1..bc027dea 100644 --- a/docs/articles/AddNewCheck.html +++ b/docs/articles/AddNewCheck.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -177,7 +180,7 @@

    Add a New Data Quality Check

    Don Torok

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/AddNewCheck.rmd diff --git a/docs/articles/CheckStatusDefinitions.html b/docs/articles/CheckStatusDefinitions.html index 34423e5e..919cfebc 100644 --- a/docs/articles/CheckStatusDefinitions.html +++ b/docs/articles/CheckStatusDefinitions.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    Check Status Definitions

    Dmitry Ilyn, Maxim Moinat

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/CheckStatusDefinitions.rmd diff --git a/docs/articles/CheckTypeDescriptions.html b/docs/articles/CheckTypeDescriptions.html index ced9cdd5..6e89315a 100644 --- a/docs/articles/CheckTypeDescriptions.html +++ b/docs/articles/CheckTypeDescriptions.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    Data Quality Check Type Definitions

    Clair Blacketer

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/CheckTypeDescriptions.rmd diff --git a/docs/articles/DataQualityDashboard.html b/docs/articles/DataQualityDashboard.html index d370a9b3..f1cf33d8 100644 --- a/docs/articles/DataQualityDashboard.html +++ b/docs/articles/DataQualityDashboard.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    Getting Started

    Clair Blacketer

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/DataQualityDashboard.rmd diff --git a/docs/articles/DqdForCohorts.html b/docs/articles/DqdForCohorts.html index cf5cd2fb..8db9ac2c 100644 --- a/docs/articles/DqdForCohorts.html +++ b/docs/articles/DqdForCohorts.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    Running the DQD on a Cohort

    Clair Blacketer

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/DqdForCohorts.rmd diff --git a/docs/articles/SqlOnly.html b/docs/articles/SqlOnly.html index 68151ee8..1c4a7d4c 100644 --- a/docs/articles/SqlOnly.html +++ b/docs/articles/SqlOnly.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    Running the DQD in SqlOnly mode

    Maxim Moinat

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/SqlOnly.rmd diff --git a/docs/articles/Thresholds.html b/docs/articles/Thresholds.html index d35f64c1..1d94bcf7 100644 --- a/docs/articles/Thresholds.html +++ b/docs/articles/Thresholds.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    Failure Thresholds and How to Change Them

    Clair Blacketer

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/Thresholds.rmd diff --git a/docs/articles/checkIndex.html b/docs/articles/checkIndex.html index aa0590ac..289565c4 100644 --- a/docs/articles/checkIndex.html +++ b/docs/articles/checkIndex.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    Index

    Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checkIndex.Rmd @@ -241,7 +244,7 @@

    Checks
  • sourceValueCompleteness
  • plausibleValueLow
  • plausibleValueHigh
    • -
  • withinVisitDates (PAGE UNDER CONSTRUCTION)
    • +
  • withinVisitDates
  • plausibleAfterBirth
  • plausibleBeforeDeath
  • plausibleStartBeforeEnd
    • diff --git a/docs/articles/checks/cdmDatatype.html b/docs/articles/checks/cdmDatatype.html index fb89677f..c0b9d6c7 100644 --- a/docs/articles/checks/cdmDatatype.html +++ b/docs/articles/checks/cdmDatatype.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +

  • @@ -178,7 +181,7 @@

    cdmDatatype

    Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/cdmDatatype.Rmd diff --git a/docs/articles/checks/cdmField.html b/docs/articles/checks/cdmField.html index 573aa1d3..99406a82 100644 --- a/docs/articles/checks/cdmField.html +++ b/docs/articles/checks/cdmField.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    cdmField

    Heidi Schmidt, Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/cdmField.Rmd diff --git a/docs/articles/checks/cdmTable.html b/docs/articles/checks/cdmTable.html index 1e3b523b..3150fd31 100644 --- a/docs/articles/checks/cdmTable.html +++ b/docs/articles/checks/cdmTable.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    cdmTable

    John Gresh, Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/cdmTable.Rmd diff --git a/docs/articles/checks/fkClass.html b/docs/articles/checks/fkClass.html index b133a025..2904a2e3 100644 --- a/docs/articles/checks/fkClass.html +++ b/docs/articles/checks/fkClass.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    fkClass

    Clair Blacketer, Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/fkClass.Rmd diff --git a/docs/articles/checks/fkDomain.html b/docs/articles/checks/fkDomain.html index 07bae9eb..d90b3fb0 100644 --- a/docs/articles/checks/fkDomain.html +++ b/docs/articles/checks/fkDomain.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    fkDomain

    Clair Blacketer, Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/fkDomain.Rmd diff --git a/docs/articles/checks/isForeignKey.html b/docs/articles/checks/isForeignKey.html index 60484a17..3983b9fa 100644 --- a/docs/articles/checks/isForeignKey.html +++ b/docs/articles/checks/isForeignKey.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    isForeignKey

    Dmytry Dymshyts, Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/isForeignKey.Rmd diff --git a/docs/articles/checks/isPrimaryKey.html b/docs/articles/checks/isPrimaryKey.html index 726e0037..8fdef03a 100644 --- a/docs/articles/checks/isPrimaryKey.html +++ b/docs/articles/checks/isPrimaryKey.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    isPrimaryKey

    John Gresh, Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/isPrimaryKey.Rmd diff --git a/docs/articles/checks/isRequired.html b/docs/articles/checks/isRequired.html index 1e2f089e..994448e6 100644 --- a/docs/articles/checks/isRequired.html +++ b/docs/articles/checks/isRequired.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    isRequired

    Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/isRequired.Rmd diff --git a/docs/articles/checks/isStandardValidConcept.html b/docs/articles/checks/isStandardValidConcept.html index 035c1c0f..6cbd6eca 100644 --- a/docs/articles/checks/isStandardValidConcept.html +++ b/docs/articles/checks/isStandardValidConcept.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -175,10 +178,10 @@

    isStandardValidConcept

    -

    Katy -Sadowski

    +

    Stephanie Hong, +Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/isStandardValidConcept.Rmd @@ -205,18 +208,14 @@

    Definitionhttps://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Mapping. -In other words, all non-zero values in the X_concept_id -column should exist in the concept_id column of the CONCEPT table and -have CONCEPT.standard_concept = ‘S’ and CONCEPT.invalid_reason = -NULL.

    • +columns should contain a standard, valid concept, or 0: https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Mapping.
  • CDM Fields/Tables: All tables with an X_concept_id column, and all X_concept_id @@ -256,13 +255,14 @@

    ETL DevelopersX_concept_id and X_source_concept_id columns to the concept table and inspecting their names and vocabularies. If the X_source_concept_id correctly represents the source code in -x_source_value, the fix will be a matter of ensuring your ETL is -correctly using the concept_relationship table to map the source concept -ID to a standard concept via the ‘Maps to’ relationship. If you are not -populating the X_source_concept_id column and/or are using -an intermediate concept mapping table, you may need to inspect the -mappings in your mapper table to ensure they’ve been generated correctly -using the ‘Maps to’ relationship for your CDM’s vocabulary version.

    +X_source_value, the fix will be a matter of ensuring your +ETL is correctly using the concept_relationship table to map the source +concept ID to a standard concept via the ‘Maps to’ relationship. If you +are not populating the X_source_concept_id column and/or +are using an intermediate concept mapping table, you may need to inspect +the mappings in your mapper table to ensure they’ve been generated +correctly using the ‘Maps to’ relationship for your CDM’s vocabulary +version.

    Data Users diff --git a/docs/articles/checks/measureConditionEraCompleteness.html b/docs/articles/checks/measureConditionEraCompleteness.html index 618b8b80..e8ce2a9f 100644 --- a/docs/articles/checks/measureConditionEraCompleteness.html +++ b/docs/articles/checks/measureConditionEraCompleteness.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +

  • @@ -177,7 +180,7 @@

    measureConditionEraCompleteness

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/measureConditionEraCompleteness.Rmd diff --git a/docs/articles/checks/measurePersonCompleteness.html b/docs/articles/checks/measurePersonCompleteness.html index 0cd476a6..4e61a71d 100644 --- a/docs/articles/checks/measurePersonCompleteness.html +++ b/docs/articles/checks/measurePersonCompleteness.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    measurePersonCompleteness

    Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/measurePersonCompleteness.Rmd @@ -208,17 +211,17 @@

    DefinitionUser GuidanceAll other tables -- Incorrect parsing of source data, resulting in loss of valid records - -Failure of an ETL step, resulting in an empty table

    +in persons missing records in a given event table:

    +
      +
    • Mis-mapping of domains, resulting in the placement of records in the +incorrect table
      +
      • +
    • Incorrect parsing of source data, resulting in loss of valid +records
      • +
    • Failure of an ETL step, resulting in an empty table
      • +

    If more persons than expected are missing data in a given table, run the violated rows SQL snippet to retrieve these persons’ person_ids, and inspect these persons’ other clinical event data in the CDM for trends. -You may also use person_source_value to trace back to these persons’ -source data to identify source data records potentially missed by the -ETL.

    +You may also use person_source_value to trace back to these +persons’ source data to identify source data records potentially missed +by the ETL.

    Note that in some cases, the failure threshold for this check may need to be adjusted according to completeness expectations for a given data source.

    @@ -289,8 +296,8 @@

    Data UsersplausibleValueLow

    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    measureValueCompleteness

    Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/measureValueCompleteness.Rmd @@ -237,14 +240,10 @@

    Violated rows query
    SELECT  
-
-'@cdmTableName.@cdmFieldName' AS violating_field,  
-
-cdmTable.*  
-
-FROM @cdmDatabaseSchema.@cdmTableName cdmTable       
-
-WHERE cdmTable.@cdmFieldName IS NULL
    + '@cdmTableName.@cdmFieldName' AS violating_field, + cdmTable.* +FROM @cdmDatabaseSchema.@cdmTableName cdmTable +WHERE cdmTable.@cdmFieldName IS NULL

    ETL Developers diff --git a/docs/articles/checks/plausibleAfterBirth.html b/docs/articles/checks/plausibleAfterBirth.html index 7c68c5fb..610bc1e4 100644 --- a/docs/articles/checks/plausibleAfterBirth.html +++ b/docs/articles/checks/plausibleAfterBirth.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +

  • @@ -178,7 +181,7 @@

    plausibleAfterBirth

    Maxim Moinat, Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleAfterBirth.Rmd diff --git a/docs/articles/checks/plausibleBeforeDeath.html b/docs/articles/checks/plausibleBeforeDeath.html index 34529a4e..2e91b839 100644 --- a/docs/articles/checks/plausibleBeforeDeath.html +++ b/docs/articles/checks/plausibleBeforeDeath.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    plausibleBeforeDeath

    Maxim Moinat

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleBeforeDeath.Rmd diff --git a/docs/articles/checks/plausibleGenderUseDescendants.html b/docs/articles/checks/plausibleGenderUseDescendants.html index c2765e30..c4e8155a 100644 --- a/docs/articles/checks/plausibleGenderUseDescendants.html +++ b/docs/articles/checks/plausibleGenderUseDescendants.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -177,7 +180,7 @@

    plausibleGender

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleGenderUseDescendants.Rmd diff --git a/docs/articles/checks/plausibleStartBeforeEnd.html b/docs/articles/checks/plausibleStartBeforeEnd.html index cc874c5e..f80c0a9b 100644 --- a/docs/articles/checks/plausibleStartBeforeEnd.html +++ b/docs/articles/checks/plausibleStartBeforeEnd.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    plausibleStartBeforeEnd

    Maxim Moinat

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleStartBeforeEnd.Rmd diff --git a/docs/articles/checks/plausibleTemporalAfter.html b/docs/articles/checks/plausibleTemporalAfter.html index 00b0b159..3a01db40 100644 --- a/docs/articles/checks/plausibleTemporalAfter.html +++ b/docs/articles/checks/plausibleTemporalAfter.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -177,7 +180,7 @@

    plausibleTemporalAfter

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleTemporalAfter.Rmd diff --git a/docs/articles/checks/plausibleUnitConceptIds.html b/docs/articles/checks/plausibleUnitConceptIds.html index a6f1d972..6855948c 100644 --- a/docs/articles/checks/plausibleUnitConceptIds.html +++ b/docs/articles/checks/plausibleUnitConceptIds.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -177,7 +180,7 @@

    plausibleUnitConceptIds

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleUnitConceptIds.Rmd diff --git a/docs/articles/checks/plausibleValueHigh.html b/docs/articles/checks/plausibleValueHigh.html index b937dbdf..799422ff 100644 --- a/docs/articles/checks/plausibleValueHigh.html +++ b/docs/articles/checks/plausibleValueHigh.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    plausibleValueHigh

    Dymytry Dymshyts

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleValueHigh.Rmd diff --git a/docs/articles/checks/plausibleValueLow.html b/docs/articles/checks/plausibleValueLow.html index 303fb846..9cdbe4b0 100644 --- a/docs/articles/checks/plausibleValueLow.html +++ b/docs/articles/checks/plausibleValueLow.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    plausibleValueLow

    Dymytry Dymshyts

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/plausibleValueLow.Rmd diff --git a/docs/articles/checks/sourceConceptRecordCompleteness.html b/docs/articles/checks/sourceConceptRecordCompleteness.html index 54445850..b5f59d0b 100644 --- a/docs/articles/checks/sourceConceptRecordCompleteness.html +++ b/docs/articles/checks/sourceConceptRecordCompleteness.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • @@ -178,7 +181,7 @@

    sourceConceptRecordCompleteness

    Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/sourceConceptRecordCompleteness.Rmd @@ -208,12 +211,12 @@

    DefinitionSource concept mapping @@ -222,9 +225,13 @@

    Definition @@ -277,9 +284,8 @@

    ETL DevelopersIf source values do NOT return matches on concept_code and you are NOT handling concept mapping locally for a non-OMOP source vocabulary, then you likely have a malformed source code or one that does not exist -in the OMOP vocabulary. Please see the documentation in the -standardConceptRecordCompleteness page for instructions on how to handle -this scenario.

    +in the OMOP vocabulary. Please see the documentation in the     standardConceptRecordCompleteness +page for instructions on how to handle this scenario.

    Data Users diff --git a/docs/articles/checks/sourceValueCompleteness.html b/docs/articles/checks/sourceValueCompleteness.html index 8baafec8..82e2d3af 100644 --- a/docs/articles/checks/sourceValueCompleteness.html +++ b/docs/articles/checks/sourceValueCompleteness.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +

  • @@ -178,7 +181,7 @@

    sourceValueCompleteness

    Jared Houghtaling, Clair Blacketer

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/sourceValueCompleteness.Rmd @@ -203,7 +206,8 @@

    DefinitionUser GuidancestandardConceptRecordCompleteness check +check should be used in conjunction with the standardConceptRecordCompleteness check to identify potential mapping issues in the ETL.

    This check is a good measure of the overall mapping rate within each domain. For example, a table may have high diff --git a/docs/articles/checks/standardConceptRecordCompleteness.html b/docs/articles/checks/standardConceptRecordCompleteness.html index 2981148a..d99d0ca9 100644 --- a/docs/articles/checks/standardConceptRecordCompleteness.html +++ b/docs/articles/checks/standardConceptRecordCompleteness.html @@ -145,6 +145,9 @@

  • plausibleValueLow
    • +
  • + withinVisitDates +

  • @@ -178,7 +181,7 @@

    standardConceptRecordCompleteness

    Katy Sadowski

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/standardConceptRecordCompleteness.Rmd @@ -204,17 +207,16 @@

    DefinitionStandard concept mapping
    diff --git a/docs/articles/checks/withinVisitDates.html b/docs/articles/checks/withinVisitDates.html index 9a19d0de..9ff9e5ba 100644 --- a/docs/articles/checks/withinVisitDates.html +++ b/docs/articles/checks/withinVisitDates.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +

  • @@ -175,9 +178,10 @@

    withinVisitDates

    -

    +

    Clair +Blacketer

    -

    2024-06-26

    +

    2024-06-29

    Source: vignettes/checks/withinVisitDates.Rmd @@ -189,45 +193,126 @@

    2024-06-26

    Summary

    -

    Level: FIELD
    Context: Verification
    Category: Conformance
    Subcategory:
    Severity:

    +

    Level: FIELD
    Context: Verification
    Category: Conformance
    Subcategory:
    Severity: Characterization ✔

    Description

    -

    The number and percent of records not within one week on either side -of the corresponding visit occurrence start and end date

    +

    The number and percent of records that occur one week before the +corresponding visit_start_date or one week after the +corresponding visit_end_date

    Definition

    • -Numerator:
      • +Numerator: The number of rows in a table where the event +date occurs more than 7 days prior to the corresponding visit start date +or more than 7 days after the corresponding visit end date.
    • -Denominator:
      • +Denominator: The total number of rows in the table with a +corresponding visit (linked through +visit_occurrence_id)
    • -Related CDM Convention(s):
      • +Related CDM Convention(s): There is no explicit convention +tied to this check. However, the CDM documentation describes the +visit_occurrence_id foreign key in the event tables as “The +visit during which the occurred.” The underlying assumption is +that if a record is tied to a visit, then the date of the record should +fall in some reasonable time period around the visit dates. This gives a +week of leeway on either side for physician notes or other activities +related to a visit to be recorded.
       +
    • -CDM Fields/Tables:
      • +CDM Fields/Tables: This check runs on all event tables: +CONDITION_OCCURRENCE, PROCEDURE_OCCURRENCE, +DRUG_EXPOSURE, DEVICE_EXPOSURE, +MEASUREMENT, NOTE, OBSERVATION, +and VISIT_DETAIL. It will check either the +X_date or X_start_date fields for alignment +with corresponding VISIT_OCCURRENCE dates by linking on the +visit_occurrence_id. (Note: For +VISIT_DETAIL it will check both the visit_detail_start_date and +visit_detail_end_date. The default threshold for these two checks is +1%.)
    • -Default Threshold Value:
      • +Default Threshold Value: 1% for VISIT_DETAIL, +5% for all other tables

    User Guidance

    +

    There is no explicit convention that describes how events should +align temporally with the visits they correspond to. This check is meant +to identify egregious mismatches in dates that could signify an +incorrect date field was used in the ETL or that the data should be used +with caution if there is no reason for the mismatch (history of a +condition, for example).

    +

    If this check fails the first action should be to investigate the +failing rows for any patterns. The main query to find failing rows is +below:

    Violated rows query

    -
    +
    SELECT 
+  '@cdmTableName.@cdmFieldName' AS violating_field,  
+  vo.visit_start_date, vo.visit_end_date, vo.person_id, 
+  cdmTable.* 
+FROM @cdmDatabaseSchema.@cdmTableName cdmTable 
+JOIN @cdmDatabaseSchema.visit_occurrence vo 
+  ON cdmTable.visit_occurrence_id = vo.visit_occurrence_id 
+WHERE cdmTable.@cdmFieldName < dateadd(day, -7, vo.visit_start_date) 
+  OR cdmTable.@cdmFieldName > dateadd(day, 7, vo.visit_end_date)

    ETL Developers

    +

    The first step is to investigate whether visit and event indeed +should be linked - e.g., do they belong to the same person; how far are +the dates apart; is it possible the event was recorded during the visit. +If they should be linked, then the next step is to investigate which of +the event date and visit date is accurate.

    +

    One suggestion would be to identify if all of the failures are due to +many events all having the same date. In some institutions there is a +default date given to events in the case where a date is not given. +Should this be the problem, the first step should be to identify if +there is a different date field in the native data that can be used. If +not, considerations should be made to determine if the rows should be +dropped. Without a correct date it is challenging to use such events in +health outcomes research.

    +

    Another reason for the discrepancy could be that the wrong date has +been used for events. For instance, in some systems a diagnosis could +have both an ‘observation date’ and an ‘administration date’. If the +physician is updating records at a later date, the administration date +can be later than the actual date the diagnosis was observed. In those +cases, the observation date has to be used. If there is only an +administration date, it is in some cases arguable to use the visit date +for the diagnosis date.

    +

    Another suggestion would be to investigate if the failures are +related to ‘History of’ conditions. It is often the case that a +patient’s history is recorded during a visit, in which case they may +report a diagnosis date prior to the given visit. In some cases it may +be appropriate to conserve these records; the decision to do so will +depend on the reliability of the recorded date in your source data.

    Data Users

    +

    If the failure percentage of withinVisitDates is high, a data user +should be careful with using the data. This check might indicate a +larger underlying conformance issue with either the event dates or +linkage with visits. At the same time, there might be a valid reason why +events do not happen within 7 days of the linked visit.

    +

    Make sure to understand why this check fails. Specifically, be +careful when using such data in outcomes research. Without specific +dates for an event, it is challenging to determine if an adverse event +occurred after a drug exposure, for example.

    +

    Note that this check specifically compares event dates to +VISIT_OCCURRENCE dates. There is no equivalent check for +VISIT_DETAIL that verifies whether the event date is within +(a week of) the visit detail start and end dates.

    diff --git a/docs/articles/index.html b/docs/articles/index.html index c01a0cff..a9c50ea9 100644 --- a/docs/articles/index.html +++ b/docs/articles/index.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/authors.html b/docs/authors.html index 796cd4c8..89196518 100644 --- a/docs/authors.html +++ b/docs/authors.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/index.html b/docs/index.html index b742927b..f76a3b99 100644 --- a/docs/index.html +++ b/docs/index.html @@ -145,6 +145,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • diff --git a/docs/news/index.html b/docs/news/index.html index 3d1f6026..83e23076 100644 --- a/docs/news/index.html +++ b/docs/news/index.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/pkgdown.yml b/docs/pkgdown.yml index 4d9ad23c..a3657ed9 100644 --- a/docs/pkgdown.yml +++ b/docs/pkgdown.yml @@ -34,7 +34,7 @@ articles: standardConceptRecordCompleteness: checks/standardConceptRecordCompleteness.html Thresholds: Thresholds.html withinVisitDates: checks/withinVisitDates.html -last_built: 2024-06-26T15:29Z +last_built: 2024-06-29T20:33Z urls: reference: https://ohdsi.github.io/DataQualityDashboard/reference article: https://ohdsi.github.io/DataQualityDashboard/articles diff --git a/docs/reference/convertJsonResultsFileCase.html b/docs/reference/convertJsonResultsFileCase.html index 47819d16..0b2871f8 100644 --- a/docs/reference/convertJsonResultsFileCase.html +++ b/docs/reference/convertJsonResultsFileCase.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-evaluateThresholds.html b/docs/reference/dot-evaluateThresholds.html index ba89215c..0e3a4c2c 100644 --- a/docs/reference/dot-evaluateThresholds.html +++ b/docs/reference/dot-evaluateThresholds.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-getCheckId.html b/docs/reference/dot-getCheckId.html index 8f26d7fb..b631cb51 100644 --- a/docs/reference/dot-getCheckId.html +++ b/docs/reference/dot-getCheckId.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-processCheck.html b/docs/reference/dot-processCheck.html index 0431d9b5..7f29c99a 100644 --- a/docs/reference/dot-processCheck.html +++ b/docs/reference/dot-processCheck.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-recordResult.html b/docs/reference/dot-recordResult.html index 0b3a1151..96673dd9 100644 --- a/docs/reference/dot-recordResult.html +++ b/docs/reference/dot-recordResult.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-runCheck.html b/docs/reference/dot-runCheck.html index 9407dceb..9b4c5b79 100644 --- a/docs/reference/dot-runCheck.html +++ b/docs/reference/dot-runCheck.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-summarizeResults.html b/docs/reference/dot-summarizeResults.html index 52d47039..8d22ee63 100644 --- a/docs/reference/dot-summarizeResults.html +++ b/docs/reference/dot-summarizeResults.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-writeResultsToCsv.html b/docs/reference/dot-writeResultsToCsv.html index 95bf0b43..1a209ffe 100644 --- a/docs/reference/dot-writeResultsToCsv.html +++ b/docs/reference/dot-writeResultsToCsv.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-writeResultsToJson.html b/docs/reference/dot-writeResultsToJson.html index fdfeaba6..62410ef1 100644 --- a/docs/reference/dot-writeResultsToJson.html +++ b/docs/reference/dot-writeResultsToJson.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/dot-writeResultsToTable.html b/docs/reference/dot-writeResultsToTable.html index 5004b036..2e215376 100644 --- a/docs/reference/dot-writeResultsToTable.html +++ b/docs/reference/dot-writeResultsToTable.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/executeDqChecks.html b/docs/reference/executeDqChecks.html index 4e42ef4a..6382ed42 100644 --- a/docs/reference/executeDqChecks.html +++ b/docs/reference/executeDqChecks.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/index.html b/docs/reference/index.html index be3c8cf5..f9d83e64 100644 --- a/docs/reference/index.html +++ b/docs/reference/index.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/listDqChecks.html b/docs/reference/listDqChecks.html index a3500e52..606e2d87 100644 --- a/docs/reference/listDqChecks.html +++ b/docs/reference/listDqChecks.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/reEvaluateThresholds.html b/docs/reference/reEvaluateThresholds.html index e9305b59..dd74b726 100644 --- a/docs/reference/reEvaluateThresholds.html +++ b/docs/reference/reEvaluateThresholds.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/viewDqDashboard.html b/docs/reference/viewDqDashboard.html index 36efe47f..ece1e45c 100644 --- a/docs/reference/viewDqDashboard.html +++ b/docs/reference/viewDqDashboard.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/writeDBResultsToJson.html b/docs/reference/writeDBResultsToJson.html index 090d5c45..283d3e2a 100644 --- a/docs/reference/writeDBResultsToJson.html +++ b/docs/reference/writeDBResultsToJson.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/writeJsonResultsToCsv.html b/docs/reference/writeJsonResultsToCsv.html index 8ec3afb0..12a8f937 100644 --- a/docs/reference/writeJsonResultsToCsv.html +++ b/docs/reference/writeJsonResultsToCsv.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/docs/reference/writeJsonResultsToTable.html b/docs/reference/writeJsonResultsToTable.html index 30c65c4a..e944a846 100644 --- a/docs/reference/writeJsonResultsToTable.html +++ b/docs/reference/writeJsonResultsToTable.html @@ -125,6 +125,9 @@
  • plausibleValueLow
    • +
  • + withinVisitDates +
  • Changelog diff --git a/inst/csv/OMOP_CDMv5.2_Check_Descriptions.csv b/inst/csv/OMOP_CDMv5.2_Check_Descriptions.csv index 7932187a..63a4c7df 100644 --- a/inst/csv/OMOP_CDMv5.2_Check_Descriptions.csv +++ b/inst/csv/OMOP_CDMv5.2_Check_Descriptions.csv @@ -1,8 +1,8 @@ checkLevel,checkName,checkDescription,kahnContext,kahnCategory,kahnSubcategory,sqlFile,evaluationFilter,severity TABLE,cdmTable,A yes or no value indicating if @cdmTableName table is present as expected based on the specification. ,Verification,Conformance,Relational,table_cdm_table.sql,cdmTableName!='',fatal -TABLE,measurePersonCompleteness,The number and percent of persons in the CDM that do not have at least one record in the @cdmTableName table,Validation,Completeness,,table_person_completeness.sql,measurePersonCompleteness=='Yes', +TABLE,measurePersonCompleteness,The number and percent of persons in the CDM that do not have at least one record in the @cdmTableName table,Validation,Completeness,,table_person_completeness.sql,measurePersonCompleteness=='Yes',convention TABLE,measureConditionEraCompleteness,"The number and Percent of persons that does not have condition_era built successfully -for all persons in condition_occurrence",Validation,Completeness,,table_condition_era_completeness.sql,measureConditionEraCompleteness=='Yes', +for all persons in condition_occurrence",Validation,Completeness,,table_condition_era_completeness.sql,measureConditionEraCompleteness=='Yes',convention FIELD,cdmField,A yes or no value indicating if @cdmFieldName is present in the @cdmTableName table as expected based on the specification. ,Verification,Conformance,Relational,field_cdm_field.sql,cdmFieldName!='',fatal FIELD,isRequired,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName that is considered not nullable.,Validation,Conformance,Relational,field_is_not_nullable.sql,isRequired=='Yes',fatal FIELD,cdmDatatype,A yes or no value indicating if the @cdmFieldName in the @cdmTableName is the expected data type based on the specification. Only checks integer fields.,Verification,Conformance,Value,field_cdm_datatype.sql,cdmDatatype=='integer',fatal @@ -10,19 +10,19 @@ FIELD,isPrimaryKey,The number and percent of records that have a duplicate value FIELD,isForeignKey,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that does not exist in the @fkTableName table.,Verification,Conformance,Relational,is_foreign_key.sql,isForeignKey=='Yes',fatal FIELD,fkDomain,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that do not conform to the @fkDomain domain.,Verification,Conformance,Value,field_fk_domain.sql,isForeignKey=='Yes' & fkDomain!= '',convention FIELD,fkClass,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that do not conform to the @fkClass class.,Verification,Conformance,Computational,field_fk_class.sql,isForeignKey=='Yes' & fkClass!='',convention -FIELD,isStandardValidConcept,"The number and percent of records that do not have a standard, valid concept in the @cdmFieldName field in the @cdmTableName table. ",Verification,Conformance,Value,field_is_standard_valid_concept.sql,isStandardValidConcept=='Yes', -FIELD,measureValueCompleteness,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName.,Verification,Completeness,,field_measure_value_completeness.sql,measureValueCompleteness=='Yes', -FIELD,standardConceptRecordCompleteness,The number and percent of records with a value of 0 in the standard concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,standardConceptRecordCompleteness=='Yes', -FIELD,sourceConceptRecordCompleteness,The number and percent of records with a value of 0 in the source concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,sourceConceptRecordCompleteness=='Yes', -FIELD,sourceValueCompleteness,The number and percent of distinct source values in the @cdmFieldName field of the @cdmTableName table mapped to 0.,Verification,Completeness,,field_source_value_completeness.sql,sourceValueCompleteness=='Yes', -FIELD,plausibleValueLow,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table less than @plausibleValueLow.,Verification,Plausibility,Atemporal,field_plausible_value_low.sql,plausibleValueLow!='', -FIELD,plausibleValueHigh,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table greater than @plausibleValueHigh.,Verification,Plausibility,Atemporal,field_plausible_value_high.sql,plausibleValueHigh!='', -FIELD,plausibleTemporalAfter,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs prior to the date in the @plausibleTemporalAfterFieldName field of the @plausibleTemporalAfterTableName table.,Verification,Plausibility,Temporal,field_plausible_temporal_after.sql,plausibleTemporalAfter=='Yes', -FIELD,plausibleDuringLife,"If yes, the number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_during_life.sql,plausibleDuringLife=='Yes', -FIELD,withinVisitDates,The number and percent of records not within one week on either side of the corresponding visit occurrence start and end date,Verification,Conformance,,field_within_visit_dates.sql,withinVisitDates=='Yes', +FIELD,isStandardValidConcept,"The number and percent of records that do not have a standard, valid concept in the @cdmFieldName field in the @cdmTableName table. ",Verification,Conformance,Value,field_is_standard_valid_concept.sql,isStandardValidConcept=='Yes',convention +FIELD,measureValueCompleteness,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName.,Verification,Completeness,,field_measure_value_completeness.sql,measureValueCompleteness=='Yes',characterization +FIELD,standardConceptRecordCompleteness,The number and percent of records with a value of 0 in the standard concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,standardConceptRecordCompleteness=='Yes',convention +FIELD,sourceConceptRecordCompleteness,The number and percent of records with a value of 0 in the source concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,sourceConceptRecordCompleteness=='Yes',convention +FIELD,sourceValueCompleteness,The number and percent of distinct source values in the @cdmFieldName field of the @cdmTableName table mapped to 0.,Verification,Completeness,,field_source_value_completeness.sql,sourceValueCompleteness=='Yes',convention +FIELD,plausibleValueLow,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table less than @plausibleValueLow.,Verification,Plausibility,Atemporal,field_plausible_value_low.sql,plausibleValueLow!='',characterization +FIELD,plausibleValueHigh,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table greater than @plausibleValueHigh.,Verification,Plausibility,Atemporal,field_plausible_value_high.sql,plausibleValueHigh!='',characterization +FIELD,plausibleTemporalAfter,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs prior to the date in the @plausibleTemporalAfterFieldName field of the @plausibleTemporalAfterTableName table.,Verification,Plausibility,Temporal,field_plausible_temporal_after.sql,plausibleTemporalAfter=='Yes',characterization +FIELD,plausibleDuringLife,"If yes, the number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_during_life.sql,plausibleDuringLife=='Yes',characterization +FIELD,withinVisitDates,The number and percent of records not within one week on either side of the corresponding visit occurrence start and end date,Verification,Conformance,,field_within_visit_dates.sql,withinVisitDates=='Yes',characterization FIELD,plausibleAfterBirth,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs prior to birth.",Verification,Plausibility,Temporal,field_plausible_after_birth.sql,plausibleAfterBirth=='Yes',characterization -FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes', -FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes', -CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!='', -CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!='', -CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds)).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!='', \ No newline at end of file +FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes',characterization +FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes',characterization +CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!='',characterization +CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!='',characterization +CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds)).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!='',characterization \ No newline at end of file diff --git a/inst/csv/OMOP_CDMv5.3_Check_Descriptions.csv b/inst/csv/OMOP_CDMv5.3_Check_Descriptions.csv index 5a35f724..d0411ce5 100644 --- a/inst/csv/OMOP_CDMv5.3_Check_Descriptions.csv +++ b/inst/csv/OMOP_CDMv5.3_Check_Descriptions.csv @@ -1,8 +1,8 @@ checkLevel,checkName,checkDescription,kahnContext,kahnCategory,kahnSubcategory,sqlFile,evaluationFilter,severity TABLE,cdmTable,A yes or no value indicating if @cdmTableName table is present as expected based on the specification. ,Verification,Conformance,Relational,table_cdm_table.sql,cdmTableName!='',fatal -TABLE,measurePersonCompleteness,The number and percent of persons in the CDM that do not have at least one record in the @cdmTableName table,Validation,Completeness,,table_person_completeness.sql,measurePersonCompleteness=='Yes', +TABLE,measurePersonCompleteness,The number and percent of persons in the CDM that do not have at least one record in the @cdmTableName table,Validation,Completeness,,table_person_completeness.sql,measurePersonCompleteness=='Yes',convention TABLE,measureConditionEraCompleteness,"The number and Percent of persons that does not have condition_era built successfully -for all persons in condition_occurrence",Validation,Completeness,,table_condition_era_completeness.sql,measureConditionEraCompleteness=='Yes', +for all persons in condition_occurrence",Validation,Completeness,,table_condition_era_completeness.sql,measureConditionEraCompleteness=='Yes',convention FIELD,cdmField,A yes or no value indicating if @cdmFieldName is present in the @cdmTableName table as expected based on the specification. ,Verification,Conformance,Relational,field_cdm_field.sql,cdmFieldName!='',fatal FIELD,isRequired,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName that is considered not nullable.,Validation,Conformance,Relational,field_is_not_nullable.sql,isRequired=='Yes',fatal FIELD,cdmDatatype,A yes or no value indicating if the @cdmFieldName in the @cdmTableName is the expected data type based on the specification. Only checks integer fields.,Verification,Conformance,Value,field_cdm_datatype.sql,cdmDatatype=='integer',fatal @@ -10,19 +10,19 @@ FIELD,isPrimaryKey,The number and percent of records that have a duplicate value FIELD,isForeignKey,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that does not exist in the @fkTableName table.,Verification,Conformance,Relational,is_foreign_key.sql,isForeignKey=='Yes',fatal FIELD,fkDomain,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that do not conform to the @fkDomain domain.,Verification,Conformance,Value,field_fk_domain.sql,isForeignKey=='Yes' & fkDomain!= '',convention FIELD,fkClass,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that do not conform to the @fkClass class.,Verification,Conformance,Computational,field_fk_class.sql,isForeignKey=='Yes' & fkClass!='',convention -FIELD,isStandardValidConcept,"The number and percent of records that do not have a standard, valid concept in the @cdmFieldName field in the @cdmTableName table. ",Verification,Conformance,Value,field_is_standard_valid_concept.sql,isStandardValidConcept=='Yes', -FIELD,measureValueCompleteness,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName.,Verification,Completeness,,field_measure_value_completeness.sql,measureValueCompleteness=='Yes', -FIELD,standardConceptRecordCompleteness,The number and percent of records with a value of 0 in the standard concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,standardConceptRecordCompleteness=='Yes', -FIELD,sourceConceptRecordCompleteness,The number and percent of records with a value of 0 in the source concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,sourceConceptRecordCompleteness=='Yes', -FIELD,sourceValueCompleteness,The number and percent of distinct source values in the @cdmFieldName field of the @cdmTableName table mapped to 0.,Verification,Completeness,,field_source_value_completeness.sql,sourceValueCompleteness=='Yes', -FIELD,plausibleValueLow,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table less than @plausibleValueLow.,Verification,Plausibility,Atemporal,field_plausible_value_low.sql,plausibleValueLow!='', -FIELD,plausibleValueHigh,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table greater than @plausibleValueHigh.,Verification,Plausibility,Atemporal,field_plausible_value_high.sql,plausibleValueHigh!='', -FIELD,plausibleTemporalAfter,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs prior to the date in the @plausibleTemporalAfterFieldName field of the @plausibleTemporalAfterTableName table.,Verification,Plausibility,Temporal,field_plausible_temporal_after.sql,plausibleTemporalAfter=='Yes', -FIELD,plausibleDuringLife,"If yes, the number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_during_life.sql,plausibleDuringLife=='Yes', -FIELD,withinVisitDates,The number and percent of records not within one week on either side of the corresponding visit occurrence start and end date,Verification,Conformance,,field_within_visit_dates.sql,withinVisitDates=='Yes', +FIELD,isStandardValidConcept,"The number and percent of records that do not have a standard, valid concept in the @cdmFieldName field in the @cdmTableName table. ",Verification,Conformance,Value,field_is_standard_valid_concept.sql,isStandardValidConcept=='Yes',convention +FIELD,measureValueCompleteness,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName.,Verification,Completeness,,field_measure_value_completeness.sql,measureValueCompleteness=='Yes',characterization +FIELD,standardConceptRecordCompleteness,The number and percent of records with a value of 0 in the standard concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,standardConceptRecordCompleteness=='Yes',convention +FIELD,sourceConceptRecordCompleteness,The number and percent of records with a value of 0 in the source concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,sourceConceptRecordCompleteness=='Yes',convention +FIELD,sourceValueCompleteness,The number and percent of distinct source values in the @cdmFieldName field of the @cdmTableName table mapped to 0.,Verification,Completeness,,field_source_value_completeness.sql,sourceValueCompleteness=='Yes',convention +FIELD,plausibleValueLow,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table less than @plausibleValueLow.,Verification,Plausibility,Atemporal,field_plausible_value_low.sql,plausibleValueLow!='',characterization +FIELD,plausibleValueHigh,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table greater than @plausibleValueHigh.,Verification,Plausibility,Atemporal,field_plausible_value_high.sql,plausibleValueHigh!='',characterization +FIELD,plausibleTemporalAfter,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs prior to the date in the @plausibleTemporalAfterFieldName field of the @plausibleTemporalAfterTableName table.,Verification,Plausibility,Temporal,field_plausible_temporal_after.sql,plausibleTemporalAfter=='Yes',characterization +FIELD,plausibleDuringLife,"If yes, the number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_during_life.sql,plausibleDuringLife=='Yes',characterization +FIELD,withinVisitDates,The number and percent of records not within one week on either side of the corresponding visit occurrence start and end date,Verification,Conformance,,field_within_visit_dates.sql,withinVisitDates=='Yes',characterization FIELD,plausibleAfterBirth,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs prior to birth.",Verification,Plausibility,Temporal,field_plausible_after_birth.sql,plausibleAfterBirth=='Yes',characterization -FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes', -FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes', -CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!='', -CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!='', -CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds)).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!='', \ No newline at end of file +FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes',characterization +FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes',characterization +CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!='',characterization +CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!='',characterization +CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds)).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!='',characterization \ No newline at end of file diff --git a/inst/csv/OMOP_CDMv5.4_Check_Descriptions.csv b/inst/csv/OMOP_CDMv5.4_Check_Descriptions.csv index 7a8e9227..a9244366 100644 --- a/inst/csv/OMOP_CDMv5.4_Check_Descriptions.csv +++ b/inst/csv/OMOP_CDMv5.4_Check_Descriptions.csv @@ -1,8 +1,8 @@ checkLevel,checkName,checkDescription,kahnContext,kahnCategory,kahnSubcategory,sqlFile,evaluationFilter,severity TABLE,cdmTable,A yes or no value indicating if @cdmTableName table is present as expected based on the specification. ,Verification,Conformance,Relational,table_cdm_table.sql,cdmTableName!='',fatal -TABLE,measurePersonCompleteness,The number and percent of persons in the CDM that do not have at least one record in the @cdmTableName table,Validation,Completeness,,table_person_completeness.sql,measurePersonCompleteness=='Yes', +TABLE,measurePersonCompleteness,The number and percent of persons in the CDM that do not have at least one record in the @cdmTableName table,Validation,Completeness,,table_person_completeness.sql,measurePersonCompleteness=='Yes',convention TABLE,measureConditionEraCompleteness,"The number and Percent of persons that does not have condition_era built successfully -for all persons in condition_occurrence",Validation,Completeness,,table_condition_era_completeness.sql,measureConditionEraCompleteness=='Yes', +for all persons in condition_occurrence",Validation,Completeness,,table_condition_era_completeness.sql,measureConditionEraCompleteness=='Yes',convention FIELD,cdmField,A yes or no value indicating if @cdmFieldName is present in the @cdmTableName table as expected based on the specification. ,Verification,Conformance,Relational,field_cdm_field.sql,cdmFieldName!='',fatal FIELD,isRequired,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName that is considered not nullable.,Validation,Conformance,Relational,field_is_not_nullable.sql,isRequired=='Yes',fatal FIELD,cdmDatatype,A yes or no value indicating if the @cdmFieldName in the @cdmTableName is the expected data type based on the specification. Only checks integer fields.,Verification,Conformance,Value,field_cdm_datatype.sql,cdmDatatype=='integer',fatal @@ -10,19 +10,19 @@ FIELD,isPrimaryKey,The number and percent of records that have a duplicate value FIELD,isForeignKey,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that does not exist in the @fkTableName table.,Verification,Conformance,Relational,is_foreign_key.sql,isForeignKey=='Yes',fatal FIELD,fkDomain,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that do not conform to the @fkDomain domain.,Verification,Conformance,Value,field_fk_domain.sql,isForeignKey=='Yes' & fkDomain!= '',convention FIELD,fkClass,The number and percent of records that have a value in the @cdmFieldName field in the @cdmTableName table that do not conform to the @fkClass class.,Verification,Conformance,Computational,field_fk_class.sql,isForeignKey=='Yes' & fkClass!='',convention -FIELD,isStandardValidConcept,"The number and percent of records that do not have a standard, valid concept in the @cdmFieldName field in the @cdmTableName table. ",Verification,Conformance,Value,field_is_standard_valid_concept.sql,isStandardValidConcept=='Yes', -FIELD,measureValueCompleteness,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName.,Verification,Completeness,,field_measure_value_completeness.sql,measureValueCompleteness=='Yes', -FIELD,standardConceptRecordCompleteness,The number and percent of records with a value of 0 in the standard concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,standardConceptRecordCompleteness=='Yes', -FIELD,sourceConceptRecordCompleteness,The number and percent of records with a value of 0 in the source concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,sourceConceptRecordCompleteness=='Yes', -FIELD,sourceValueCompleteness,The number and percent of distinct source values in the @cdmFieldName field of the @cdmTableName table mapped to 0.,Verification,Completeness,,field_source_value_completeness.sql,sourceValueCompleteness=='Yes', -FIELD,plausibleValueLow,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table less than @plausibleValueLow.,Verification,Plausibility,Atemporal,field_plausible_value_low.sql,plausibleValueLow!='', -FIELD,plausibleValueHigh,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table greater than @plausibleValueHigh.,Verification,Plausibility,Atemporal,field_plausible_value_high.sql,plausibleValueHigh!='', -FIELD,plausibleTemporalAfter,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs prior to the date in the @plausibleTemporalAfterFieldName field of the @plausibleTemporalAfterTableName table.,Verification,Plausibility,Temporal,field_plausible_temporal_after.sql,plausibleTemporalAfter=='Yes', -FIELD,plausibleDuringLife,"If yes, the number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_during_life.sql,plausibleDuringLife=='Yes', -FIELD,withinVisitDates,The number and percent of records not within one week on either side of the corresponding visit occurrence start and end date,Verification,Conformance,,field_within_visit_dates.sql,withinVisitDates=='Yes', +FIELD,isStandardValidConcept,"The number and percent of records that do not have a standard, valid concept in the @cdmFieldName field in the @cdmTableName table. ",Verification,Conformance,Value,field_is_standard_valid_concept.sql,isStandardValidConcept=='Yes',convention +FIELD,measureValueCompleteness,The number and percent of records with a NULL value in the @cdmFieldName of the @cdmTableName.,Verification,Completeness,,field_measure_value_completeness.sql,measureValueCompleteness=='Yes',characterization +FIELD,standardConceptRecordCompleteness,The number and percent of records with a value of 0 in the standard concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,standardConceptRecordCompleteness=='Yes',convention +FIELD,sourceConceptRecordCompleteness,The number and percent of records with a value of 0 in the source concept field @cdmFieldName in the @cdmTableName table.,Verification,Completeness,,field_concept_record_completeness.sql,sourceConceptRecordCompleteness=='Yes',convention +FIELD,sourceValueCompleteness,The number and percent of distinct source values in the @cdmFieldName field of the @cdmTableName table mapped to 0.,Verification,Completeness,,field_source_value_completeness.sql,sourceValueCompleteness=='Yes',convention +FIELD,plausibleValueLow,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table less than @plausibleValueLow.,Verification,Plausibility,Atemporal,field_plausible_value_low.sql,plausibleValueLow!='',characterization +FIELD,plausibleValueHigh,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName table greater than @plausibleValueHigh.,Verification,Plausibility,Atemporal,field_plausible_value_high.sql,plausibleValueHigh!='',characterization +FIELD,plausibleTemporalAfter,The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs prior to the date in the @plausibleTemporalAfterFieldName field of the @plausibleTemporalAfterTableName table.,Verification,Plausibility,Temporal,field_plausible_temporal_after.sql,plausibleTemporalAfter=='Yes',characterization +FIELD,plausibleDuringLife,"If yes, the number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_during_life.sql,plausibleDuringLife=='Yes',characterization +FIELD,withinVisitDates,The number and percent of records not within one week on either side of the corresponding visit occurrence start and end date,Verification,Conformance,,field_within_visit_dates.sql,withinVisitDates=='Yes',characterization FIELD,plausibleAfterBirth,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs prior to birth.",Verification,Plausibility,Temporal,field_plausible_after_birth.sql,plausibleAfterBirth=='Yes',characterization -FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes', -FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes', -CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!='', -CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!='', -CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds)).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!='', \ No newline at end of file +FIELD,plausibleBeforeDeath,"The number and percent of records with a date value in the @cdmFieldName field of the @cdmTableName table that occurs after death.",Verification,Plausibility,Temporal,field_plausible_before_death.sql,plausibleBeforeDeath=='Yes',characterization +FIELD,plausibleStartBeforeEnd,"The number and percent of records with a value in the @cdmFieldName field of the @cdmTableName that occurs after the date in the @plausibleStartBeforeEndFieldName.",Verification,Plausibility,Temporal,field_plausible_start_before_end.sql,plausibleStartBeforeEnd=='Yes',characterization +CONCEPT,plausibleGender,"For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGender).",Validation,Plausibility,Atemporal,concept_plausible_gender.sql,plausibleGender!='',characterization +CONCEPT,plausibleGenderUseDescendants,"For descendants of CONCEPT_ID @conceptId (@conceptName), the number and percent of records associated with patients with an implausible gender (correct gender = @plausibleGenderUseDescendants).",Validation,Plausibility,Atemporal,concept_plausible_gender_use_descendants.sql,plausibleGenderUseDescendants!='',characterization +CONCEPT,plausibleUnitConceptIds,"The number and percent of records for a given CONCEPT_ID @conceptId (@conceptName) with implausible units (i.e., UNIT_CONCEPT_ID NOT IN (@plausibleUnitConceptIds)).",Verification,Plausibility,Atemporal,concept_plausible_unit_concept_ids.sql,plausibleUnitConceptIdsThreshold!='',characterization \ No newline at end of file diff --git a/vignettes/checkIndex.Rmd b/vignettes/checkIndex.Rmd index 7f5c9ab2..4e93a905 100644 --- a/vignettes/checkIndex.Rmd +++ b/vignettes/checkIndex.Rmd @@ -43,7 +43,7 @@ above to navigate to the check's documentation page.\ - [sourceValueCompleteness](checks/sourceValueCompleteness.html) - [plausibleValueLow](checks/plausibleValueLow.html) - [plausibleValueHigh](checks/plausibleValueHigh.html) -- withinVisitDates (PAGE UNDER CONSTRUCTION) +- [withinVisitDates](checks/withinVisitDates.html) - [plausibleAfterBirth](checks/plausibleAfterBirth.html) - [plausibleBeforeDeath](checks/plausibleBeforeDeath.html) - [plausibleStartBeforeEnd](checks/plausibleStartBeforeEnd.html) diff --git a/vignettes/checks/isStandardValidConcept.Rmd b/vignettes/checks/isStandardValidConcept.Rmd index 0e1f596a..d1001aea 100644 --- a/vignettes/checks/isStandardValidConcept.Rmd +++ b/vignettes/checks/isStandardValidConcept.Rmd @@ -1,6 +1,6 @@ --- title: "isStandardValidConcept" -author: "Katy Sadowski" +author: "Stephanie Hong, Katy Sadowski" date: "`r Sys.Date()`" output: html_document: @@ -23,9 +23,9 @@ The number and percent of records that do not have a standard, valid concept in ## Definition -- *Numerator*: The number of rows with an `X_concept_id` that exists in concept.concept_id but does not equal zero, and has a standard_concept != ‘S’ or a non-NULL invalid_reason. +- *Numerator*: The number of rows with an `X_concept_id` that exists in `CONCEPT.concept_id` but does not equal zero, and has `CONCEPT.standard_concept` != ‘S’ or non-NULL `CONCEPT.invalid_reason`. - *Denominator*: The total number of rows in the table. -- *Related CDM Convention(s)*: All `X_concept_id` columns should contain a standard, valid concept, or 0: https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Mapping. In other words, all non-zero values in the `X_concept_id` column should exist in the concept_id column of the CONCEPT table and have CONCEPT.standard_concept = ‘S’ and CONCEPT.invalid_reason = NULL. +- *Related CDM Convention(s)*: All `X_concept_id` columns should contain a standard, valid concept, or 0: https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Mapping. - *CDM Fields/Tables*: All tables with an `X_concept_id` column, and all `X_concept_id` columns in those tables. - *Default Threshold Value*: 0 @@ -48,7 +48,8 @@ FROM @schema.@cdmTableName cdmTable WHERE co.concept_id != 0 AND (co.standard_concept != 'S' OR co.invalid_reason IS NOT NULL) ``` -You may build upon this query by joining the relevant `X_concept_id` and `X_source_concept_id` columns to the concept table and inspecting their names and vocabularies. If the `X_source_concept_id` correctly represents the source code in x_source_value, the fix will be a matter of ensuring your ETL is correctly using the concept_relationship table to map the source concept ID to a standard concept via the ‘Maps to’ relationship. If you are not populating the `X_source_concept_id` column and/or are using an intermediate concept mapping table, you may need to inspect the mappings in your mapper table to ensure they’ve been generated correctly using the ‘Maps to’ relationship for your CDM’s vocabulary version. + +You may build upon this query by joining the relevant `X_concept_id` and `X_source_concept_id` columns to the concept table and inspecting their names and vocabularies. If the `X_source_concept_id` correctly represents the source code in `X_source_value`, the fix will be a matter of ensuring your ETL is correctly using the concept_relationship table to map the source concept ID to a standard concept via the ‘Maps to’ relationship. If you are not populating the `X_source_concept_id` column and/or are using an intermediate concept mapping table, you may need to inspect the mappings in your mapper table to ensure they’ve been generated correctly using the ‘Maps to’ relationship for your CDM’s vocabulary version. ### Data Users This check failure means that the failing rows will not be picked up in a standard OHDSI analysis. It is highly recommended to work with your ETL team or data provider, if possible, to resolve this issue. diff --git a/vignettes/checks/measurePersonCompleteness.Rmd b/vignettes/checks/measurePersonCompleteness.Rmd index 910c0d22..c1a346d0 100644 --- a/vignettes/checks/measurePersonCompleteness.Rmd +++ b/vignettes/checks/measurePersonCompleteness.Rmd @@ -24,16 +24,16 @@ The number and percent of persons in the CDM that do not have at least one recor ## Definition - *Numerator*: The number of persons with 0 rows in a given CDM table. -- *Denominator*: The total number of persons in the `person` table. -- *Related CDM Convention(s)*: Each Person needs to have at least one `observation_period` record. Otherwise, CDM conventions do not dictate any rules for person completeness. -- *CDM Fields/Tables*: By default, this check runs on all tables with a foreign key to the `person` table. -- *Default Threshold Value*: Set to 95 or 100 for most tables but 0 for `observation_period` +- *Denominator*: The total number of persons in the `PERSON` table. +- *Related CDM Convention(s)*: Each Person needs to have at least one `OBSERVATION_PERIOD` record. Otherwise, CDM conventions do not dictate any rules for person completeness. +- *CDM Fields/Tables*: By default, this check runs on all tables with a foreign key to the `PERSON` table. +- *Default Threshold Value*: Set to 95 or 100 for most tables but 0 for `OBSERVATION_PERIOD` ## User Guidance -For most tables, this check is a characterization of the completeness of various data types in the source data. However, in the case of `observation_period`, this check should actually be considered a CDM convention check as it is used to enforce the requirement that all persons have at least one observation period. -A failure of this check on the `observation_period` table is a serious issue as persons without an `observation_period` cannot be included in any standard OHDSI analysis. +For most tables, this check is a characterization of the completeness of various data types in the source data. However, in the case of `OBSERVATION_PERIOD`, this check should actually be considered a CDM convention check as it is used to enforce the requirement that all persons have at least one observation period. +A failure of this check on the `OBSERVATION_PERIOD` table is a serious issue as persons without an `OBSERVATION_PERIOD` cannot be included in any standard OHDSI analysis. Run the following query to obtain a list of persons who had no data in a given table. From this list of person_ids you may join to other tables in the CDM to understand trends in these individuals' data which may provide clues as to the root cause of the issue. @@ -55,18 +55,19 @@ All persons in the CDM must have an observation period; OHDSI analytics tools on #### All other tables Action on persons missing records in other clinical event tables will depend on the characteristics of the source database. In certain cases, missingness is expected – some persons may just not have a given type of data available in the source. In others, various ETL issues may result in persons missing records in a given event table: + - Mis-mapping of domains, resulting in the placement of records in the incorrect table - Incorrect parsing of source data, resulting in loss of valid records - Failure of an ETL step, resulting in an empty table -If more persons than expected are missing data in a given table, run the violated rows SQL snippet to retrieve these persons’ person_ids, and inspect these persons’ other clinical event data in the CDM for trends. You may also use person_source_value to trace back to these persons’ source data to identify source data records potentially missed by the ETL. +If more persons than expected are missing data in a given table, run the violated rows SQL snippet to retrieve these persons’ person_ids, and inspect these persons’ other clinical event data in the CDM for trends. You may also use `person_source_value` to trace back to these persons’ source data to identify source data records potentially missed by the ETL. Note that in some cases, the failure threshold for this check may need to be adjusted according to completeness expectations for a given data source. ### Data Users -Severe failures, such as unexpected nearly empty tables, must be fixed by the ETL team before a dataset can be used. Note as well that any person missing an `observation_period` will not be able to be included in any analysis using OHDSI tools. +Severe failures, such as unexpected nearly empty tables, must be fixed by the ETL team before a dataset can be used. Note as well that any person missing an observation period will not be able to be included in any analysis using OHDSI tools. Failures with a result close to the specified failure threshold may be accepted, at your own risk and only if the result matches your understanding of the source data. The violated rows SQL may be used to inspect the full records for persons missing data in a given table in order to validate your expectations or point to potential issues in the ETL which need to be resolved. diff --git a/vignettes/checks/measureValueCompleteness.Rmd b/vignettes/checks/measureValueCompleteness.Rmd index 1defb0b1..045837a5 100644 --- a/vignettes/checks/measureValueCompleteness.Rmd +++ b/vignettes/checks/measureValueCompleteness.Rmd @@ -40,13 +40,9 @@ Use this SQL query to inspect rows with a missing value in a given field: ```sql SELECT - -'@cdmTableName.@cdmFieldName' AS violating_field, - -cdmTable.* - + '@cdmTableName.@cdmFieldName' AS violating_field, + cdmTable.* FROM @cdmDatabaseSchema.@cdmTableName cdmTable - WHERE cdmTable.@cdmFieldName IS NULL ``` diff --git a/vignettes/checks/sourceConceptRecordCompleteness.Rmd b/vignettes/checks/sourceConceptRecordCompleteness.Rmd index bf7c0e01..0827a1ab 100644 --- a/vignettes/checks/sourceConceptRecordCompleteness.Rmd +++ b/vignettes/checks/sourceConceptRecordCompleteness.Rmd @@ -23,11 +23,13 @@ The number and percent of records with a value of 0 in the source concept field ## Definition -- *Numerator*: The number of rows with a value of 0 in the `X_source_concept_id` source concept field. In the case of `MEASUREMENT.unit_source_concept_id` and `OBSERVATION.unit_source_concept_id`, the number of rows with a value of 0 in the `X_source_concept_id` source concept field AND a non-NULL value_as_number. -- *Denominator*: The total number of rows in the table. In the case of `MEASUREMENT.unit_source_concept_id` and `OBSERVATION.unit_source_concept_id`, the number of rows with non-NULL value_as_number. +- *Numerator*: The number of rows with a value of 0 in the `X_source_concept_id` source concept field. In the case of `MEASUREMENT.unit_source_concept_id` and `OBSERVATION.unit_source_concept_id`, the number of rows with a value of 0 in the `X_source_concept_id` source concept field AND a non-NULL `value_as_number`. +- *Denominator*: The total number of rows in the table. In the case of `MEASUREMENT.unit_source_concept_id` and `OBSERVATION.unit_source_concept_id`, the number of rows with non-NULL `value_as_number`. - *Related CDM Convention(s)*: [Source concept mapping](https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Fields) - *CDM Fields/Tables*: All source concept ID (`X_source_concept_id`) columns in all event tables. -- *Default Threshold Value*: 10 for primary source concept ID columns in condition, drug, measurement, procedure, device, and observation tables; 100 for all other source concept ID columns. +- *Default Threshold Value*: + - 10 for primary source concept ID columns in condition, drug, measurement, procedure, device, and observation tables + - 100 for all other source concept ID columns ## User Guidance @@ -62,7 +64,7 @@ The query results will give you a summary of the source codes which failed to ma If source values return legitimate matches on concept_code, it’s possible that there is an error in the concept mapping step of your ETL. Please note that while the `X_source_concept_id` fields are technically not required, it is highly recommended to populate them with OMOP concepts whenever possible. This will greatly aid analysts in understanding the provenance of the data. -If source values do NOT return matches on concept_code and you are NOT handling concept mapping locally for a non-OMOP source vocabulary, then you likely have a malformed source code or one that does not exist in the OMOP vocabulary. Please see the documentation in the standardConceptRecordCompleteness page for instructions on how to handle this scenario. +If source values do NOT return matches on concept_code and you are NOT handling concept mapping locally for a non-OMOP source vocabulary, then you likely have a malformed source code or one that does not exist in the OMOP vocabulary. Please see the documentation in the [standardConceptRecordCompleteness](standardConceptRecordCompleteness.html) page for instructions on how to handle this scenario. ### Data Users Since most standard OHDSI analytic workflows rely on the standard concept field and not the source concept field, failures of this check will not necessarily impact your analysis. However, if your analysis depends on understanding source coding practices or on codes you know may not be fully mapped to OMOP standard concepts, then this will be a critical check failure to understand. diff --git a/vignettes/checks/sourceValueCompleteness.Rmd b/vignettes/checks/sourceValueCompleteness.Rmd index 6dc5e3b2..9e39cfba 100644 --- a/vignettes/checks/sourceValueCompleteness.Rmd +++ b/vignettes/checks/sourceValueCompleteness.Rmd @@ -23,7 +23,7 @@ The number and percent of distinct source values in the @cdmFieldName field of t ## Definition -- *Numerator*: Distinct `X_source_value` entries where the corresponding standard x_concept_id field is 0. +- *Numerator*: Distinct `X_source_value` entries where the corresponding standard `X_concept_id` field is 0. - *Denominator*: Total distinct `X_source_value` entries, including NULL, in the respective event table. - *Related CDM Convention(s)*: The OMOP Common Data Model specifies that codes that are present in a native database should be mapped to standard concepts using either the intrinsic mappings defined in the standard vocabularies or extrinsic mappings defined by the data owner or ETL development team. Note also that variations of this check logic are also used in the [EHDEN CDM Inspection Report](https://github.com/EHDEN/CdmInspection) package, as well as the [AresIndexer](https://github.com/OHDSI/AresIndexer) package for generating indices of unmapped codes. - *CDM Fields/Tables*: Runs on all event tables that have `X_source_value` fields. @@ -33,7 +33,7 @@ The number and percent of distinct source values in the @cdmFieldName field of t ## User Guidance -This check will look at all distinct source values in the specified field and calculate how many are mapped to a standard concept of 0. This check should be used in conjunction with the [standardConceptRecordCompleteness](checks/standardConceptRecordCompleteness.html) check to identify potential mapping issues in the ETL. \ +This check will look at all distinct source values in the specified field and calculate how many are mapped to a standard concept of 0. This check should be used in conjunction with the [standardConceptRecordCompleteness](standardConceptRecordCompleteness.html) check to identify potential mapping issues in the ETL. \ This check is a good measure of the overall mapping rate within each domain. For example, a table may have high standardConceptRecordCompleteness (that is, a large percentage of records with a non-zero standard concept) but a low score on this check. This would indicate that the "long tail" of rarer codes have not been mapped while more common codes have good mapping coverage. It is always important to interrogate the results of these two checks together to ensure complete understanding of vocabulary mapping in your CDM. diff --git a/vignettes/checks/standardConceptRecordCompleteness.Rmd b/vignettes/checks/standardConceptRecordCompleteness.Rmd index 6a67aab8..e745a21f 100644 --- a/vignettes/checks/standardConceptRecordCompleteness.Rmd +++ b/vignettes/checks/standardConceptRecordCompleteness.Rmd @@ -23,8 +23,8 @@ The number and percent of records with a value of 0 in the standard concept fiel ## Definition -- *Numerator*: The number of rows with a value of 0 in the `X_concept_id` standard concept field. (**NOTE** in the case of `measurement.unit_concept_id` and `observation.unit_concept_id`, the number of rows with a value of 0 in the `X_concept_id` standard concept field AND a non-NULL value_as_number.) -- *Denominator*: The total number of rows in the table. (**NOTE** in the case of `measurement.unit_concept_id` and `observation.unit_concept_id`, the number of rows with non-NULL `value_as_number`.) +- *Numerator*: The number of rows with a value of 0 in the `X_concept_id` standard concept field. In the case of `MEASUREMENT.unit_concept_id` and `OBSERVATION.unit_concept_id`, the number of rows with a value of 0 in the `X_concept_id` standard concept field AND a non-NULL `value_as_number`. +- *Denominator*: The total number of rows in the table. In the case of `MEASUREMENT.unit_concept_id` and `OBSERVATION.unit_concept_id`, the number of rows with a non-NULL `value_as_number`. - *Related CDM Convention(s)*: [Standard concept mapping](https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Fields) - *CDM Fields/Tables*: All standard concept ID (`X_concept_id`) columns in all event tables. - *Default Threshold Value*: diff --git a/vignettes/checks/withinVisitDates.Rmd b/vignettes/checks/withinVisitDates.Rmd index b6c9ec1e..6bd7aaa8 100644 --- a/vignettes/checks/withinVisitDates.Rmd +++ b/vignettes/checks/withinVisitDates.Rmd @@ -1,6 +1,6 @@ --- title: "withinVisitDates" -author: "" +author: "Clair Blacketer" date: "`r Sys.Date()`" output: html_document: @@ -14,33 +14,52 @@ output: **Context**: Verification\ **Category**: Conformance\ **Subcategory**: \ -**Severity**: +**Severity**: Characterization ✔ ## Description -The number and percent of records not within one week on either side of the corresponding visit occurrence start and end date +The number and percent of records that occur one week before the corresponding `visit_start_date` or one week after the corresponding `visit_end_date` ## Definition -- *Numerator*: -- *Denominator*: -- *Related CDM Convention(s)*: -- *CDM Fields/Tables*: -- *Default Threshold Value*: +- *Numerator*: The number of rows in a table where the event date occurs more than 7 days prior to the corresponding visit start date or more than 7 days after the corresponding visit end date. +- *Denominator*: The total number of rows in the table with a corresponding visit (linked through `visit_occurrence_id`) +- *Related CDM Convention(s)*: There is no explicit convention tied to this check. However, the CDM documentation describes the `visit_occurrence_id` foreign key in the event tables as “The visit during which the occurred.” The underlying assumption is that if a record is tied to a visit, then the date of the record should fall in some reasonable time period around the visit dates. This gives a week of leeway on either side for physician notes or other activities related to a visit to be recorded. +- *CDM Fields/Tables*: This check runs on all event tables: `CONDITION_OCCURRENCE`, `PROCEDURE_OCCURRENCE`, `DRUG_EXPOSURE`, `DEVICE_EXPOSURE`, `MEASUREMENT`, `NOTE`, `OBSERVATION`, and `VISIT_DETAIL`. It will check either the `X_date` or `X_start_date` fields for alignment with corresponding `VISIT_OCCURRENCE` dates by linking on the `visit_occurrence_id`. (**Note:** For VISIT_DETAIL it will check both the visit_detail_start_date and visit_detail_end_date. The default threshold for these two checks is 1%.) +- *Default Threshold Value*: 1% for `VISIT_DETAIL`, 5% for all other tables ## User Guidance +There is no explicit convention that describes how events should align temporally with the visits they correspond to. This check is meant to identify egregious mismatches in dates that could signify an incorrect date field was used in the ETL or that the data should be used with caution if there is no reason for the mismatch (history of a condition, for example). +If this check fails the first action should be to investigate the failing rows for any patterns. The main query to find failing rows is below: ### Violated rows query ```sql - +SELECT + '@cdmTableName.@cdmFieldName' AS violating_field, + vo.visit_start_date, vo.visit_end_date, vo.person_id, + cdmTable.* +FROM @cdmDatabaseSchema.@cdmTableName cdmTable +JOIN @cdmDatabaseSchema.visit_occurrence vo + ON cdmTable.visit_occurrence_id = vo.visit_occurrence_id +WHERE cdmTable.@cdmFieldName < dateadd(day, -7, vo.visit_start_date) + OR cdmTable.@cdmFieldName > dateadd(day, 7, vo.visit_end_date) ``` - ### ETL Developers +The first step is to investigate whether visit and event indeed should be linked - e.g., do they belong to the same person; how far are the dates apart; is it possible the event was recorded during the visit. If they should be linked, then the next step is to investigate which of the event date and visit date is accurate. + +One suggestion would be to identify if all of the failures are due to many events all having the same date. In some institutions there is a default date given to events in the case where a date is not given. Should this be the problem, the first step should be to identify if there is a different date field in the native data that can be used. If not, considerations should be made to determine if the rows should be dropped. Without a correct date it is challenging to use such events in health outcomes research. +Another reason for the discrepancy could be that the wrong date has been used for events. For instance, in some systems a diagnosis could have both an ‘observation date’ and an ‘administration date’. If the physician is updating records at a later date, the administration date can be later than the actual date the diagnosis was observed. In those cases, the observation date has to be used. If there is only an administration date, it is in some cases arguable to use the visit date for the diagnosis date. + +Another suggestion would be to investigate if the failures are related to ‘History of’ conditions. It is often the case that a patient’s history is recorded during a visit, in which case they may report a diagnosis date prior to the given visit. In some cases it may be appropriate to conserve these records; the decision to do so will depend on the reliability of the recorded date in your source data. ### Data Users +If the failure percentage of withinVisitDates is high, a data user should be careful with using the data. This check might indicate a larger underlying conformance issue with either the event dates or linkage with visits. At the same time, there might be a valid reason why events do not happen within 7 days of the linked visit. + +Make sure to understand why this check fails. Specifically, be careful when using such data in outcomes research. Without specific dates for an event, it is challenging to determine if an adverse event occurred after a drug exposure, for example. +Note that this check specifically compares event dates to `VISIT_OCCURRENCE` dates. There is no equivalent check for `VISIT_DETAIL` that verifies whether the event date is within (a week of) the visit detail start and end dates.