From 327870de3725fadb5fd9d60ce40e8c63b2b1d19b Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Sat, 6 Jul 2024 22:18:37 +0200 Subject: [PATCH 01/16] add description for conditionEraCompleteness --- .../measureConditionEraCompleteness.Rmd | 32 ++++++++++++------- 1 file changed, 21 insertions(+), 11 deletions(-) diff --git a/vignettes/checks/measureConditionEraCompleteness.Rmd b/vignettes/checks/measureConditionEraCompleteness.Rmd index 1eaf2b86..f2ffdb46 100644 --- a/vignettes/checks/measureConditionEraCompleteness.Rmd +++ b/vignettes/checks/measureConditionEraCompleteness.Rmd @@ -14,34 +14,44 @@ output: **Context**: Validation\ **Category**: Completeness\ **Subcategory**: \ -**Severity**: +**Severity**: CDM convention ⚠ ## Description -The number and Percent of persons that does not have condition_era built successfully, -for all persons in condition_occurrence +The number and percent of persons that does not have condition_era built successfully, +for all persons in condition_occurrence. ## Definition -- *Numerator*: -- *Denominator*: -- *Related CDM Convention(s)*: -- *CDM Fields/Tables*: -- *Default Threshold Value*: +- *Numerator*: Number of unique person_ids that exist in the condition_occurrence table but not in the condition_era table. +- *Denominator*: Number of unique person_ids in the condition_occurrence table. +- *Related CDM Convention(s)*: Condition Era is derived from Condition Occurrence. +- *CDM Fields/Tables*: condition_era +- *Default Threshold Value*: 0% ## User Guidance +The [Condition Era CDM documentation](https://ohdsi.github.io/CommonDataModel/cdm54.html#condition_era) states that the condition era should be derived by combining condition occurrences. This implies that each person with a condition occurrence should have at least a condition era. +It does NOT clearly state that the condition_era table is required when there are condition occurrences. Still, it is has always been a common convention in the OHDSI community to derive condition era. +There is no THEMIS convention on condition era. ### Violated rows query ```sql - +SELECT DISTINCT + co.person_id +FROM @cdmDatabaseSchema.condition_occurrence co +LEFT JOIN @cdmDatabaseSchema.condition_era cdmTable + ON co.person_id = cdmTable.person_id +WHERE cdmTable.person_id IS NULL ``` - ### ETL Developers +If this check fails, it is likely that there is an issue with the condition era derivation script. Please review the ETL execution log. It might be that this script was not executed and the condition era table is empty, or it had issues running and the condition era table has been partially populated. +If no issues with the ETL run found, the condition era derivation script might have bugs. Please review the code. An example script can be found on [the CDM Documentation page](https://ohdsi.github.io/CommonDataModel/sqlScripts.html#Condition_Eras). +In both cases it is advised to truncate the condition era table and rerun the derivation script. ### Data Users - +The condition era might seem redundant, as for most uses the condition occurrence table can be used. However, tools like FeatureExtraction use condition era to build some covariates and network studies might use cohorts that are based on condition era. It is therefore important that the condition era table is fully populated and captures the same persons as in condition occurrence. From 63d4126053f4613292bce4eec48be557861a2ae5 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Sun, 7 Jul 2024 23:27:52 +0200 Subject: [PATCH 02/16] table syntax --- .../measureConditionEraCompleteness.Rmd | 24 +++++++++---------- 1 file changed, 12 insertions(+), 12 deletions(-) diff --git a/vignettes/checks/measureConditionEraCompleteness.Rmd b/vignettes/checks/measureConditionEraCompleteness.Rmd index f2ffdb46..9655ef69 100644 --- a/vignettes/checks/measureConditionEraCompleteness.Rmd +++ b/vignettes/checks/measureConditionEraCompleteness.Rmd @@ -19,22 +19,22 @@ output: ## Description The number and percent of persons that does not have condition_era built successfully, -for all persons in condition_occurrence. +for all persons in `CONDITION_OCCURRENCE`. ## Definition -- *Numerator*: Number of unique person_ids that exist in the condition_occurrence table but not in the condition_era table. -- *Denominator*: Number of unique person_ids in the condition_occurrence table. -- *Related CDM Convention(s)*: Condition Era is derived from Condition Occurrence. -- *CDM Fields/Tables*: condition_era +- *Numerator*: Number of unique person_ids that exist in the `CONDITION_OCCURRENCE` table but not in the `CONDITION_ERA` table. +- *Denominator*: Number of unique person_ids in the `CONDITION_OCCURRENCE` table. +- *Related CDM Convention(s)*: Condition Era's are directly derived from Condition Occurrence. +- *CDM Fields/Tables*: `CONDITION_ERA` - *Default Threshold Value*: 0% ## User Guidance -The [Condition Era CDM documentation](https://ohdsi.github.io/CommonDataModel/cdm54.html#condition_era) states that the condition era should be derived by combining condition occurrences. This implies that each person with a condition occurrence should have at least a condition era. -It does NOT clearly state that the condition_era table is required when there are condition occurrences. Still, it is has always been a common convention in the OHDSI community to derive condition era. -There is no THEMIS convention on condition era. +The [Condition Era CDM documentation](https://ohdsi.github.io/CommonDataModel/cdm54.html#condition_era) states that the condition era's should be derived by combining condition occurrences. This implies that each person with a condition occurrence should have at least a condition era. +It does NOT clearly state that the `CONDITION_ERA` table is required when there are condition occurrences. Still, it is has always been a common convention in the OHDSI community to derive condition era. +There is currently no THEMIS convention on condition eras. ### Violated rows query @@ -42,16 +42,16 @@ There is no THEMIS convention on condition era. SELECT DISTINCT co.person_id FROM @cdmDatabaseSchema.condition_occurrence co -LEFT JOIN @cdmDatabaseSchema.condition_era cdmTable - ON co.person_id = cdmTable.person_id + LEFT JOIN @cdmDatabaseSchema.condition_era cdmTable + ON co.person_id = cdmTable.person_id WHERE cdmTable.person_id IS NULL ``` ### ETL Developers If this check fails, it is likely that there is an issue with the condition era derivation script. Please review the ETL execution log. It might be that this script was not executed and the condition era table is empty, or it had issues running and the condition era table has been partially populated. If no issues with the ETL run found, the condition era derivation script might have bugs. Please review the code. An example script can be found on [the CDM Documentation page](https://ohdsi.github.io/CommonDataModel/sqlScripts.html#Condition_Eras). -In both cases it is advised to truncate the condition era table and rerun the derivation script. +In both cases it is advised to truncate the `CONDITION_ERA` table and rerun the derivation script. ### Data Users -The condition era might seem redundant, as for most uses the condition occurrence table can be used. However, tools like FeatureExtraction use condition era to build some covariates and network studies might use cohorts that are based on condition era. It is therefore important that the condition era table is fully populated and captures the same persons as in condition occurrence. +The `CONDITION_ERA` table might seem to contain redundant information, as for most uses the `CONDITION_OCCURRENCE` table can be used. However, tools like FeatureExtraction use condition eras to build some covariates and network studies might use cohorts that are based on condition eras. It is therefore important that the `CONDITION_ERA` table is fully populated and captures the same persons as in condition occurrence. From 83e795fa656322c7d8c5688e0e4a6e41efd7bb68 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Fri, 12 Jul 2024 16:50:32 +0200 Subject: [PATCH 03/16] add author --- vignettes/checks/measureConditionEraCompleteness.Rmd | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/vignettes/checks/measureConditionEraCompleteness.Rmd b/vignettes/checks/measureConditionEraCompleteness.Rmd index 9655ef69..4673c616 100644 --- a/vignettes/checks/measureConditionEraCompleteness.Rmd +++ b/vignettes/checks/measureConditionEraCompleteness.Rmd @@ -1,6 +1,6 @@ --- title: "measureConditionEraCompleteness" -author: "" +author: "Maxim Moinat" date: "`r Sys.Date()`" output: html_document: From 9713cbafdb7d9375b7173e4ceae839974eed9ab9 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Tue, 13 Feb 2024 16:59:28 +0100 Subject: [PATCH 04/16] refactor NA status calculation --- R/calculateNotApplicableStatus.R | 147 +++++++++++++++++++++++++++++++ R/evaluateThresholds.R | 114 +----------------------- 2 files changed, 148 insertions(+), 113 deletions(-) create mode 100644 R/calculateNotApplicableStatus.R diff --git a/R/calculateNotApplicableStatus.R b/R/calculateNotApplicableStatus.R new file mode 100644 index 00000000..2ee7fae4 --- /dev/null +++ b/R/calculateNotApplicableStatus.R @@ -0,0 +1,147 @@ +# Copyright 2023 Observational Health Data Sciences and Informatics +# +# This file is part of DataQualityDashboard +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +#' Determines if check should be notApplicable +#' +#' @param checkResults A dataframe containing the results of the data quality checks +#' +#' @keywords internal + +.calculateNotApplicableStatus <- function(checkResults) { + # Look up missing tables and add variable tableIsMissing to checkResults + missingTables <- checkResults %>% + dplyr::filter( + .data$checkName == "cdmTable" + ) %>% + dplyr::mutate( + .data$cdmTableName, + tableIsMissing = .data$failed == 1, + .keep = "none" + ) + + # Look up missing fields and add variable fieldIsMissing to checkResults + missingFields <- checkResults %>% + dplyr::filter( + .data$checkName == "cdmField" + ) %>% + dplyr::mutate( + .data$cdmTableName, + .data$cdmFieldName, + fieldIsMissing = .data$failed == 1, + .keep = "none" + ) + + # Look up empty tables and add variable tableIsEmpty to checkResults + emptyTables <- checkResults %>% + dplyr::filter( + .data$checkName == "measureValueCompleteness" + ) %>% + dplyr::mutate( + .data$cdmTableName, + tableIsEmpty = .data$numDenominatorRows == 0, + .keep = "none" + ) %>% + dplyr::distinct() + + # Look up empty fields and add variable tableIsEmpty to checkResults + emptyFields <- checkResults %>% + dplyr::filter( + .data$checkName == "measureValueCompleteness" + ) %>% + dplyr::mutate( + .data$cdmTableName, + .data$cdmFieldName, + fieldIsEmpty = .data$numDenominatorRows == .data$numViolatedRows, + .keep = "none" + ) + + # Assign notApplicable status + checkResults <- checkResults %>% + dplyr::left_join( + missingTables, + by = "cdmTableName" + ) %>% + dplyr::left_join( + missingFields, + by = c("cdmTableName", "cdmFieldName") + ) %>% + dplyr::left_join( + emptyTables, + by = "cdmTableName" + ) %>% + dplyr::left_join( + emptyFields, + by = c("cdmTableName", "cdmFieldName") + ) %>% + dplyr::mutate( + conceptIsMissing = .data$checkLevel == "CONCEPT" & is.na(.data$unitConceptId) & .data$numDenominatorRows == 0, + conceptAndUnitAreMissing = .data$checkLevel == "CONCEPT" & !is.na(.data$unitConceptId) & .data$numDenominatorRows == 0, + fieldIsMissing = dplyr::coalesce(.data$fieldIsMissing, !is.na(.data$cdmFieldName)), + fieldIsEmpty = dplyr::coalesce(.data$fieldIsEmpty, !is.na(.data$cdmFieldName)), + ) + + for (i in seq_len(nrow(checkResults))) { + checkResults$notApplicable[i] <- .applyNotApplicable(checkResults[i, ]) + } + + checkResults <- checkResults %>% + dplyr::mutate( + notApplicableReason = ifelse( + .data$notApplicable == 1, + dplyr::case_when( + .data$tableIsMissing ~ sprintf("Table %s does not exist.", .data$cdmTableName), + .data$fieldIsMissing ~ sprintf("Field %s.%s does not exist.", .data$cdmTableName, .data$cdmFieldName), + .data$tableIsEmpty ~ sprintf("Table %s is empty.", .data$cdmTableName), + .data$fieldIsEmpty ~ sprintf("Field %s.%s is not populated.", .data$cdmTableName, .data$cdmFieldName), + .data$conceptIsMissing ~ sprintf("%s=%s is missing from the %s table.", .data$cdmFieldName, .data$conceptId, .data$cdmTableName), + .data$conceptAndUnitAreMissing ~ sprintf("Combination of %s=%s, unitConceptId=%s and VALUE_AS_NUMBER IS NOT NULL is missing from the %s table.", .data$cdmFieldName, .data$conceptId, .data$unitConceptId, .data$cdmTableName) + ), + NA + ), + failed = ifelse(.data$notApplicable == 1, 0, .data$failed), + passed = ifelse(.data$failed == 0 & .data$isError == 0 & .data$notApplicable == 0, 1, 0) + ) %>% + dplyr::select(-c("tableIsMissing", "fieldIsMissing", "tableIsEmpty", "fieldIsEmpty", "conceptIsMissing", "conceptAndUnitAreMissing")) + + return(checkResults) +} + +.applyNotApplicable <- function(x) { + # Errors precede all other statuses + if (x$isError == 1) { + return(0) + } + + # No NA status for cdmTable and cdmField if missing + if (x$checkName == "cdmTable" || x$checkName == "cdmField") { + return(0) + } + + if (any(x$tableIsMissing, x$fieldIsMissing, x$tableIsEmpty)) { + return(1) + } + + # No NA status for measureValueCompleteness if empty + if (x$checkName == "measureValueCompleteness") { + return(0) + } + + if (any(x$fieldIsEmpty, x$conceptIsMissing, x$conceptAndUnitAreMissing)) { + return(1) + } + + return(0) +} diff --git a/R/evaluateThresholds.R b/R/evaluateThresholds.R index 21280ba0..accb6a21 100644 --- a/R/evaluateThresholds.R +++ b/R/evaluateThresholds.R @@ -163,119 +163,7 @@ } } - missingTables <- dplyr::select( - dplyr::filter(checkResults, .data$checkName == "cdmTable" & .data$failed == 1), - "cdmTableName" - ) - if (nrow(missingTables) > 0) { - missingTables$tableIsMissing <- 1 - checkResults <- dplyr::mutate( - dplyr::left_join(checkResults, missingTables, by = "cdmTableName"), - tableIsMissing = ifelse(.data$checkName != "cdmTable" & .data$isError == 0, .data$tableIsMissing, NA) - ) - } else { - checkResults$tableIsMissing <- NA - } - - missingFields <- dplyr::select( - dplyr::filter(checkResults, .data$checkName == "cdmField" & .data$failed == 1 & is.na(.data$tableIsMissing)), - "cdmTableName", "cdmFieldName" - ) - if (nrow(missingFields) > 0) { - missingFields$fieldIsMissing <- 1 - checkResults <- dplyr::mutate( - dplyr::left_join(checkResults, missingFields, by = c("cdmTableName", "cdmFieldName")), - fieldIsMissing = ifelse(.data$checkName != "cdmField" & .data$isError == 0, .data$fieldIsMissing, NA) - ) - } else { - checkResults$fieldIsMissing <- NA - } - - emptyTables <- dplyr::distinct( - dplyr::select( - dplyr::filter(checkResults, .data$checkName == "measureValueCompleteness" & - .data$numDenominatorRows == 0 & - .data$isError == 0 & - is.na(.data$tableIsMissing) & - is.na(.data$fieldIsMissing)), - "cdmTableName" - ) - ) - if (nrow(emptyTables) > 0) { - emptyTables$tableIsEmpty <- 1 - checkResults <- dplyr::mutate( - dplyr::left_join(checkResults, emptyTables, by = c("cdmTableName")), - tableIsEmpty = ifelse(.data$checkName != "cdmField" & .data$checkName != "cdmTable" & .data$isError == 0, .data$tableIsEmpty, NA) - ) - } else { - checkResults$tableIsEmpty <- NA - } - - emptyFields <- - dplyr::select( - dplyr::filter(checkResults, .data$checkName == "measureValueCompleteness" & - .data$numDenominatorRows == .data$numViolatedRows & - is.na(.data$tableIsMissing) & is.na(.data$fieldIsMissing) & is.na(.data$tableIsEmpty)), - "cdmTableName", "cdmFieldName" - ) - if (nrow(emptyFields) > 0) { - emptyFields$fieldIsEmpty <- 1 - checkResults <- dplyr::mutate( - dplyr::left_join(checkResults, emptyFields, by = c("cdmTableName", "cdmFieldName")), - fieldIsEmpty = ifelse(.data$checkName != "measureValueCompleteness" & .data$checkName != "cdmField" & .data$checkName != "isRequired" & .data$isError == 0, .data$fieldIsEmpty, NA) - ) - } else { - checkResults$fieldIsEmpty <- NA - } - - checkResults <- dplyr::mutate( - checkResults, - conceptIsMissing = ifelse( - .data$isError == 0 & - is.na(.data$tableIsMissing) & - is.na(.data$fieldIsMissing) & - is.na(.data$tableIsEmpty) & - is.na(.data$fieldIsEmpty) & - .data$checkLevel == "CONCEPT" & - is.na(.data$unitConceptId) & - .data$numDenominatorRows == 0, - 1, - NA - ) - ) - - checkResults <- dplyr::mutate( - checkResults, - conceptAndUnitAreMissing = ifelse( - .data$isError == 0 & - is.na(.data$tableIsMissing) & - is.na(.data$fieldIsMissing) & - is.na(.data$tableIsEmpty) & - is.na(.data$fieldIsEmpty) & - .data$checkLevel == "CONCEPT" & - !is.na(.data$unitConceptId) & - .data$numDenominatorRows == 0, - 1, - NA - ) - ) - - checkResults <- dplyr::mutate( - checkResults, - notApplicable = dplyr::coalesce(.data$tableIsMissing, .data$fieldIsMissing, .data$tableIsEmpty, .data$fieldIsEmpty, .data$conceptIsMissing, .data$conceptAndUnitAreMissing, 0), - notApplicableReason = dplyr::case_when( - !is.na(.data$tableIsMissing) ~ sprintf("Table %s does not exist.", .data$cdmTableName), - !is.na(.data$fieldIsMissing) ~ sprintf("Field %s.%s does not exist.", .data$cdmTableName, .data$cdmFieldName), - !is.na(.data$tableIsEmpty) ~ sprintf("Table %s is empty.", .data$cdmTableName), - !is.na(.data$fieldIsEmpty) ~ sprintf("Field %s.%s is not populated.", .data$cdmTableName, .data$cdmFieldName), - !is.na(.data$conceptIsMissing) ~ sprintf("%s=%s is missing from the %s table.", .data$cdmFieldName, .data$conceptId, .data$cdmTableName), - !is.na(.data$conceptAndUnitAreMissing) ~ sprintf("Combination of %s=%s, unitConceptId=%s and VALUE_AS_NUMBER IS NOT NULL is missing from the %s table.", .data$cdmFieldName, .data$conceptId, .data$unitConceptId, .data$cdmTableName) - ) - ) - - checkResults <- dplyr::select(checkResults, -c("tableIsMissing", "fieldIsMissing", "tableIsEmpty", "fieldIsEmpty", "conceptIsMissing", "conceptAndUnitAreMissing")) - checkResults <- dplyr::mutate(checkResults, failed = ifelse(.data$notApplicable == 1, 0, .data$failed)) - checkResults <- dplyr::mutate(checkResults, passed = ifelse(.data$failed == 0 & .data$isError == 0 & .data$notApplicable == 0, 1, 0)) + checkResults <- .calculateNotApplicableStatus(checkResults) checkResults } From d8de10cea0dff9c680269a39b5b0eac0fae5d0f3 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Tue, 13 Feb 2024 17:57:56 +0100 Subject: [PATCH 05/16] fixes #526 with an exception for NA status of measureConditionEraCompleteness --- R/calculateNotApplicableStatus.R | 28 ++++++++++++++++++++++++++-- 1 file changed, 26 insertions(+), 2 deletions(-) diff --git a/R/calculateNotApplicableStatus.R b/R/calculateNotApplicableStatus.R index 2ee7fae4..41536c43 100644 --- a/R/calculateNotApplicableStatus.R +++ b/R/calculateNotApplicableStatus.R @@ -14,7 +14,7 @@ # See the License for the specific language governing permissions and # limitations under the License. -#' Determines if check should be notApplicable +#' Determines if check should be notApplicable and the notApplicableReason #' #' @param checkResults A dataframe containing the results of the data quality checks #' @@ -93,8 +93,31 @@ fieldIsEmpty = dplyr::coalesce(.data$fieldIsEmpty, !is.na(.data$cdmFieldName)), ) + checkResults$notApplicable <- NA + checkResults$notApplicableReason <- NA + + conditionOccurrenceIsEmpty <- emptyTables %>% dplyr::filter(.data$cdmTableName == "CONDITION_OCCURRENCE") %>% dplyr::pull(tableIsEmpty) + # drugExposureIsEmpty <- emptyTables %>% filter(.data$cdmTableName == "DRUG_EXPOSURE") %>% pull(tableIsEmpty) for (i in seq_len(nrow(checkResults))) { - checkResults$notApplicable[i] <- .applyNotApplicable(checkResults[i, ]) + if (checkResults[i, "checkName"] == "measureConditionEraCompleteness") { + # Special rule for measureConditionEraCompleteness, which should be notApplicable if CONDITION_OCCURRENCE is empty + if (conditionOccurrenceIsEmpty) { + checkResults$notApplicable[i] <- 1 + checkResults$notApplicableReason[i] <- "Table CONDITION_OCCURRENCE is empty." + } else { + checkResults$notApplicable[i] <- 0 + } + # } else if (checkResult[i, "checkName"] == "measureDrugEraCompleteness") { + # # Special rule for measureDrugEraCompleteness, which should be notApplicable if DRUG_EXPOSURE is empty + # if (drugExposureIsEmpty) { + # checkResults$notApplicable[i] <- 1 + # checkResults$notApplicableReason[i] <- "Table DRUG_EXPOSURE is empty." + # } else { + # checkResults$notApplicable[i] <- 0 + # } + } else { + checkResults$notApplicable[i] <- .applyNotApplicable(checkResults[i, ]) + } } checkResults <- checkResults %>% @@ -102,6 +125,7 @@ notApplicableReason = ifelse( .data$notApplicable == 1, dplyr::case_when( + !is.na(.data$notApplicableReason) ~ .data$notApplicableReason, .data$tableIsMissing ~ sprintf("Table %s does not exist.", .data$cdmTableName), .data$fieldIsMissing ~ sprintf("Field %s.%s does not exist.", .data$cdmTableName, .data$cdmFieldName), .data$tableIsEmpty ~ sprintf("Table %s is empty.", .data$cdmTableName), From 68da184d347d8b00929cb734b71a3fb23d1580bc Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Tue, 13 Feb 2024 18:03:27 +0100 Subject: [PATCH 06/16] add rule for missing condition occurrence --- R/calculateNotApplicableStatus.R | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/R/calculateNotApplicableStatus.R b/R/calculateNotApplicableStatus.R index 41536c43..81f0c7c3 100644 --- a/R/calculateNotApplicableStatus.R +++ b/R/calculateNotApplicableStatus.R @@ -96,12 +96,14 @@ checkResults$notApplicable <- NA checkResults$notApplicableReason <- NA + conditionOccurrenceIsMissing <- missingTables %>% dplyr::filter(.data$cdmTableName == "CONDITION_OCCURRENCE") %>% dplyr::pull(tableIsMissing) conditionOccurrenceIsEmpty <- emptyTables %>% dplyr::filter(.data$cdmTableName == "CONDITION_OCCURRENCE") %>% dplyr::pull(tableIsEmpty) + # drugExposureIsMissing <- missingTables %>% filter(.data$cdmTableName == "DRUG_EXPOSURE") %>% pull(tableIsMissing) # drugExposureIsEmpty <- emptyTables %>% filter(.data$cdmTableName == "DRUG_EXPOSURE") %>% pull(tableIsEmpty) for (i in seq_len(nrow(checkResults))) { if (checkResults[i, "checkName"] == "measureConditionEraCompleteness") { # Special rule for measureConditionEraCompleteness, which should be notApplicable if CONDITION_OCCURRENCE is empty - if (conditionOccurrenceIsEmpty) { + if (conditionOccurrenceIsMissing || conditionOccurrenceIsEmpty) { checkResults$notApplicable[i] <- 1 checkResults$notApplicableReason[i] <- "Table CONDITION_OCCURRENCE is empty." } else { @@ -109,7 +111,7 @@ } # } else if (checkResult[i, "checkName"] == "measureDrugEraCompleteness") { # # Special rule for measureDrugEraCompleteness, which should be notApplicable if DRUG_EXPOSURE is empty - # if (drugExposureIsEmpty) { + # if (drugExposureIsMissing || drugExposureIsEmpty) { # checkResults$notApplicable[i] <- 1 # checkResults$notApplicableReason[i] <- "Table DRUG_EXPOSURE is empty." # } else { From 8a1b3f5cfa73e815a03bc34b86177c548fa5d01f Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Thu, 22 Feb 2024 08:51:14 +0100 Subject: [PATCH 07/16] fix failing test by checking if required checks are present --- R/calculateNotApplicableStatus.R | 41 ++++++++++++++++++++++---------- R/evaluateThresholds.R | 4 +++- R/executeDqChecks.R | 10 ++------ 3 files changed, 33 insertions(+), 22 deletions(-) diff --git a/R/calculateNotApplicableStatus.R b/R/calculateNotApplicableStatus.R index 81f0c7c3..ace2c329 100644 --- a/R/calculateNotApplicableStatus.R +++ b/R/calculateNotApplicableStatus.R @@ -14,6 +14,31 @@ # See the License for the specific language governing permissions and # limitations under the License. +#' Determines if all checks are present expected to calculate the 'Not Applicable' status +#' +#' @param checkResults A dataframe containing the results of the data quality checks +#' +#' @keywords internal +.hasNAchecks <- function(checkResults) { + checkNames <- unique(checkResults$checkName) + return(.containsNAchecks(checkNames)) +} + +#' Determines if all checks required for 'Not Applicable' status are in the checkNames +#' +#' @param checkNames A character vector of check names +#' +#' @keywords internal +.containsNAchecks <- function(checkNames) { + naCheckNames <- c("cdmTable", "cdmField", "measureValueCompleteness") + missingNAChecks <- !(naCheckNames %in% checkNames) + if (any(missingNAChecks)) { + missingNACheckNames <- paste(naCheckNames[missingNAChecks], collapse = ", ") + return(FALSE) + } + return(TRUE) +} + #' Determines if check should be notApplicable and the notApplicableReason #' #' @param checkResults A dataframe containing the results of the data quality checks @@ -98,25 +123,15 @@ conditionOccurrenceIsMissing <- missingTables %>% dplyr::filter(.data$cdmTableName == "CONDITION_OCCURRENCE") %>% dplyr::pull(tableIsMissing) conditionOccurrenceIsEmpty <- emptyTables %>% dplyr::filter(.data$cdmTableName == "CONDITION_OCCURRENCE") %>% dplyr::pull(tableIsEmpty) - # drugExposureIsMissing <- missingTables %>% filter(.data$cdmTableName == "DRUG_EXPOSURE") %>% pull(tableIsMissing) - # drugExposureIsEmpty <- emptyTables %>% filter(.data$cdmTableName == "DRUG_EXPOSURE") %>% pull(tableIsEmpty) for (i in seq_len(nrow(checkResults))) { + # Special rule for measureConditionEraCompleteness, which should be notApplicable if CONDITION_OCCURRENCE is empty if (checkResults[i, "checkName"] == "measureConditionEraCompleteness") { - # Special rule for measureConditionEraCompleteness, which should be notApplicable if CONDITION_OCCURRENCE is empty if (conditionOccurrenceIsMissing || conditionOccurrenceIsEmpty) { checkResults$notApplicable[i] <- 1 checkResults$notApplicableReason[i] <- "Table CONDITION_OCCURRENCE is empty." } else { checkResults$notApplicable[i] <- 0 } - # } else if (checkResult[i, "checkName"] == "measureDrugEraCompleteness") { - # # Special rule for measureDrugEraCompleteness, which should be notApplicable if DRUG_EXPOSURE is empty - # if (drugExposureIsMissing || drugExposureIsEmpty) { - # checkResults$notApplicable[i] <- 1 - # checkResults$notApplicableReason[i] <- "Table DRUG_EXPOSURE is empty." - # } else { - # checkResults$notApplicable[i] <- 0 - # } } else { checkResults$notApplicable[i] <- .applyNotApplicable(checkResults[i, ]) } @@ -156,7 +171,7 @@ return(0) } - if (any(x$tableIsMissing, x$fieldIsMissing, x$tableIsEmpty)) { + if (any(x$tableIsMissing, x$fieldIsMissing, x$tableIsEmpty, na.rm = TRUE)) { return(1) } @@ -165,7 +180,7 @@ return(0) } - if (any(x$fieldIsEmpty, x$conceptIsMissing, x$conceptAndUnitAreMissing)) { + if (any(x$fieldIsEmpty, x$conceptIsMissing, x$conceptAndUnitAreMissing, na.rm = TRUE)) { return(1) } diff --git a/R/evaluateThresholds.R b/R/evaluateThresholds.R index accb6a21..904d0af4 100644 --- a/R/evaluateThresholds.R +++ b/R/evaluateThresholds.R @@ -163,7 +163,9 @@ } } - checkResults <- .calculateNotApplicableStatus(checkResults) + if (.hasNAchecks(checkResults)) { + checkResults <- .calculateNotApplicableStatus(checkResults) + } checkResults } diff --git a/R/executeDqChecks.R b/R/executeDqChecks.R index 7bb40e75..e153b3c9 100644 --- a/R/executeDqChecks.R +++ b/R/executeDqChecks.R @@ -114,16 +114,10 @@ executeDqChecks <- function(connectionDetails, stopifnot(is.character(cdmVersion)) # Warning if check names for determining NA is missing - if (!length(checkNames) == 0) { - naCheckNames <- c("cdmTable", "cdmField", "measureValueCompleteness") - missingNAChecks <- !(naCheckNames %in% checkNames) - if (any(missingNAChecks)) { - missingNACheckNames <- paste(naCheckNames[missingNAChecks], collapse = ", ") - warning(sprintf("Missing check names to calculate the 'Not Applicable' status: %s", missingNACheckNames)) - } + if (length(checkNames) > 0 && !.containsNAchecks(checkNames)) { + warning("Missing check names to calculate the 'Not Applicable' status.") } - # temporary patch to work around vroom 1.6.4 bug readr::local_edition(1) From e09adabd1bc1b4a0b46e77b247ec494982baf8a6 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Thu, 22 Feb 2024 09:32:39 +0100 Subject: [PATCH 08/16] add tests for calculating Not Applicable status --- .../test-calculateNotApplicableStatus.R | 87 +++++++++++++++++++ 1 file changed, 87 insertions(+) create mode 100644 tests/testthat/test-calculateNotApplicableStatus.R diff --git a/tests/testthat/test-calculateNotApplicableStatus.R b/tests/testthat/test-calculateNotApplicableStatus.R new file mode 100644 index 00000000..a70eef48 --- /dev/null +++ b/tests/testthat/test-calculateNotApplicableStatus.R @@ -0,0 +1,87 @@ +library(testthat) + +test_that("Not Applicable status Table Empty", { + outputFolder <- tempfile("dqd_") + on.exit(unlink(outputFolder, recursive = TRUE)) + + results <- executeDqChecks( + connectionDetails = connectionDetailsEunomia, + cdmDatabaseSchema = cdmDatabaseSchemaEunomia, + resultsDatabaseSchema = resultsDatabaseSchemaEunomia, + cdmSourceName = "Eunomia", + checkNames = c("cdmTable", "cdmField", "measureValueCompleteness"), + # Eunomia COST table has misspelled 'REVEUE_CODE_SOURCE_VALUE' + tablesToExclude = c("COST", "CONCEPT", "VOCABULARY", "CONCEPT_ANCESTOR", "CONCEPT_RELATIONSHIP", "CONCEPT_CLASS", "CONCEPT_SYNONYM", "RELATIONSHIP", "DOMAIN"), + outputFolder = outputFolder, + writeToTable = F + ) + + # Assumption that Eunomia has empty device_exposure table + r <- results$CheckResults[results$CheckResults$checkName == "measureValueCompleteness" & + results$CheckResults$tableName == "device_exposure", ] + expect_true(all(r$notApplicable == 1)) +}) + +test_that("measureConditionEraCompleteness Not Applicable if condition_occurrence empty", { + outputFolder <- tempfile("dqd_") + on.exit(unlink(outputFolder, recursive = TRUE)) + + # Remove records from Condition Occurrence + connection <- DatabaseConnector::connect(connectionDetailsEunomia) + DatabaseConnector::executeSql(connection, "CREATE TABLE CONDITION_OCCURRENCE_BACK AS SELECT * FROM CONDITION_OCCURRENCE;") + DatabaseConnector::executeSql(connection, "DELETE FROM CONDITION_OCCURRENCE;") + DatabaseConnector::disconnect(connection) + + results <- executeDqChecks( + connectionDetails = connectionDetailsEunomia, + cdmDatabaseSchema = cdmDatabaseSchemaEunomia, + resultsDatabaseSchema = resultsDatabaseSchemaEunomia, + cdmSourceName = "Eunomia", + checkNames = c("cdmTable", "cdmField", "measureValueCompleteness", "measureConditionEraCompleteness"), + # Eunomia COST table has misspelled 'REVEUE_CODE_SOURCE_VALUE' + tablesToExclude = c("COST", "CONCEPT", "VOCABULARY", "CONCEPT_ANCESTOR", "CONCEPT_RELATIONSHIP", "CONCEPT_CLASS", "CONCEPT_SYNONYM", "RELATIONSHIP", "DOMAIN"), + outputFolder = outputFolder, + writeToTable = F + ) + + # Reinstate Condition Occurrence + connection <- DatabaseConnector::connect(connectionDetailsEunomia) + DatabaseConnector::executeSql(connection, "INSERT INTO CONDITION_OCCURRENCE SELECT * FROM CONDITION_OCCURRENCE_BACK;") + DatabaseConnector::executeSql(connection, "DROP TABLE CONDITION_OCCURRENCE_BACK;") + disconnect(connection) + + r <- results$CheckResults[results$CheckResults$checkName == "measureConditionEraCompleteness", ] + expect_true(r$notApplicable == 1) +}) + +test_that("measureConditionEraCompleteness Fails if condition_era empty", { + outputFolder <- tempfile("dqd_") + on.exit(unlink(outputFolder, recursive = TRUE)) + + # Remove records from Condition Era + connection <- DatabaseConnector::connect(connectionDetailsEunomia) + DatabaseConnector::executeSql(connection, "CREATE TABLE CONDITION_ERA_BACK AS SELECT * FROM CONDITION_ERA;") + DatabaseConnector::executeSql(connection, "DELETE FROM CONDITION_ERA;") + DatabaseConnector::disconnect(connection) + + results <- executeDqChecks( + connectionDetails = connectionDetailsEunomia, + cdmDatabaseSchema = cdmDatabaseSchemaEunomia, + resultsDatabaseSchema = resultsDatabaseSchemaEunomia, + cdmSourceName = "Eunomia", + checkNames = c("cdmTable", "cdmField", "measureValueCompleteness", "measureConditionEraCompleteness"), + # Eunomia COST table has misspelled 'REVEUE_CODE_SOURCE_VALUE' + tablesToExclude = c("COST", "CONCEPT", "VOCABULARY", "CONCEPT_ANCESTOR", "CONCEPT_RELATIONSHIP", "CONCEPT_CLASS", "CONCEPT_SYNONYM", "RELATIONSHIP", "DOMAIN"), + outputFolder = outputFolder, + writeToTable = F + ) + + # Reinstate the Condition Era + connection <- DatabaseConnector::connect(connectionDetailsEunomia) + DatabaseConnector::executeSql(connection, "INSERT INTO CONDITION_ERA SELECT * FROM CONDITION_ERA_BACK;") + DatabaseConnector::executeSql(connection, "DROP TABLE CONDITION_ERA_BACK;") + DatabaseConnector::disconnect(connection) + + r <- results$CheckResults[results$CheckResults$checkName == "measureConditionEraCompleteness", ] + expect_true(r$failed == 1) +}) \ No newline at end of file From 8c3c4adad2db21148e78385fdcd8610b767a625e Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Mon, 13 May 2024 13:06:41 +0200 Subject: [PATCH 09/16] refactoring --- R/calculateNotApplicableStatus.R | 65 ++++++++++--------- .../test-calculateNotApplicableStatus.R | 12 ++-- 2 files changed, 42 insertions(+), 35 deletions(-) diff --git a/R/calculateNotApplicableStatus.R b/R/calculateNotApplicableStatus.R index ace2c329..eeafb203 100644 --- a/R/calculateNotApplicableStatus.R +++ b/R/calculateNotApplicableStatus.R @@ -33,18 +33,48 @@ naCheckNames <- c("cdmTable", "cdmField", "measureValueCompleteness") missingNAChecks <- !(naCheckNames %in% checkNames) if (any(missingNAChecks)) { - missingNACheckNames <- paste(naCheckNames[missingNAChecks], collapse = ", ") return(FALSE) } return(TRUE) } +#' Applies the 'Not Applicable' status to a single check +#' +#' @param x Results from a single check +#' +#' @keywords internal +.applyNotApplicable <- function(x) { + # Errors precede all other statuses + if (x$isError == 1) { + return(0) + } + + # No NA status for cdmTable and cdmField if missing + if (x$checkName == "cdmTable" || x$checkName == "cdmField") { + return(0) + } + + if (any(x$tableIsMissing, x$fieldIsMissing, x$tableIsEmpty, na.rm = TRUE)) { + return(1) + } + + # No NA status for measureValueCompleteness if empty + if (x$checkName == "measureValueCompleteness") { + return(0) + } + + if (any(x$fieldIsEmpty, x$conceptIsMissing, x$conceptAndUnitAreMissing, na.rm = TRUE)) { + return(1) + } + + return(0) +} + #' Determines if check should be notApplicable and the notApplicableReason #' #' @param checkResults A dataframe containing the results of the data quality checks #' #' @keywords internal - .calculateNotApplicableStatus <- function(checkResults) { # Look up missing tables and add variable tableIsMissing to checkResults missingTables <- checkResults %>% @@ -81,7 +111,7 @@ ) %>% dplyr::distinct() - # Look up empty fields and add variable tableIsEmpty to checkResults + # Look up empty fields and add variable fieldIsEmpty to checkResults emptyFields <- checkResults %>% dplyr::filter( .data$checkName == "measureValueCompleteness" @@ -148,7 +178,7 @@ .data$tableIsEmpty ~ sprintf("Table %s is empty.", .data$cdmTableName), .data$fieldIsEmpty ~ sprintf("Field %s.%s is not populated.", .data$cdmTableName, .data$cdmFieldName), .data$conceptIsMissing ~ sprintf("%s=%s is missing from the %s table.", .data$cdmFieldName, .data$conceptId, .data$cdmTableName), - .data$conceptAndUnitAreMissing ~ sprintf("Combination of %s=%s, unitConceptId=%s and VALUE_AS_NUMBER IS NOT NULL is missing from the %s table.", .data$cdmFieldName, .data$conceptId, .data$unitConceptId, .data$cdmTableName) + .data$conceptAndUnitAreMissing ~ sprintf("Combination of %s=%s, unitConceptId=%s and VALUE_AS_NUMBER IS NOT NULL is missing from the %s table.", .data$cdmFieldName, .data$conceptId, .data$unitConceptId, .data$cdmTableName) #nolint ), NA ), @@ -159,30 +189,3 @@ return(checkResults) } - -.applyNotApplicable <- function(x) { - # Errors precede all other statuses - if (x$isError == 1) { - return(0) - } - - # No NA status for cdmTable and cdmField if missing - if (x$checkName == "cdmTable" || x$checkName == "cdmField") { - return(0) - } - - if (any(x$tableIsMissing, x$fieldIsMissing, x$tableIsEmpty, na.rm = TRUE)) { - return(1) - } - - # No NA status for measureValueCompleteness if empty - if (x$checkName == "measureValueCompleteness") { - return(0) - } - - if (any(x$fieldIsEmpty, x$conceptIsMissing, x$conceptAndUnitAreMissing, na.rm = TRUE)) { - return(1) - } - - return(0) -} diff --git a/tests/testthat/test-calculateNotApplicableStatus.R b/tests/testthat/test-calculateNotApplicableStatus.R index a70eef48..6c19decd 100644 --- a/tests/testthat/test-calculateNotApplicableStatus.R +++ b/tests/testthat/test-calculateNotApplicableStatus.R @@ -4,6 +4,11 @@ test_that("Not Applicable status Table Empty", { outputFolder <- tempfile("dqd_") on.exit(unlink(outputFolder, recursive = TRUE)) + # Make sure the device exposure table is empty + connection <- DatabaseConnector::connect(connectionDetailsEunomia) + DatabaseConnector::executeSql(connection, "DELETE FROM DEVICE_EXPOSURE;") + DatabaseConnector::disconnect(connection) + results <- executeDqChecks( connectionDetails = connectionDetailsEunomia, cdmDatabaseSchema = cdmDatabaseSchemaEunomia, @@ -13,10 +18,9 @@ test_that("Not Applicable status Table Empty", { # Eunomia COST table has misspelled 'REVEUE_CODE_SOURCE_VALUE' tablesToExclude = c("COST", "CONCEPT", "VOCABULARY", "CONCEPT_ANCESTOR", "CONCEPT_RELATIONSHIP", "CONCEPT_CLASS", "CONCEPT_SYNONYM", "RELATIONSHIP", "DOMAIN"), outputFolder = outputFolder, - writeToTable = F + writeToTable = FALSE ) - # Assumption that Eunomia has empty device_exposure table r <- results$CheckResults[results$CheckResults$checkName == "measureValueCompleteness" & results$CheckResults$tableName == "device_exposure", ] expect_true(all(r$notApplicable == 1)) @@ -41,7 +45,7 @@ test_that("measureConditionEraCompleteness Not Applicable if condition_occurrenc # Eunomia COST table has misspelled 'REVEUE_CODE_SOURCE_VALUE' tablesToExclude = c("COST", "CONCEPT", "VOCABULARY", "CONCEPT_ANCESTOR", "CONCEPT_RELATIONSHIP", "CONCEPT_CLASS", "CONCEPT_SYNONYM", "RELATIONSHIP", "DOMAIN"), outputFolder = outputFolder, - writeToTable = F + writeToTable = FALSE ) # Reinstate Condition Occurrence @@ -73,7 +77,7 @@ test_that("measureConditionEraCompleteness Fails if condition_era empty", { # Eunomia COST table has misspelled 'REVEUE_CODE_SOURCE_VALUE' tablesToExclude = c("COST", "CONCEPT", "VOCABULARY", "CONCEPT_ANCESTOR", "CONCEPT_RELATIONSHIP", "CONCEPT_CLASS", "CONCEPT_SYNONYM", "RELATIONSHIP", "DOMAIN"), outputFolder = outputFolder, - writeToTable = F + writeToTable = FALSE ) # Reinstate the Condition Era From 96618a4c4dd04522e3064e96951ace01a74ed321 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Sat, 6 Jul 2024 14:28:16 +0200 Subject: [PATCH 10/16] separate connectionDetails for NA Checks --- tests/testthat/setup.R | 3 +++ tests/testthat/test-calculateNotApplicableStatus.R | 14 +++++++------- 2 files changed, 10 insertions(+), 7 deletions(-) diff --git a/tests/testthat/setup.R b/tests/testthat/setup.R index f1aa8636..761fdeab 100644 --- a/tests/testthat/setup.R +++ b/tests/testthat/setup.R @@ -12,3 +12,6 @@ if (Sys.getenv("DONT_DOWNLOAD_JDBC_DRIVERS", "") == "TRUE") { connectionDetailsEunomia <- Eunomia::getEunomiaConnectionDetails() cdmDatabaseSchemaEunomia <- "main" resultsDatabaseSchemaEunomia <- "main" + +# Separate connection details for NA tests, as this requires removing records +connectionDetailsEunomiaNaChecks <- Eunomia::getEunomiaConnectionDetails() \ No newline at end of file diff --git a/tests/testthat/test-calculateNotApplicableStatus.R b/tests/testthat/test-calculateNotApplicableStatus.R index 6c19decd..321dc6cf 100644 --- a/tests/testthat/test-calculateNotApplicableStatus.R +++ b/tests/testthat/test-calculateNotApplicableStatus.R @@ -10,7 +10,7 @@ test_that("Not Applicable status Table Empty", { DatabaseConnector::disconnect(connection) results <- executeDqChecks( - connectionDetails = connectionDetailsEunomia, + connectionDetails = connectionDetailsEunomiaNaChecks, cdmDatabaseSchema = cdmDatabaseSchemaEunomia, resultsDatabaseSchema = resultsDatabaseSchemaEunomia, cdmSourceName = "Eunomia", @@ -31,13 +31,13 @@ test_that("measureConditionEraCompleteness Not Applicable if condition_occurrenc on.exit(unlink(outputFolder, recursive = TRUE)) # Remove records from Condition Occurrence - connection <- DatabaseConnector::connect(connectionDetailsEunomia) + connection <- DatabaseConnector::connect(connectionDetailsEunomiaNaChecks) DatabaseConnector::executeSql(connection, "CREATE TABLE CONDITION_OCCURRENCE_BACK AS SELECT * FROM CONDITION_OCCURRENCE;") DatabaseConnector::executeSql(connection, "DELETE FROM CONDITION_OCCURRENCE;") DatabaseConnector::disconnect(connection) results <- executeDqChecks( - connectionDetails = connectionDetailsEunomia, + connectionDetails = connectionDetailsEunomiaNaChecks, cdmDatabaseSchema = cdmDatabaseSchemaEunomia, resultsDatabaseSchema = resultsDatabaseSchemaEunomia, cdmSourceName = "Eunomia", @@ -49,7 +49,7 @@ test_that("measureConditionEraCompleteness Not Applicable if condition_occurrenc ) # Reinstate Condition Occurrence - connection <- DatabaseConnector::connect(connectionDetailsEunomia) + connection <- DatabaseConnector::connect(connectionDetailsEunomiaNaChecks) DatabaseConnector::executeSql(connection, "INSERT INTO CONDITION_OCCURRENCE SELECT * FROM CONDITION_OCCURRENCE_BACK;") DatabaseConnector::executeSql(connection, "DROP TABLE CONDITION_OCCURRENCE_BACK;") disconnect(connection) @@ -63,13 +63,13 @@ test_that("measureConditionEraCompleteness Fails if condition_era empty", { on.exit(unlink(outputFolder, recursive = TRUE)) # Remove records from Condition Era - connection <- DatabaseConnector::connect(connectionDetailsEunomia) + connection <- DatabaseConnector::connect(connectionDetailsEunomiaNaChecks) DatabaseConnector::executeSql(connection, "CREATE TABLE CONDITION_ERA_BACK AS SELECT * FROM CONDITION_ERA;") DatabaseConnector::executeSql(connection, "DELETE FROM CONDITION_ERA;") DatabaseConnector::disconnect(connection) results <- executeDqChecks( - connectionDetails = connectionDetailsEunomia, + connectionDetails = connectionDetailsEunomiaNaChecks, cdmDatabaseSchema = cdmDatabaseSchemaEunomia, resultsDatabaseSchema = resultsDatabaseSchemaEunomia, cdmSourceName = "Eunomia", @@ -81,7 +81,7 @@ test_that("measureConditionEraCompleteness Fails if condition_era empty", { ) # Reinstate the Condition Era - connection <- DatabaseConnector::connect(connectionDetailsEunomia) + connection <- DatabaseConnector::connect(connectionDetailsEunomiaNaChecks) DatabaseConnector::executeSql(connection, "INSERT INTO CONDITION_ERA SELECT * FROM CONDITION_ERA_BACK;") DatabaseConnector::executeSql(connection, "DROP TABLE CONDITION_ERA_BACK;") DatabaseConnector::disconnect(connection) From bafb1e42d7f151dfab212d06e778e12e91600c58 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Thu, 11 Jul 2024 14:48:53 +0200 Subject: [PATCH 11/16] fix na status test --- tests/testthat/test-calculateNotApplicableStatus.R | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/tests/testthat/test-calculateNotApplicableStatus.R b/tests/testthat/test-calculateNotApplicableStatus.R index 321dc6cf..0f3bc64c 100644 --- a/tests/testthat/test-calculateNotApplicableStatus.R +++ b/tests/testthat/test-calculateNotApplicableStatus.R @@ -5,7 +5,7 @@ test_that("Not Applicable status Table Empty", { on.exit(unlink(outputFolder, recursive = TRUE)) # Make sure the device exposure table is empty - connection <- DatabaseConnector::connect(connectionDetailsEunomia) + connection <- DatabaseConnector::connect(connectionDetailsEunomiaNaChecks) DatabaseConnector::executeSql(connection, "DELETE FROM DEVICE_EXPOSURE;") DatabaseConnector::disconnect(connection) From 9bdd11382988e7ef0077204cb043ca50f96283c0 Mon Sep 17 00:00:00 2001 From: Maxim Moinat Date: Sun, 7 Jul 2024 23:21:34 +0200 Subject: [PATCH 12/16] consistent use of % for threshold values --- vignettes/checks/isStandardValidConcept.Rmd | 2 +- vignettes/checks/measurePersonCompleteness.Rmd | 4 +++- vignettes/checks/measureValueCompleteness.Rmd | 6 ++++-- vignettes/checks/plausibleValueHigh.Rmd | 2 +- vignettes/checks/plausibleValueLow.Rmd | 2 +- vignettes/checks/sourceConceptRecordCompleteness.Rmd | 4 ++-- vignettes/checks/sourceValueCompleteness.Rmd | 4 ++-- vignettes/checks/standardConceptRecordCompleteness.Rmd | 6 +++--- vignettes/checks/withinVisitDates.Rmd | 4 +++- 9 files changed, 20 insertions(+), 14 deletions(-) diff --git a/vignettes/checks/isStandardValidConcept.Rmd b/vignettes/checks/isStandardValidConcept.Rmd index d1001aea..75226264 100644 --- a/vignettes/checks/isStandardValidConcept.Rmd +++ b/vignettes/checks/isStandardValidConcept.Rmd @@ -27,7 +27,7 @@ The number and percent of records that do not have a standard, valid concept in - *Denominator*: The total number of rows in the table. - *Related CDM Convention(s)*: All `X_concept_id` columns should contain a standard, valid concept, or 0: https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Mapping. - *CDM Fields/Tables*: All tables with an `X_concept_id` column, and all `X_concept_id` columns in those tables. -- *Default Threshold Value*: 0 +- *Default Threshold Value*: 0% ## User Guidance diff --git a/vignettes/checks/measurePersonCompleteness.Rmd b/vignettes/checks/measurePersonCompleteness.Rmd index c1a346d0..402fe48f 100644 --- a/vignettes/checks/measurePersonCompleteness.Rmd +++ b/vignettes/checks/measurePersonCompleteness.Rmd @@ -27,7 +27,9 @@ The number and percent of persons in the CDM that do not have at least one recor - *Denominator*: The total number of persons in the `PERSON` table. - *Related CDM Convention(s)*: Each Person needs to have at least one `OBSERVATION_PERIOD` record. Otherwise, CDM conventions do not dictate any rules for person completeness. - *CDM Fields/Tables*: By default, this check runs on all tables with a foreign key to the `PERSON` table. -- *Default Threshold Value*: Set to 95 or 100 for most tables but 0 for `OBSERVATION_PERIOD` +- *Default Threshold Value*: + - 0% for `OBSERVATION_PERIOD` + - 95% or 100% for other tables ## User Guidance diff --git a/vignettes/checks/measureValueCompleteness.Rmd b/vignettes/checks/measureValueCompleteness.Rmd index 045837a5..1ac6771c 100644 --- a/vignettes/checks/measureValueCompleteness.Rmd +++ b/vignettes/checks/measureValueCompleteness.Rmd @@ -27,7 +27,9 @@ The number and percent of records with a NULL value in the @cdmFieldName of the - *Denominator*: The total number of rows in the table. - *Related CDM Convention(s)*: None. This check should be used to check local expectations about completeness of a field given characteristics of the source data. - *CDM Fields/Tables*: All fields in all event tables. -- *Default Threshold Value*: 0 for required fields; 100 for all others +- *Default Threshold Value*: + - 0% for required fields + - 100% for all others ## User Guidance @@ -57,7 +59,7 @@ ETL developers have 2 main options for the use of this check on non-required fie Unexpectedly missing values should be investigated for a potential root cause in the ETL. For expected missingness, rows that violate this check in non-required fields are acceptable but should be clearly communicated to data users so that they can know when and when not to expect data to be present in each field. To avoid confusion for users, however, thresholds should be modified to avoid check failures at expected levels. ### Data Users -This check informs you of the level of missing data in each column of the CDM. If data is missing in a required column, see the isRequired documentation for more information. +This check informs you of the level of missing data in each column of the CDM. If data is missing in a required column, see the `isRequired` documentation for more information. The interpretation of a check failure on a non-required column will depend on the context. In some cases, the threshold for this check will have been very deliberately set, and any failure should be cause for concern unless justified and explained by your ETL provider. In other cases, even if the check fails it may not be worrisome if the check result is in line with your expectations given the source of the data. When in doubt, utilize the inspection query above to ensure you can explain the missing values. diff --git a/vignettes/checks/plausibleValueHigh.Rmd b/vignettes/checks/plausibleValueHigh.Rmd index b6c810dc..1c02484a 100644 --- a/vignettes/checks/plausibleValueHigh.Rmd +++ b/vignettes/checks/plausibleValueHigh.Rmd @@ -34,7 +34,7 @@ The number and percent of records with a value in the @cdmFieldName field of the - `DRUG_EXPOSURE.refills` (compared to 24) - `DRUG_EXPOSURE.days_supply` (compared to 365) - `DRUG_EXPOSURE.quantity` (compared to 1095) -- *Default Threshold Value*: 1 +- *Default Threshold Value*: 1% ## User Guidance diff --git a/vignettes/checks/plausibleValueLow.Rmd b/vignettes/checks/plausibleValueLow.Rmd index 455b7334..e17d9d23 100644 --- a/vignettes/checks/plausibleValueLow.Rmd +++ b/vignettes/checks/plausibleValueLow.Rmd @@ -39,7 +39,7 @@ The number and percent of records with a value in the @cdmFieldName field of the - `DEVICE_EXPOSURE.quantity`, `SPECIMEN.quantity`, `PROCEDURE_OCCURRENCE.quantity` (compared to 1) - `DRUG_ERA.dose_value`, `DRUG_ERA.gap_days` (compared to 0) - `DRUG_ERA.drug_exposure_count` (compared to 1) -- *Default Threshold Value*: 1 +- *Default Threshold Value*: 1% ## User Guidance diff --git a/vignettes/checks/sourceConceptRecordCompleteness.Rmd b/vignettes/checks/sourceConceptRecordCompleteness.Rmd index 0827a1ab..25ac71aa 100644 --- a/vignettes/checks/sourceConceptRecordCompleteness.Rmd +++ b/vignettes/checks/sourceConceptRecordCompleteness.Rmd @@ -28,8 +28,8 @@ The number and percent of records with a value of 0 in the source concept field - *Related CDM Convention(s)*: [Source concept mapping](https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Fields) - *CDM Fields/Tables*: All source concept ID (`X_source_concept_id`) columns in all event tables. - *Default Threshold Value*: - - 10 for primary source concept ID columns in condition, drug, measurement, procedure, device, and observation tables - - 100 for all other source concept ID columns + - 10% for source concept ID columns in condition, drug, measurement, procedure, device, and observation tables + - 100% for all other source concept ID columns ## User Guidance diff --git a/vignettes/checks/sourceValueCompleteness.Rmd b/vignettes/checks/sourceValueCompleteness.Rmd index 9e39cfba..06c4f1ca 100644 --- a/vignettes/checks/sourceValueCompleteness.Rmd +++ b/vignettes/checks/sourceValueCompleteness.Rmd @@ -28,8 +28,8 @@ The number and percent of distinct source values in the @cdmFieldName field of t - *Related CDM Convention(s)*: The OMOP Common Data Model specifies that codes that are present in a native database should be mapped to standard concepts using either the intrinsic mappings defined in the standard vocabularies or extrinsic mappings defined by the data owner or ETL development team. Note also that variations of this check logic are also used in the [EHDEN CDM Inspection Report](https://github.com/EHDEN/CdmInspection) package, as well as the [AresIndexer](https://github.com/OHDSI/AresIndexer) package for generating indices of unmapped codes. - *CDM Fields/Tables*: Runs on all event tables that have `X_source_value` fields. - *Default Threshold Value*: - - 10 for critical event tables' `X_source_value` fields (condition, measurement, procedure, drug, visit) - - 100 for all other fields - to be adjusted based on source-specific expectations + - 10% for `_source_value` fields in condition, measurement, procedure, drug, visit. + - 100% for all other fields ## User Guidance diff --git a/vignettes/checks/standardConceptRecordCompleteness.Rmd b/vignettes/checks/standardConceptRecordCompleteness.Rmd index e745a21f..3f037c8a 100644 --- a/vignettes/checks/standardConceptRecordCompleteness.Rmd +++ b/vignettes/checks/standardConceptRecordCompleteness.Rmd @@ -28,9 +28,9 @@ The number and percent of records with a value of 0 in the standard concept fiel - *Related CDM Convention(s)*: [Standard concept mapping](https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Fields) - *CDM Fields/Tables*: All standard concept ID (`X_concept_id`) columns in all event tables. - *Default Threshold Value*: - - 0 for type concept fields and standard concept fields in era tables - - 5 for most standard concept fields in clinical event tables - - 100 for fields more susceptible to specific ETL implementation context + - 0% for type concept fields and standard concept fields in era tables + - 5% for most standard concept fields in clinical event tables + - 100% for fields more susceptible to specific ETL implementation context ## User Guidance diff --git a/vignettes/checks/withinVisitDates.Rmd b/vignettes/checks/withinVisitDates.Rmd index 6bd7aaa8..f15d3a4c 100644 --- a/vignettes/checks/withinVisitDates.Rmd +++ b/vignettes/checks/withinVisitDates.Rmd @@ -27,7 +27,9 @@ The number and percent of records that occur one week before the corresponding ` - *Denominator*: The total number of rows in the table with a corresponding visit (linked through `visit_occurrence_id`) - *Related CDM Convention(s)*: There is no explicit convention tied to this check. However, the CDM documentation describes the `visit_occurrence_id` foreign key in the event tables as “The visit during which the occurred.” The underlying assumption is that if a record is tied to a visit, then the date of the record should fall in some reasonable time period around the visit dates. This gives a week of leeway on either side for physician notes or other activities related to a visit to be recorded. - *CDM Fields/Tables*: This check runs on all event tables: `CONDITION_OCCURRENCE`, `PROCEDURE_OCCURRENCE`, `DRUG_EXPOSURE`, `DEVICE_EXPOSURE`, `MEASUREMENT`, `NOTE`, `OBSERVATION`, and `VISIT_DETAIL`. It will check either the `X_date` or `X_start_date` fields for alignment with corresponding `VISIT_OCCURRENCE` dates by linking on the `visit_occurrence_id`. (**Note:** For VISIT_DETAIL it will check both the visit_detail_start_date and visit_detail_end_date. The default threshold for these two checks is 1%.) -- *Default Threshold Value*: 1% for `VISIT_DETAIL`, 5% for all other tables +- *Default Threshold Value*: + - 1% for `VISIT_DETAIL` + - 5% for all other tables ## User Guidance From 5f66b436f4bb856d1c310dab39f74675ccc07bf6 Mon Sep 17 00:00:00 2001 From: Katy Sadowski Date: Tue, 9 Jul 2024 08:55:55 -0400 Subject: [PATCH 13/16] Apply suggestions from Maxim's code review Co-authored-by: Maxim Moinat --- vignettes/checks/isStandardValidConcept.Rmd | 4 ++-- vignettes/checks/measurePersonCompleteness.Rmd | 3 +-- vignettes/checks/measureValueCompleteness.Rmd | 4 ++-- vignettes/checks/sourceConceptRecordCompleteness.Rmd | 2 +- vignettes/checks/standardConceptRecordCompleteness.Rmd | 2 +- 5 files changed, 7 insertions(+), 8 deletions(-) diff --git a/vignettes/checks/isStandardValidConcept.Rmd b/vignettes/checks/isStandardValidConcept.Rmd index 75226264..bce6180c 100644 --- a/vignettes/checks/isStandardValidConcept.Rmd +++ b/vignettes/checks/isStandardValidConcept.Rmd @@ -26,7 +26,7 @@ The number and percent of records that do not have a standard, valid concept in - *Numerator*: The number of rows with an `X_concept_id` that exists in `CONCEPT.concept_id` but does not equal zero, and has `CONCEPT.standard_concept` != ‘S’ or non-NULL `CONCEPT.invalid_reason`. - *Denominator*: The total number of rows in the table. - *Related CDM Convention(s)*: All `X_concept_id` columns should contain a standard, valid concept, or 0: https://ohdsi.github.io/CommonDataModel/dataModelConventions.html#Mapping. -- *CDM Fields/Tables*: All tables with an `X_concept_id` column, and all `X_concept_id` columns in those tables. +- *CDM Fields/Tables*: All standard concept ID (`X_concept_id`) columns in all event tables. - *Default Threshold Value*: 0% @@ -52,7 +52,7 @@ WHERE co.concept_id != 0 You may build upon this query by joining the relevant `X_concept_id` and `X_source_concept_id` columns to the concept table and inspecting their names and vocabularies. If the `X_source_concept_id` correctly represents the source code in `X_source_value`, the fix will be a matter of ensuring your ETL is correctly using the concept_relationship table to map the source concept ID to a standard concept via the ‘Maps to’ relationship. If you are not populating the `X_source_concept_id` column and/or are using an intermediate concept mapping table, you may need to inspect the mappings in your mapper table to ensure they’ve been generated correctly using the ‘Maps to’ relationship for your CDM’s vocabulary version. ### Data Users -This check failure means that the failing rows will not be picked up in a standard OHDSI analysis. It is highly recommended to work with your ETL team or data provider, if possible, to resolve this issue. +This check failure means that the failing rows will not be picked up in a standard OHDSI analysis. Especially when participating in network research, where only standard concepts are used, this might result in invalid results. It is highly recommended to work with your ETL team or data provider, if possible, to resolve this issue. However, you may work around it at your own risk by determining whether or not the affected rows are relevant for your analysis. Here’s an example query you could run to inspect failing rows in the condition_occurrence table: diff --git a/vignettes/checks/measurePersonCompleteness.Rmd b/vignettes/checks/measurePersonCompleteness.Rmd index 402fe48f..7c7e9735 100644 --- a/vignettes/checks/measurePersonCompleteness.Rmd +++ b/vignettes/checks/measurePersonCompleteness.Rmd @@ -14,7 +14,7 @@ output: **Context**: Validation\ **Category**: Completeness\ **Subcategory**: \ -**Severity**: CDM convention ⚠ Characterization ✔ \ +**Severity**: CDM convention ⚠ (for observation period), Characterization ✔ \ (for all other tables) ## Description @@ -64,7 +64,6 @@ Action on persons missing records in other clinical event tables will depend on If more persons than expected are missing data in a given table, run the violated rows SQL snippet to retrieve these persons’ person_ids, and inspect these persons’ other clinical event data in the CDM for trends. You may also use `person_source_value` to trace back to these persons’ source data to identify source data records potentially missed by the ETL. -Note that in some cases, the failure threshold for this check may need to be adjusted according to completeness expectations for a given data source. ### Data Users diff --git a/vignettes/checks/measureValueCompleteness.Rmd b/vignettes/checks/measureValueCompleteness.Rmd index 1ac6771c..1983d3dd 100644 --- a/vignettes/checks/measureValueCompleteness.Rmd +++ b/vignettes/checks/measureValueCompleteness.Rmd @@ -49,11 +49,11 @@ WHERE cdmTable.@cdmFieldName IS NULL ``` ### ETL Developers -Failures of this check on fields required in the CDM specification are redundant with failures of `isRequired`. See [isRequired documentation](isRequired.html) for more information. +Failures of this check on required fields are redundant with failures of `isRequired`. See [isRequired documentation](isRequired.html) for more information. ETL developers have 2 main options for the use of this check on non-required fields: -- The check threshold may be set to 100% for non-required fields such that the check will never fail. The check result can be used simply to understand completeness for these fields +- The check threshold may be left on 100% for non-required fields such that the check will never fail. The check result can be used simply to understand completeness for these fields - The check threshold may be set to an appropriate value corresponding to completeness expectations for each field given what’s available in the source data. The check may be disabled for fields known not to exist in the source data. Other fields may be set to whichever threshold is deemed worthy of investigation Unexpectedly missing values should be investigated for a potential root cause in the ETL. For expected missingness, rows that violate this check in non-required fields are acceptable but should be clearly communicated to data users so that they can know when and when not to expect data to be present in each field. To avoid confusion for users, however, thresholds should be modified to avoid check failures at expected levels. diff --git a/vignettes/checks/sourceConceptRecordCompleteness.Rmd b/vignettes/checks/sourceConceptRecordCompleteness.Rmd index 25ac71aa..51627af4 100644 --- a/vignettes/checks/sourceConceptRecordCompleteness.Rmd +++ b/vignettes/checks/sourceConceptRecordCompleteness.Rmd @@ -67,6 +67,6 @@ If source values return legitimate matches on concept_code, it’s possible that If source values do NOT return matches on concept_code and you are NOT handling concept mapping locally for a non-OMOP source vocabulary, then you likely have a malformed source code or one that does not exist in the OMOP vocabulary. Please see the documentation in the [standardConceptRecordCompleteness](standardConceptRecordCompleteness.html) page for instructions on how to handle this scenario. ### Data Users -Since most standard OHDSI analytic workflows rely on the standard concept field and not the source concept field, failures of this check will not necessarily impact your analysis. However, if your analysis depends on understanding source coding practices or on codes you know may not be fully mapped to OMOP standard concepts, then this will be a critical check failure to understand. +Since most standard OHDSI analytic workflows rely on the standard concept field and not the source concept field, failures of this check will not necessarily impact your analysis. However, having the source concept will give you a better understanding of the provenance of the code and highlight potential issues where meaning is lost due to mapping to a standard concept. Utilize the investigation queries above to understand the scope and impact of the mapping failures on your specific analytic use case. If none of the affected codes seem to be relevant for your analysis, it may be acceptable to ignore the failure. However, since it is not always possible to understand exactly what a given source value represents, you should proceed with caution and confirm any findings with your ETL provider if possible. diff --git a/vignettes/checks/standardConceptRecordCompleteness.Rmd b/vignettes/checks/standardConceptRecordCompleteness.Rmd index 3f037c8a..80f77bf7 100644 --- a/vignettes/checks/standardConceptRecordCompleteness.Rmd +++ b/vignettes/checks/standardConceptRecordCompleteness.Rmd @@ -87,7 +87,7 @@ WHERE concept_code = - If no result is returned, consider whether the source value may be a malformed version of a legitimate code (for example, sometimes ICD10-CM codes do not contain a “.” in source data). If you can confirm that the code is properly formatted, then you have a source code which does not exist in the OMOP vocabulary. If you believe the code was omitted from the vocabulary in error, please report this issue to the vocabulary team. Otherwise, the short-term course of action will be to generate a mapping for the code locally and implement the mapping in your ETL. For the longer term, the vocabulary team provides a workflow to submit new vocabularies for inclusion in the OMOP vocabularies - Note that in some cases, you will find that no standard concept exists to which to map your source code. In this case, the standard concept ID field should be left as 0 in the short term; in the longer term please work with the vocabulary team to address this gap as recommended above - - Finally, if the investigation query returns no source value, you must trace the relevant record(s) back to their source and confirm if the missing value is expected. If not, identify and fix the related issue in your ETL. If the record legitimately has no value/code in the source data, then the standard concept ID may be left as 0. However, in some cases these “code-less” records represent junk data which should be filtered out in the ETL. The proper approach will be context-dependent + - Finally, if the investigation query returns no source value, you must trace the relevant record(s) back to their source and confirm if the missing source value is expected. If not, identify and fix the related issue in your ETL. If the record legitimately has no value/code in the source data, then the standard concept ID may be left as 0. However, in some cases these “code-less” records represent junk data which should be filtered out in the ETL. The proper approach will be context-dependent - Note in the special case of unitless measurements/observations, the unit_concept_id field should NOT be coded as 0 and rather should be left NULL (the unit_concept_id fields are optional in the CDM spec) It is important to note that records with a 0 standard concept ID field will be unusable in standard OHDSI analyses and thus should only be preserved if there is truly no standard concept ID for a given record. Depending on the significance of the records in question, one should consider removing them from the dataset; however, this choice will depend on a variety of context-specific factors and should be made carefully. Either way, the presence/absence of these unmappable records and an explanation for why they could not be mapped should be clearly documented in the ETL documentation. From 47a0ca42fdf0d71b0b2c7e83f982f221a9e53701 Mon Sep 17 00:00:00 2001 From: Katy Sadowski Date: Thu, 11 Jul 2024 21:47:20 -0400 Subject: [PATCH 14/16] updates from PR review --- docs/articles/AddNewCheck.html | 2 +- docs/articles/CheckStatusDefinitions.html | 2 +- docs/articles/CheckTypeDescriptions.html | 2 +- docs/articles/DataQualityDashboard.html | 2 +- docs/articles/DqdForCohorts.html | 2 +- docs/articles/SqlOnly.html | 2 +- docs/articles/Thresholds.html | 2 +- docs/articles/checkIndex.html | 2 +- docs/articles/checks/cdmDatatype.html | 2 +- docs/articles/checks/cdmField.html | 2 +- docs/articles/checks/cdmTable.html | 2 +- docs/articles/checks/fkClass.html | 2 +- docs/articles/checks/fkDomain.html | 2 +- docs/articles/checks/isForeignKey.html | 8 +-- docs/articles/checks/isPrimaryKey.html | 2 +- docs/articles/checks/isRequired.html | 8 +-- .../checks/isStandardValidConcept.html | 64 ++++++++++--------- .../measureConditionEraCompleteness.html | 2 +- .../checks/measurePersonCompleteness.html | 24 ++++--- .../checks/measureValueCompleteness.html | 32 +++++----- docs/articles/checks/plausibleAfterBirth.html | 2 +- .../articles/checks/plausibleBeforeDeath.html | 2 +- .../checks/plausibleGenderUseDescendants.html | 2 +- .../checks/plausibleStartBeforeEnd.html | 2 +- .../checks/plausibleTemporalAfter.html | 2 +- .../checks/plausibleUnitConceptIds.html | 2 +- docs/articles/checks/plausibleValueHigh.html | 4 +- docs/articles/checks/plausibleValueLow.html | 4 +- .../sourceConceptRecordCompleteness.html | 53 +++++++-------- .../checks/sourceValueCompleteness.html | 17 +++-- .../standardConceptRecordCompleteness.html | 34 +++++----- docs/articles/checks/withinVisitDates.html | 11 +++- docs/pkgdown.yml | 2 +- vignettes/checks/isForeignKey.Rmd | 6 +- vignettes/checks/isRequired.Rmd | 2 +- vignettes/checks/isStandardValidConcept.Rmd | 15 +++-- .../checks/measurePersonCompleteness.Rmd | 4 +- vignettes/checks/measureValueCompleteness.Rmd | 2 +- .../sourceConceptRecordCompleteness.Rmd | 19 +++--- vignettes/checks/sourceValueCompleteness.Rmd | 6 +- .../standardConceptRecordCompleteness.Rmd | 12 ++-- 41 files changed, 193 insertions(+), 176 deletions(-) diff --git a/docs/articles/AddNewCheck.html b/docs/articles/AddNewCheck.html index bc027dea..0e7ab7f1 100644 --- a/docs/articles/AddNewCheck.html +++ b/docs/articles/AddNewCheck.html @@ -180,7 +180,7 @@

Add a New Data Quality Check

Don Torok

-

2024-06-29

+

2024-07-11

Source: vignettes/AddNewCheck.rmd diff --git a/docs/articles/CheckStatusDefinitions.html b/docs/articles/CheckStatusDefinitions.html index 919cfebc..fe8df052 100644 --- a/docs/articles/CheckStatusDefinitions.html +++ b/docs/articles/CheckStatusDefinitions.html @@ -181,7 +181,7 @@

Check Status Definitions

Dmitry Ilyn, Maxim Moinat

-

2024-06-29

+

2024-07-11

Source: vignettes/CheckStatusDefinitions.rmd diff --git a/docs/articles/CheckTypeDescriptions.html b/docs/articles/CheckTypeDescriptions.html index 6e89315a..a58ea494 100644 --- a/docs/articles/CheckTypeDescriptions.html +++ b/docs/articles/CheckTypeDescriptions.html @@ -181,7 +181,7 @@

Data Quality Check Type Definitions

Clair Blacketer

-

2024-06-29

+

2024-07-11

Source: vignettes/CheckTypeDescriptions.rmd diff --git a/docs/articles/DataQualityDashboard.html b/docs/articles/DataQualityDashboard.html index f1cf33d8..d22f577a 100644 --- a/docs/articles/DataQualityDashboard.html +++ b/docs/articles/DataQualityDashboard.html @@ -181,7 +181,7 @@

Getting Started

Clair Blacketer

-

2024-06-29

+

2024-07-11

Source: vignettes/DataQualityDashboard.rmd diff --git a/docs/articles/DqdForCohorts.html b/docs/articles/DqdForCohorts.html index 8db9ac2c..d9530035 100644 --- a/docs/articles/DqdForCohorts.html +++ b/docs/articles/DqdForCohorts.html @@ -181,7 +181,7 @@

Running the DQD on a Cohort

Clair Blacketer

-

2024-06-29

+

2024-07-11

Source: vignettes/DqdForCohorts.rmd diff --git a/docs/articles/SqlOnly.html b/docs/articles/SqlOnly.html index 1c4a7d4c..71653b00 100644 --- a/docs/articles/SqlOnly.html +++ b/docs/articles/SqlOnly.html @@ -181,7 +181,7 @@

Running the DQD in SqlOnly mode

Maxim Moinat

-

2024-06-29

+

2024-07-11

Source: vignettes/SqlOnly.rmd diff --git a/docs/articles/Thresholds.html b/docs/articles/Thresholds.html index 1d94bcf7..7bcce30c 100644 --- a/docs/articles/Thresholds.html +++ b/docs/articles/Thresholds.html @@ -181,7 +181,7 @@

Failure Thresholds and How to Change Them

Clair Blacketer

-

2024-06-29

+

2024-07-11

Source: vignettes/Thresholds.rmd diff --git a/docs/articles/checkIndex.html b/docs/articles/checkIndex.html index 289565c4..9289ad6a 100644 --- a/docs/articles/checkIndex.html +++ b/docs/articles/checkIndex.html @@ -181,7 +181,7 @@

Index

Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checkIndex.Rmd diff --git a/docs/articles/checks/cdmDatatype.html b/docs/articles/checks/cdmDatatype.html index c0b9d6c7..bd765aeb 100644 --- a/docs/articles/checks/cdmDatatype.html +++ b/docs/articles/checks/cdmDatatype.html @@ -181,7 +181,7 @@

cdmDatatype

Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/cdmDatatype.Rmd diff --git a/docs/articles/checks/cdmField.html b/docs/articles/checks/cdmField.html index 99406a82..3464b260 100644 --- a/docs/articles/checks/cdmField.html +++ b/docs/articles/checks/cdmField.html @@ -181,7 +181,7 @@

cdmField

Heidi Schmidt, Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/cdmField.Rmd diff --git a/docs/articles/checks/cdmTable.html b/docs/articles/checks/cdmTable.html index 3150fd31..03d2cd40 100644 --- a/docs/articles/checks/cdmTable.html +++ b/docs/articles/checks/cdmTable.html @@ -181,7 +181,7 @@

cdmTable

John Gresh, Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/cdmTable.Rmd diff --git a/docs/articles/checks/fkClass.html b/docs/articles/checks/fkClass.html index 2904a2e3..bbfbd439 100644 --- a/docs/articles/checks/fkClass.html +++ b/docs/articles/checks/fkClass.html @@ -181,7 +181,7 @@

fkClass

Clair Blacketer, Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/fkClass.Rmd diff --git a/docs/articles/checks/fkDomain.html b/docs/articles/checks/fkDomain.html index d90b3fb0..bc151de1 100644 --- a/docs/articles/checks/fkDomain.html +++ b/docs/articles/checks/fkDomain.html @@ -181,7 +181,7 @@

fkDomain

Clair Blacketer, Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/fkDomain.Rmd diff --git a/docs/articles/checks/isForeignKey.html b/docs/articles/checks/isForeignKey.html index 3983b9fa..49b25eeb 100644 --- a/docs/articles/checks/isForeignKey.html +++ b/docs/articles/checks/isForeignKey.html @@ -181,7 +181,7 @@

isForeignKey

Dmytry Dymshyts, Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/isForeignKey.Rmd @@ -246,7 +246,7 @@

User GuidanceViolated rows query
-- @cdmTableName.@cdmFieldName is the x_concept_id or x_source_concept_id field in a CDM table
--- Inspect the contents of the x_source_value field to investigate the source of the error
+
-- @cdmTableName.@cdmFieldName is the _concept_id or _source_concept_id field in a CDM table
+-- Inspect the contents of the _source_value field to investigate the source of the error
 
 SELECT 
   '@cdmTableName.@cdmFieldName' AS violating_field,  
diff --git a/docs/articles/checks/isPrimaryKey.html b/docs/articles/checks/isPrimaryKey.html
index 8fdef03a..cc5d3275 100644
--- a/docs/articles/checks/isPrimaryKey.html
+++ b/docs/articles/checks/isPrimaryKey.html
@@ -181,7 +181,7 @@ 
isPrimaryKey

                         
John Gresh,
 Katy Sadowski

             
-            
2024-06-29

+            
2024-07-11

       
       Source: vignettes/checks/isPrimaryKey.Rmd
       
diff --git a/docs/articles/checks/isRequired.html b/docs/articles/checks/isRequired.html
index 994448e6..3da6dfaa 100644
--- a/docs/articles/checks/isRequired.html
+++ b/docs/articles/checks/isRequired.html
@@ -181,7 +181,7 @@ 
isRequired

                         
Katy
 Sadowski

             
-            
2024-06-29

+            
2024-07-11

       
       Source: vignettes/checks/isRequired.Rmd
       
@@ -253,9 +253,9 @@ 
ETL DevelopersFill in the missing values:
 
 

 

@@ -240,36 +239,43 @@ 
ETL DevelopersX_concept_id
-column with 0. See the Book of OHDSI for additional guidance on the
-concept mapping process: https://ohdsi.github.io/TheBookOfOhdsi/ExtractTransformLoad.html#step-2-create-the-code-mappings

+for a source code, you MUST populate its _concept_id column
+with 0. See the Book of OHDSI for additional guidance on the concept
+mapping process: https://ohdsi.github.io/TheBookOfOhdsi/ExtractTransformLoad.html#step-2-create-the-code-mappings

 
You may inspect the failing rows using the following SQL:

 
SELECT  
   '@cdmTableName.@cdmFieldName' AS violating_field,  
-  cdmTable.*  
-FROM @schema.@cdmTableName cdmTable 
-  JOIN @vocabDatabaseSchema.concept co ON cdmTable.@cdmFieldName = co.concept_id 
-WHERE co.concept_id != 0  
-  AND (co.standard_concept != 'S' OR co.invalid_reason IS NOT NULL) 

-
You may build upon this query by joining the relevant
-X_concept_id and X_source_concept_id columns
-to the concept table and inspecting their names and vocabularies. If the
-X_source_concept_id correctly represents the source code in
-X_source_value, the fix will be a matter of ensuring your
-ETL is correctly using the concept_relationship table to map the source
-concept ID to a standard concept via the ‘Maps to’ relationship. If you
-are not populating the X_source_concept_id column and/or
-are using an intermediate concept mapping table, you may need to inspect
-the mappings in your mapper table to ensure they’ve been generated
-correctly using the ‘Maps to’ relationship for your CDM’s vocabulary
-version.

+  cdmTable.*,
+  co.*
+FROM @schema.@cdmTableName cdmTable 
+  JOIN @vocabDatabaseSchema.concept co ON cdmTable.@cdmFieldName = co.concept_id 
+WHERE co.concept_id != 0  
+  AND (co.standard_concept != 'S' OR co.invalid_reason IS NOT NULL)
+

You may build upon this query by joining the +_source_concept_id column to the concept table and +inspecting the source concepts from which the failing non-standard +concepts were mapped. If the _source_concept_id correctly +represents the source code in _source_value, the fix will +be a matter of ensuring your ETL is correctly using the +concept_relationship table to map the source concept ID to a standard +concept via the ‘Maps to’ relationship. If you are not populating the +_source_concept_id column and/or are using an intermediate +concept mapping table, you may need to inspect the mappings in your +mapper table to ensure they’ve been generated correctly using the ‘Maps +to’ relationship for your CDM’s vocabulary version.

+

Also note that when updating the OMOP vocabularies, previously +standard concepts could have been become non-standard and need +remapping. Often this remapping can be done programatically, by +following the ‘Maps to’ relationship to the new standard concept.

Data Users

This check failure means that the failing rows will not be picked up -in a standard OHDSI analysis. It is highly recommended to work with your -ETL team or data provider, if possible, to resolve this issue.

+in a standard OHDSI analysis. Especially when participating in network +research, where only standard concepts are used, this might result in +invalid results. It is highly recommended to work with your ETL team or +data provider, if possible, to resolve this issue.

However, you may work around it at your own risk by determining whether or not the affected rows are relevant for your analysis. Here’s an example query you could run to inspect failing rows in the diff --git a/docs/articles/checks/measureConditionEraCompleteness.html b/docs/articles/checks/measureConditionEraCompleteness.html index e8ce2a9f..2689e33c 100644 --- a/docs/articles/checks/measureConditionEraCompleteness.html +++ b/docs/articles/checks/measureConditionEraCompleteness.html @@ -180,7 +180,7 @@

measureConditionEraCompleteness

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/measureConditionEraCompleteness.Rmd diff --git a/docs/articles/checks/measurePersonCompleteness.html b/docs/articles/checks/measurePersonCompleteness.html index 4e61a71d..3d68ce34 100644 --- a/docs/articles/checks/measurePersonCompleteness.html +++ b/docs/articles/checks/measurePersonCompleteness.html @@ -181,7 +181,7 @@

measurePersonCompleteness

Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/measurePersonCompleteness.Rmd @@ -193,7 +193,8 @@

2024-06-29

Summary

-

Level: TABLE
Context: Validation
Category: Completeness
Subcategory:
Severity: CDM convention ⚠ Characterization ✔

+

Level: TABLE
Context: Validation
Category: Completeness
Subcategory:
Severity: CDM convention ⚠ (for observation period), +Characterization ✔  (for all other tables)

Description @@ -220,8 +221,12 @@

DefinitionAll other tablesAction on persons missing records in other clinical event tables will depend on the characteristics of the source database. In certain cases, missingness is expected – some persons may just not have a given type of -data available in the source. In others, various ETL issues may result -in persons missing records in a given event table:

+data available in the source. For instance, in most data sources, one +would expect most patients to have at least one visit, diagnosis, and +drug, while one would not expect every single patient to have +had a medical device.

+

Various ETL issues may result in persons missing records in a given +event table:

diff --git a/docs/articles/checks/measureValueCompleteness.html b/docs/articles/checks/measureValueCompleteness.html index f0be5e07..114f34ad 100644 --- a/docs/articles/checks/measureValueCompleteness.html +++ b/docs/articles/checks/measureValueCompleteness.html @@ -181,7 +181,7 @@

measureValueCompleteness

Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/measureValueCompleteness.Rmd @@ -216,8 +216,12 @@

Definition @@ -248,13 +252,13 @@

Violated rows query

ETL Developers

-

Failures of this check on fields required in the CDM specification -are redundant with failures of isRequired. See isRequired documentation for more -information.

+

Failures of this check on required fields are redundant with failures +of isRequired. See isRequired +documentation for more information.

ETL developers have 2 main options for the use of this check on non-required fields:

    -
  • The check threshold may be set to 100% for non-required fields such +
  • The check threshold may be left on 100% for non-required fields such that the check will never fail. The check result can be used simply to understand completeness for these fields
  • The check threshold may be set to an appropriate value corresponding @@ -264,19 +268,17 @@

    ETL Developers

Unexpectedly missing values should be investigated for a potential -root cause in the ETL. For expected missingness, rows that violate this -check in non-required fields are acceptable but should be clearly -communicated to data users so that they can know when and when not to -expect data to be present in each field. To avoid confusion for users, -however, thresholds should be modified to avoid check failures at -expected levels.

+root cause in the ETL. If a threshold has been adjusted to account for +expected missingness, this should be clearly communicated to data users +so that they can know when and when not to expect data to be present in +each field.

Data Users

This check informs you of the level of missing data in each column of -the CDM. If data is missing in a required column, see the isRequired -documentation for more information.

+the CDM. If data is missing in a required column, see the +isRequired documentation for more information.

The interpretation of a check failure on a non-required column will depend on the context. In some cases, the threshold for this check will have been very deliberately set, and any failure should be cause for diff --git a/docs/articles/checks/plausibleAfterBirth.html b/docs/articles/checks/plausibleAfterBirth.html index 610bc1e4..cacfbb0a 100644 --- a/docs/articles/checks/plausibleAfterBirth.html +++ b/docs/articles/checks/plausibleAfterBirth.html @@ -181,7 +181,7 @@

plausibleAfterBirth

Maxim Moinat, Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleAfterBirth.Rmd diff --git a/docs/articles/checks/plausibleBeforeDeath.html b/docs/articles/checks/plausibleBeforeDeath.html index 2e91b839..a2d11704 100644 --- a/docs/articles/checks/plausibleBeforeDeath.html +++ b/docs/articles/checks/plausibleBeforeDeath.html @@ -181,7 +181,7 @@

plausibleBeforeDeath

Maxim Moinat

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleBeforeDeath.Rmd diff --git a/docs/articles/checks/plausibleGenderUseDescendants.html b/docs/articles/checks/plausibleGenderUseDescendants.html index c4e8155a..2e92d60d 100644 --- a/docs/articles/checks/plausibleGenderUseDescendants.html +++ b/docs/articles/checks/plausibleGenderUseDescendants.html @@ -180,7 +180,7 @@

plausibleGender

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleGenderUseDescendants.Rmd diff --git a/docs/articles/checks/plausibleStartBeforeEnd.html b/docs/articles/checks/plausibleStartBeforeEnd.html index f80c0a9b..489a615d 100644 --- a/docs/articles/checks/plausibleStartBeforeEnd.html +++ b/docs/articles/checks/plausibleStartBeforeEnd.html @@ -181,7 +181,7 @@

plausibleStartBeforeEnd

Maxim Moinat

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleStartBeforeEnd.Rmd diff --git a/docs/articles/checks/plausibleTemporalAfter.html b/docs/articles/checks/plausibleTemporalAfter.html index 3a01db40..2c11f327 100644 --- a/docs/articles/checks/plausibleTemporalAfter.html +++ b/docs/articles/checks/plausibleTemporalAfter.html @@ -180,7 +180,7 @@

plausibleTemporalAfter

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleTemporalAfter.Rmd diff --git a/docs/articles/checks/plausibleUnitConceptIds.html b/docs/articles/checks/plausibleUnitConceptIds.html index 6855948c..cf0df5fb 100644 --- a/docs/articles/checks/plausibleUnitConceptIds.html +++ b/docs/articles/checks/plausibleUnitConceptIds.html @@ -180,7 +180,7 @@

plausibleUnitConceptIds

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleUnitConceptIds.Rmd diff --git a/docs/articles/checks/plausibleValueHigh.html b/docs/articles/checks/plausibleValueHigh.html index 799422ff..7879501f 100644 --- a/docs/articles/checks/plausibleValueHigh.html +++ b/docs/articles/checks/plausibleValueHigh.html @@ -181,7 +181,7 @@

plausibleValueHigh

Dymytry Dymshyts

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleValueHigh.Rmd @@ -234,7 +234,7 @@

Definition diff --git a/docs/articles/checks/plausibleValueLow.html b/docs/articles/checks/plausibleValueLow.html index 9cdbe4b0..40d10b8a 100644 --- a/docs/articles/checks/plausibleValueLow.html +++ b/docs/articles/checks/plausibleValueLow.html @@ -181,7 +181,7 @@

plausibleValueLow

Dymytry Dymshyts

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/plausibleValueLow.Rmd @@ -247,7 +247,7 @@

Definition diff --git a/docs/articles/checks/sourceConceptRecordCompleteness.html b/docs/articles/checks/sourceConceptRecordCompleteness.html index b5f59d0b..2d4effb4 100644 --- a/docs/articles/checks/sourceConceptRecordCompleteness.html +++ b/docs/articles/checks/sourceConceptRecordCompleteness.html @@ -181,7 +181,7 @@

sourceConceptRecordCompleteness

Katy Sadowski

-

2024-06-29

+

2024-07-11

Source: vignettes/checks/sourceConceptRecordCompleteness.Rmd @@ -207,29 +207,23 @@

DefinitionSource concept mapping
  • CDM Fields/Tables: All source concept ID -(X_source_concept_id) columns in all event tables.
    • +(_source_concept_id) columns in all event tables.
  • Default Threshold Value:
      -
    • 10 for primary source concept ID columns in condition, drug, -measurement, procedure, device, and observation tables
      • -
    • 100 for all other source concept ID columns
      • +
    • 10% for source concept ID columns in condition, drug, measurement, +procedure, device, and observation tables
      • +
    • 100% for all other source concept ID columns
    • @@ -247,11 +241,11 @@

    User Guidance

    ETL Developers

    -

    Recall that the X_source_concept_id columns should +

    Recall that the _source_concept_id columns should contain the OMOP concept representing the exact code used in the source -data for a given record: “If the is coded in the source -data using an OMOP supported vocabulary put the concept id representing -the source value here.”

    +data for a given record: “If the <_source_value> is coded in the +source data using an OMOP supported vocabulary put the concept id +representing the source value here.”

    A failure of this check usually indicates a failure to map a source value to an OMOP concept. In some cases, such a failure can and should be remediated in the concept-mapping step of the ETL. In other cases, it @@ -259,25 +253,24 @@

    ETL DevelopersTo investigate the failure, run the following query:

    SELECT  
   concept.concept_name AS standard_concept_name, 
-  cdmTable.X_concept_id, -- standard concept ID field for the table 
+  cdmTable._concept_id, -- standard concept ID field for the table 
   c2.concept_name AS source_value_concept_name, 
-  cdmTable.X_source_value, -- source value field for the table 
+  cdmTable._source_value, -- source value field for the table 
   COUNT(*) 
 FROM @cdmDatabaseSchema.@cdmTableName cdmTable 
-LEFT JOIN @vocabDatabaseSchema.concept ON concept.concept_id = cdmTable.X_concept_id 
+LEFT JOIN @vocabDatabaseSchema.concept ON concept.concept_id = cdmTable._concept_id 
 -- WARNING this join may cause fanning if a source value exists in multiple vocabularies 
-LEFT JOIN @vocabDatabaseSchema.concept c2 ON concept.concept_code = cdmTable.X_source_value 
+LEFT JOIN @vocabDatabaseSchema.concept c2 ON concept.concept_code = cdmTable._source_value 
 AND c2.domain_id = <Domain of cdmTable> 
 WHERE cdmTable.@cdmFieldName = 0  
--- AND cdmTable.value_as_number IS NOT NULL -- uncomment for unit_concept_id checks 
-GROUP BY 1,2,3 
-ORDER BY 4 DESC
    +GROUP BY 1,2,3 +ORDER BY 4 DESC

    The query results will give you a summary of the source codes which failed to map to an OMOP concept. Inspecting this data should give you an initial idea of what might be going on.

    If source values return legitimate matches on concept_code, it’s possible that there is an error in the concept mapping step of your ETL. -Please note that while the X_source_concept_id fields are +Please note that while the _source_concept_id fields are technically not required, it is highly recommended to populate them with OMOP concepts whenever possible. This will greatly aid analysts in understanding the provenance of the data.

    @@ -292,10 +285,10 @@

    Data UsersJared Houghtaling, Clair Blacketer

    -

    2024-06-29

    +

    2024-07-11

    Source:     vignettes/checks/sourceValueCompleteness.Rmd @@ -205,11 +205,11 @@

    DefinitionDefinitionKaty Sadowski

    -

    2024-06-29

    +

    2024-07-11

    Source:     vignettes/checks/standardConceptRecordCompleteness.Rmd @@ -207,10 +207,10 @@

    DefinitionDefinitionETL DevelopersTo investigate the failure, run the following query:

    SELECT  
   concept_name, 
-  cdmTable.X_source_concept_id, -- source concept ID field for the table 
-  cdmTable.X_source_value, -- source value field for the table 
+  cdmTable._source_concept_id, -- source concept ID field for the table 
+  cdmTable._source_value, -- source value field for the table 
   COUNT(*) 
 FROM @cdmDatabaseSchema.@cdmTableName cdmTable 
-LEFT JOIN @vocabDatabaseSchema.concept ON concept.concept_id = cdmTable.X_source_concept_id 
+LEFT JOIN @vocabDatabaseSchema.concept ON concept.concept_id = cdmTable._source_concept_id 
 WHERE cdmTable.@cdmFieldName = 0  
 -- AND cdmTable.value_as_number IS NOT NULL -- uncomment for unit_concept_id checks 
 GROUP BY 1,2,3 
@@ -327,11 +328,12 @@ 
    ETL Developers
 
  • Finally, if the investigation query returns no source value, you
 must trace the relevant record(s) back to their source and confirm if
-the missing value is expected. If not, identify and fix the related
-issue in your ETL. If the record legitimately has no value/code in the
-source data, then the standard concept ID may be left as 0. However, in
-some cases these “code-less” records represent junk data which should be
-filtered out in the ETL. The proper approach will be context-dependent
+the missing source value is expected. If not, identify and fix the
+related issue in your ETL. If the record legitimately has no value/code
+in the source data, then the standard concept ID may be left as 0.
+However, in some cases these “code-less” records represent junk data
+which should be filtered out in the ETL. The proper approach will be
+context-dependent
 
 
    • 
 
  • 
diff --git a/docs/LICENSE-text.html b/docs/LICENSE-text.html
index aaecd811..36a3d443 100644
--- a/docs/LICENSE-text.html
+++ b/docs/LICENSE-text.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/articles/AddNewCheck.html b/docs/articles/AddNewCheck.html
index 0e7ab7f1..83f74dc5 100644
--- a/docs/articles/AddNewCheck.html
+++ b/docs/articles/AddNewCheck.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -180,7 +183,7 @@
       
    Add a New Data Quality Check
    
                         
    Don Torok
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/AddNewCheck.rmd
       
diff --git a/docs/articles/CheckStatusDefinitions.html b/docs/articles/CheckStatusDefinitions.html
index fe8df052..a081dc68 100644
--- a/docs/articles/CheckStatusDefinitions.html
+++ b/docs/articles/CheckStatusDefinitions.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    Check Status Definitions
    
                         
    Dmitry Ilyn,
 Maxim Moinat
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/CheckStatusDefinitions.rmd
       
diff --git a/docs/articles/CheckTypeDescriptions.html b/docs/articles/CheckTypeDescriptions.html
index a58ea494..45cb4259 100644
--- a/docs/articles/CheckTypeDescriptions.html
+++ b/docs/articles/CheckTypeDescriptions.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    Data Quality Check Type Definitions
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/CheckTypeDescriptions.rmd
       
diff --git a/docs/articles/DataQualityDashboard.html b/docs/articles/DataQualityDashboard.html
index d22f577a..9b4e2174 100644
--- a/docs/articles/DataQualityDashboard.html
+++ b/docs/articles/DataQualityDashboard.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    Getting Started
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/DataQualityDashboard.rmd
       
diff --git a/docs/articles/DqdForCohorts.html b/docs/articles/DqdForCohorts.html
index d9530035..7077e6a3 100644
--- a/docs/articles/DqdForCohorts.html
+++ b/docs/articles/DqdForCohorts.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    Running the DQD on a Cohort
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/DqdForCohorts.rmd
       
diff --git a/docs/articles/SqlOnly.html b/docs/articles/SqlOnly.html
index 71653b00..b40aea46 100644
--- a/docs/articles/SqlOnly.html
+++ b/docs/articles/SqlOnly.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    Running the DQD in SqlOnly mode
    
                         
    Maxim
 Moinat
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/SqlOnly.rmd
       
diff --git a/docs/articles/Thresholds.html b/docs/articles/Thresholds.html
index 7bcce30c..e6796eed 100644
--- a/docs/articles/Thresholds.html
+++ b/docs/articles/Thresholds.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    Failure Thresholds and How to Change Them
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/Thresholds.rmd
       
diff --git a/docs/articles/checkIndex.html b/docs/articles/checkIndex.html
index 9289ad6a..d75f7b67 100644
--- a/docs/articles/checkIndex.html
+++ b/docs/articles/checkIndex.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    Index
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checkIndex.Rmd
       
@@ -236,7 +239,7 @@ 
    Checks
 
  • fkDomain
    • 
 
  • fkClass
    • 
 
  • measurePersonCompleteness
    • 
-
  • measureConditionEraCompleteness (PAGE UNDER CONSTRUCTION)
    • 
+
  • measureConditionEraCompleteness
    • 
 
  • isStandardValidConcept
    • 
 
  • measureValueCompleteness
    • 
 
  • standardConceptRecordCompleteness
    • 
diff --git a/docs/articles/checks/cdmDatatype.html b/docs/articles/checks/cdmDatatype.html
index bd765aeb..0d0020a1 100644
--- a/docs/articles/checks/cdmDatatype.html
+++ b/docs/articles/checks/cdmDatatype.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
    • 
 
  • 
@@ -181,7 +184,7 @@ 
    cdmDatatype
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/cdmDatatype.Rmd
       
diff --git a/docs/articles/checks/cdmField.html b/docs/articles/checks/cdmField.html
index 3464b260..64d70791 100644
--- a/docs/articles/checks/cdmField.html
+++ b/docs/articles/checks/cdmField.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    cdmField
    
                         
    Heidi Schmidt,
 Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/cdmField.Rmd
       
diff --git a/docs/articles/checks/cdmTable.html b/docs/articles/checks/cdmTable.html
index 03d2cd40..e5c80c29 100644
--- a/docs/articles/checks/cdmTable.html
+++ b/docs/articles/checks/cdmTable.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    cdmTable
    
                         
    John Gresh,
 Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/cdmTable.Rmd
       
diff --git a/docs/articles/checks/fkClass.html b/docs/articles/checks/fkClass.html
index bbfbd439..5c6f9d9e 100644
--- a/docs/articles/checks/fkClass.html
+++ b/docs/articles/checks/fkClass.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    fkClass
    
                         
    Clair
 Blacketer, Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/fkClass.Rmd
       
diff --git a/docs/articles/checks/fkDomain.html b/docs/articles/checks/fkDomain.html
index bc151de1..8335bb08 100644
--- a/docs/articles/checks/fkDomain.html
+++ b/docs/articles/checks/fkDomain.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    fkDomain
    
                         
    Clair
 Blacketer, Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/fkDomain.Rmd
       
diff --git a/docs/articles/checks/isForeignKey.html b/docs/articles/checks/isForeignKey.html
index 49b25eeb..0e24b288 100644
--- a/docs/articles/checks/isForeignKey.html
+++ b/docs/articles/checks/isForeignKey.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    isForeignKey
    
                         
    Dmytry
 Dymshyts, Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/isForeignKey.Rmd
       
diff --git a/docs/articles/checks/isPrimaryKey.html b/docs/articles/checks/isPrimaryKey.html
index cc5d3275..89b8a58d 100644
--- a/docs/articles/checks/isPrimaryKey.html
+++ b/docs/articles/checks/isPrimaryKey.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    isPrimaryKey
    
                         
    John Gresh,
 Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/isPrimaryKey.Rmd
       
diff --git a/docs/articles/checks/isRequired.html b/docs/articles/checks/isRequired.html
index 3da6dfaa..3dab8f3c 100644
--- a/docs/articles/checks/isRequired.html
+++ b/docs/articles/checks/isRequired.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    isRequired
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/isRequired.Rmd
       
diff --git a/docs/articles/checks/isStandardValidConcept.html b/docs/articles/checks/isStandardValidConcept.html
index 7f40000a..b80a0b53 100644
--- a/docs/articles/checks/isStandardValidConcept.html
+++ b/docs/articles/checks/isStandardValidConcept.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    isStandardValidConcept
    
                         
    Stephanie Hong,
 Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/isStandardValidConcept.Rmd
       
diff --git a/docs/articles/checks/measureConditionEraCompleteness.html b/docs/articles/checks/measureConditionEraCompleteness.html
index 2689e33c..feb1d1af 100644
--- a/docs/articles/checks/measureConditionEraCompleteness.html
+++ b/docs/articles/checks/measureConditionEraCompleteness.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -178,9 +181,10 @@
   
    
     
    
       
    measureConditionEraCompleteness
    
-                        
    
+                        
    Maxim
+Moinat
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/measureConditionEraCompleteness.Rmd
       
@@ -192,45 +196,80 @@ 
    2024-07-11
    
 
    
 
    Summary
 
    
-
    Level: TABLE
    Context: Validation
    Category: Completeness
    Subcategory:
    Severity:
    
+
    Level: TABLE
    Context: Validation
    Category: Completeness
    Subcategory:
    Severity: CDM convention ⚠
    
 
    
 
    
 
    Description
 
    
-
    The number and Percent of persons that does not have condition_era
-built successfully, for all persons in condition_occurrence
    
+
    The number and percent of persons that does not have condition_era
+built successfully, for all persons in
+CONDITION_OCCURRENCE.
    
 
    
 
    
 
    Definition
 
    
 
      
 
    • 
-Numerator:
      • 
+Numerator: Number of unique person_ids that exist in the
+CONDITION_OCCURRENCE table but not in the
+CONDITION_ERA table.
 
    • 
-Denominator:
      • 
+Denominator: Number of unique person_ids in the
+CONDITION_OCCURRENCE table.
 
    • 
-Related CDM Convention(s):
      • 
+Related CDM Convention(s): Condition Era’s are directly
+derived from Condition Occurrence.
 
    • 
-CDM Fields/Tables:
      • 
+CDM Fields/Tables: CONDITION_ERA
+
 
    • 
-Default Threshold Value:
      • 
+Default Threshold Value: 0%
 
    
 
    
 
    
 
    User Guidance
 
    
+
    The Condition
+Era CDM documentation states that the condition era’s should be
+derived by combining condition occurrences. This implies that each
+person with a condition occurrence should have at least a condition era.
+It does NOT clearly state that the CONDITION_ERA table is
+required when there are condition occurrences. Still, it is has always
+been a common convention in the OHDSI community to derive condition era.
+There is currently no THEMIS convention on condition eras.
    
 
    
 
    Violated rows query
 
    
-
    
+
    SELECT DISTINCT 
+  co.person_id
+FROM @cdmDatabaseSchema.condition_occurrence co
+  LEFT JOIN @cdmDatabaseSchema.condition_era cdmTable 
+    ON co.person_id = cdmTable.person_id
+WHERE cdmTable.person_id IS NULL
    
 
    
 
    
 
    ETL Developers
 
    
+
    If this check fails, it is likely that there is an issue with the
+condition era derivation script. Please review the ETL execution log. It
+might be that this script was not executed and the condition era table
+is empty, or it had issues running and the condition era table has been
+partially populated. If no issues with the ETL run found, the condition
+era derivation script might have bugs. Please review the code. An
+example script can be found on the
+CDM Documentation page. In both cases it is advised to truncate the
+CONDITION_ERA table and rerun the derivation script.
    
 
    
 
    
 
    Data Users
 
    
+
    The CONDITION_ERA table might seem to contain redundant
+information, as for most uses the CONDITION_OCCURRENCE
+table can be used. However, tools like FeatureExtraction use condition
+eras to build some covariates and network studies might use cohorts that
+are based on condition eras. It is therefore important that the
+CONDITION_ERA table is fully populated and captures the
+same persons as in condition occurrence.
    
 
    
 
    
   
    
diff --git a/docs/articles/checks/measurePersonCompleteness.html b/docs/articles/checks/measurePersonCompleteness.html
index 3d68ce34..b0317098 100644
--- a/docs/articles/checks/measurePersonCompleteness.html
+++ b/docs/articles/checks/measurePersonCompleteness.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    measurePersonCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/measurePersonCompleteness.Rmd
       
diff --git a/docs/articles/checks/measureValueCompleteness.html b/docs/articles/checks/measureValueCompleteness.html
index 114f34ad..dbb2554e 100644
--- a/docs/articles/checks/measureValueCompleteness.html
+++ b/docs/articles/checks/measureValueCompleteness.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    measureValueCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/measureValueCompleteness.Rmd
       
diff --git a/docs/articles/checks/plausibleAfterBirth.html b/docs/articles/checks/plausibleAfterBirth.html
index cacfbb0a..a8e0f1d7 100644
--- a/docs/articles/checks/plausibleAfterBirth.html
+++ b/docs/articles/checks/plausibleAfterBirth.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    plausibleAfterBirth
    
                         
    Maxim Moinat,
 Katy Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleAfterBirth.Rmd
       
diff --git a/docs/articles/checks/plausibleBeforeDeath.html b/docs/articles/checks/plausibleBeforeDeath.html
index a2d11704..771b3883 100644
--- a/docs/articles/checks/plausibleBeforeDeath.html
+++ b/docs/articles/checks/plausibleBeforeDeath.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    plausibleBeforeDeath
    
                         
    Maxim
 Moinat
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleBeforeDeath.Rmd
       
diff --git a/docs/articles/checks/plausibleGenderUseDescendants.html b/docs/articles/checks/plausibleGenderUseDescendants.html
index 2e92d60d..23ca231b 100644
--- a/docs/articles/checks/plausibleGenderUseDescendants.html
+++ b/docs/articles/checks/plausibleGenderUseDescendants.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -180,7 +183,7 @@
       
    plausibleGender
    
                         
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleGenderUseDescendants.Rmd
       
diff --git a/docs/articles/checks/plausibleStartBeforeEnd.html b/docs/articles/checks/plausibleStartBeforeEnd.html
index 489a615d..66ae6eb9 100644
--- a/docs/articles/checks/plausibleStartBeforeEnd.html
+++ b/docs/articles/checks/plausibleStartBeforeEnd.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    plausibleStartBeforeEnd
    
                         
    Maxim
 Moinat
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleStartBeforeEnd.Rmd
       
diff --git a/docs/articles/checks/plausibleTemporalAfter.html b/docs/articles/checks/plausibleTemporalAfter.html
index 2c11f327..7c3cf881 100644
--- a/docs/articles/checks/plausibleTemporalAfter.html
+++ b/docs/articles/checks/plausibleTemporalAfter.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -180,7 +183,7 @@
       
    plausibleTemporalAfter
    
                         
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleTemporalAfter.Rmd
       
diff --git a/docs/articles/checks/plausibleUnitConceptIds.html b/docs/articles/checks/plausibleUnitConceptIds.html
index cf0df5fb..71f25bfe 100644
--- a/docs/articles/checks/plausibleUnitConceptIds.html
+++ b/docs/articles/checks/plausibleUnitConceptIds.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -180,7 +183,7 @@
       
    plausibleUnitConceptIds
    
                         
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleUnitConceptIds.Rmd
       
diff --git a/docs/articles/checks/plausibleValueHigh.html b/docs/articles/checks/plausibleValueHigh.html
index 7879501f..fdcdc415 100644
--- a/docs/articles/checks/plausibleValueHigh.html
+++ b/docs/articles/checks/plausibleValueHigh.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    plausibleValueHigh
    
                         
    Dymytry
 Dymshyts
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleValueHigh.Rmd
       
diff --git a/docs/articles/checks/plausibleValueLow.html b/docs/articles/checks/plausibleValueLow.html
index 40d10b8a..86b7bf72 100644
--- a/docs/articles/checks/plausibleValueLow.html
+++ b/docs/articles/checks/plausibleValueLow.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    plausibleValueLow
    
                         
    Dymytry
 Dymshyts
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/plausibleValueLow.Rmd
       
diff --git a/docs/articles/checks/sourceConceptRecordCompleteness.html b/docs/articles/checks/sourceConceptRecordCompleteness.html
index 2d4effb4..7dba571d 100644
--- a/docs/articles/checks/sourceConceptRecordCompleteness.html
+++ b/docs/articles/checks/sourceConceptRecordCompleteness.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    sourceConceptRecordCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/sourceConceptRecordCompleteness.Rmd
       
diff --git a/docs/articles/checks/sourceValueCompleteness.html b/docs/articles/checks/sourceValueCompleteness.html
index e02e771f..f52bd772 100644
--- a/docs/articles/checks/sourceValueCompleteness.html
+++ b/docs/articles/checks/sourceValueCompleteness.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    sourceValueCompleteness
    
                         
    Jared
 Houghtaling, Clair Blacketer
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/sourceValueCompleteness.Rmd
       
diff --git a/docs/articles/checks/standardConceptRecordCompleteness.html b/docs/articles/checks/standardConceptRecordCompleteness.html
index ee2794ce..4b1e3cbf 100644
--- a/docs/articles/checks/standardConceptRecordCompleteness.html
+++ b/docs/articles/checks/standardConceptRecordCompleteness.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    standardConceptRecordCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/standardConceptRecordCompleteness.Rmd
       
diff --git a/docs/articles/checks/withinVisitDates.html b/docs/articles/checks/withinVisitDates.html
index 1ca9c9b6..3089b657 100644
--- a/docs/articles/checks/withinVisitDates.html
+++ b/docs/articles/checks/withinVisitDates.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
@@ -181,7 +184,7 @@ 
    withinVisitDates
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-11
    
+            
    2024-07-12
    
       
       Source: vignettes/checks/withinVisitDates.Rmd
       
diff --git a/docs/articles/index.html b/docs/articles/index.html
index a9c50ea9..1eadbf9d 100644
--- a/docs/articles/index.html
+++ b/docs/articles/index.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/authors.html b/docs/authors.html
index 89196518..d68b7afa 100644
--- a/docs/authors.html
+++ b/docs/authors.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/index.html b/docs/index.html
index f76a3b99..3aa90c26 100644
--- a/docs/index.html
+++ b/docs/index.html
@@ -148,6 +148,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
 
  • 
diff --git a/docs/news/index.html b/docs/news/index.html
index 83e23076..272d626f 100644
--- a/docs/news/index.html
+++ b/docs/news/index.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/pkgdown.yml b/docs/pkgdown.yml
index ed074627..deb660db 100644
--- a/docs/pkgdown.yml
+++ b/docs/pkgdown.yml
@@ -34,7 +34,7 @@ articles:
   standardConceptRecordCompleteness: checks/standardConceptRecordCompleteness.html
   Thresholds: Thresholds.html
   withinVisitDates: checks/withinVisitDates.html
-last_built: 2024-07-12T01:44Z
+last_built: 2024-07-13T01:28Z
 urls:
   reference: https://ohdsi.github.io/DataQualityDashboard/reference
   article: https://ohdsi.github.io/DataQualityDashboard/articles
diff --git a/docs/reference/convertJsonResultsFileCase.html b/docs/reference/convertJsonResultsFileCase.html
index 0b2871f8..00d09faa 100644
--- a/docs/reference/convertJsonResultsFileCase.html
+++ b/docs/reference/convertJsonResultsFileCase.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-evaluateThresholds.html b/docs/reference/dot-evaluateThresholds.html
index 0e3a4c2c..bfe86dfa 100644
--- a/docs/reference/dot-evaluateThresholds.html
+++ b/docs/reference/dot-evaluateThresholds.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-getCheckId.html b/docs/reference/dot-getCheckId.html
index b631cb51..2a1aaa17 100644
--- a/docs/reference/dot-getCheckId.html
+++ b/docs/reference/dot-getCheckId.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-processCheck.html b/docs/reference/dot-processCheck.html
index 7f29c99a..aa5e208e 100644
--- a/docs/reference/dot-processCheck.html
+++ b/docs/reference/dot-processCheck.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-recordResult.html b/docs/reference/dot-recordResult.html
index 96673dd9..66ad9690 100644
--- a/docs/reference/dot-recordResult.html
+++ b/docs/reference/dot-recordResult.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-runCheck.html b/docs/reference/dot-runCheck.html
index 9b4c5b79..2a33c3c0 100644
--- a/docs/reference/dot-runCheck.html
+++ b/docs/reference/dot-runCheck.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-summarizeResults.html b/docs/reference/dot-summarizeResults.html
index 8d22ee63..b158e832 100644
--- a/docs/reference/dot-summarizeResults.html
+++ b/docs/reference/dot-summarizeResults.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-writeResultsToCsv.html b/docs/reference/dot-writeResultsToCsv.html
index 1a209ffe..7449327b 100644
--- a/docs/reference/dot-writeResultsToCsv.html
+++ b/docs/reference/dot-writeResultsToCsv.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-writeResultsToJson.html b/docs/reference/dot-writeResultsToJson.html
index 62410ef1..105d5a9c 100644
--- a/docs/reference/dot-writeResultsToJson.html
+++ b/docs/reference/dot-writeResultsToJson.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/dot-writeResultsToTable.html b/docs/reference/dot-writeResultsToTable.html
index 2e215376..b235d9f2 100644
--- a/docs/reference/dot-writeResultsToTable.html
+++ b/docs/reference/dot-writeResultsToTable.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/executeDqChecks.html b/docs/reference/executeDqChecks.html
index 6382ed42..8306931c 100644
--- a/docs/reference/executeDqChecks.html
+++ b/docs/reference/executeDqChecks.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/index.html b/docs/reference/index.html
index f9d83e64..cc610a6d 100644
--- a/docs/reference/index.html
+++ b/docs/reference/index.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/listDqChecks.html b/docs/reference/listDqChecks.html
index 606e2d87..3b478b14 100644
--- a/docs/reference/listDqChecks.html
+++ b/docs/reference/listDqChecks.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/reEvaluateThresholds.html b/docs/reference/reEvaluateThresholds.html
index dd74b726..f2d1bbc4 100644
--- a/docs/reference/reEvaluateThresholds.html
+++ b/docs/reference/reEvaluateThresholds.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/viewDqDashboard.html b/docs/reference/viewDqDashboard.html
index ece1e45c..2173fbd9 100644
--- a/docs/reference/viewDqDashboard.html
+++ b/docs/reference/viewDqDashboard.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/writeDBResultsToJson.html b/docs/reference/writeDBResultsToJson.html
index 283d3e2a..26b34df4 100644
--- a/docs/reference/writeDBResultsToJson.html
+++ b/docs/reference/writeDBResultsToJson.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/writeJsonResultsToCsv.html b/docs/reference/writeJsonResultsToCsv.html
index 12a8f937..bbc1d1a6 100644
--- a/docs/reference/writeJsonResultsToCsv.html
+++ b/docs/reference/writeJsonResultsToCsv.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/docs/reference/writeJsonResultsToTable.html b/docs/reference/writeJsonResultsToTable.html
index e944a846..556aff0f 100644
--- a/docs/reference/writeJsonResultsToTable.html
+++ b/docs/reference/writeJsonResultsToTable.html
@@ -128,6 +128,9 @@
     
  • 
       withinVisitDates
     
    • 
+    
  • 
+      measureConditionEraCompleteness
+    
    • 
   
 
  • 
   Changelog
diff --git a/vignettes/checkIndex.Rmd b/vignettes/checkIndex.Rmd
index 4e93a905..af436761 100644
--- a/vignettes/checkIndex.Rmd
+++ b/vignettes/checkIndex.Rmd
@@ -35,7 +35,7 @@ above to navigate to the check's documentation page.\
 - [fkDomain](checks/fkDomain.html)
 - [fkClass](checks/fkClass.html)
 - [measurePersonCompleteness](checks/measurePersonCompleteness.html)
-- measureConditionEraCompleteness (PAGE UNDER CONSTRUCTION)
+- [measureConditionEraCompleteness](checks/measureConditionEraCompleteness.html)
 - [isStandardValidConcept](checks/isStandardValidConcept.html)
 - [measureValueCompleteness](checks/measureValueCompleteness.html)
 - [standardConceptRecordCompleteness](checks/standardConceptRecordCompleteness.html)

From a923eb263b1c12f14ca02ed422193bac5989f884 Mon Sep 17 00:00:00 2001
From: Katy Sadowski 
Date: Fri, 12 Jul 2024 21:37:44 -0400
Subject: [PATCH 16/16] catch up develop

---
 docs/articles/AddNewCheck.html                |   5 +-
 docs/articles/CheckStatusDefinitions.html     |   5 +-
 docs/articles/CheckTypeDescriptions.html      |   5 +-
 docs/articles/DataQualityDashboard.html       |   5 +-
 docs/articles/DqdForCohorts.html              |   5 +-
 docs/articles/SqlOnly.html                    |   5 +-
 docs/articles/Thresholds.html                 |   5 +-
 docs/articles/checkIndex.html                 |   7 +-
 docs/articles/checks/cdmDatatype.html         |   5 +-
 docs/articles/checks/cdmField.html            |   5 +-
 docs/articles/checks/cdmTable.html            |   5 +-
 docs/articles/checks/fkClass.html             |   5 +-
 docs/articles/checks/fkDomain.html            |   5 +-
 docs/articles/checks/isForeignKey.html        |   5 +-
 docs/articles/checks/isPrimaryKey.html        |   5 +-
 docs/articles/checks/isRequired.html          |   5 +-
 .../checks/isStandardValidConcept.html        |   5 +-
 .../checks/measurePersonCompleteness.html     |   5 +-
 .../checks/measureValueCompleteness.html      |   5 +-
 docs/articles/checks/plausibleAfterBirth.html |   5 +-
 .../articles/checks/plausibleBeforeDeath.html |   5 +-
 docs/articles/checks/plausibleDuringLife.html | 246 -----------------
 docs/articles/checks/plausibleGender.html     | 247 ------------------
 .../checks/plausibleGenderUseDescendants.html |   5 +-
 .../checks/plausibleStartBeforeEnd.html       |   5 +-
 .../checks/plausibleTemporalAfter.html        |   5 +-
 .../checks/plausibleUnitConceptIds.html       |   5 +-
 docs/articles/checks/plausibleValueHigh.html  |   5 +-
 docs/articles/checks/plausibleValueLow.html   |   5 +-
 .../sourceConceptRecordCompleteness.html      |   5 +-
 .../checks/sourceValueCompleteness.html       |   5 +-
 .../standardConceptRecordCompleteness.html    |   5 +-
 docs/articles/checks/withinVisitDates.html    |   5 +-
 docs/articles/index.html                      |   3 -
 34 files changed, 32 insertions(+), 621 deletions(-)
 delete mode 100644 docs/articles/checks/plausibleDuringLife.html
 delete mode 100644 docs/articles/checks/plausibleGender.html

diff --git a/docs/articles/AddNewCheck.html b/docs/articles/AddNewCheck.html
index 83f74dc5..0e7ab7f1 100644
--- a/docs/articles/AddNewCheck.html
+++ b/docs/articles/AddNewCheck.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -183,7 +180,7 @@
       
    Add a New Data Quality Check
    
                         
    Don Torok
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/AddNewCheck.rmd
       
diff --git a/docs/articles/CheckStatusDefinitions.html b/docs/articles/CheckStatusDefinitions.html
index a081dc68..fe8df052 100644
--- a/docs/articles/CheckStatusDefinitions.html
+++ b/docs/articles/CheckStatusDefinitions.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    Check Status Definitions
    
                         
    Dmitry Ilyn,
 Maxim Moinat
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/CheckStatusDefinitions.rmd
       
diff --git a/docs/articles/CheckTypeDescriptions.html b/docs/articles/CheckTypeDescriptions.html
index 45cb4259..a58ea494 100644
--- a/docs/articles/CheckTypeDescriptions.html
+++ b/docs/articles/CheckTypeDescriptions.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    Data Quality Check Type Definitions
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/CheckTypeDescriptions.rmd
       
diff --git a/docs/articles/DataQualityDashboard.html b/docs/articles/DataQualityDashboard.html
index 9b4e2174..d22f577a 100644
--- a/docs/articles/DataQualityDashboard.html
+++ b/docs/articles/DataQualityDashboard.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    Getting Started
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/DataQualityDashboard.rmd
       
diff --git a/docs/articles/DqdForCohorts.html b/docs/articles/DqdForCohorts.html
index 7077e6a3..d9530035 100644
--- a/docs/articles/DqdForCohorts.html
+++ b/docs/articles/DqdForCohorts.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    Running the DQD on a Cohort
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/DqdForCohorts.rmd
       
diff --git a/docs/articles/SqlOnly.html b/docs/articles/SqlOnly.html
index b40aea46..71653b00 100644
--- a/docs/articles/SqlOnly.html
+++ b/docs/articles/SqlOnly.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    Running the DQD in SqlOnly mode
    
                         
    Maxim
 Moinat
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/SqlOnly.rmd
       
diff --git a/docs/articles/Thresholds.html b/docs/articles/Thresholds.html
index e6796eed..7bcce30c 100644
--- a/docs/articles/Thresholds.html
+++ b/docs/articles/Thresholds.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    Failure Thresholds and How to Change Them
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/Thresholds.rmd
       
diff --git a/docs/articles/checkIndex.html b/docs/articles/checkIndex.html
index d75f7b67..9289ad6a 100644
--- a/docs/articles/checkIndex.html
+++ b/docs/articles/checkIndex.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    Index
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checkIndex.Rmd
       
@@ -239,7 +236,7 @@ 
    Checks
 
  • fkDomain
    • 
 
  • fkClass
    • 
 
  • measurePersonCompleteness
    • 
-
  • measureConditionEraCompleteness
    • 
+
  • measureConditionEraCompleteness (PAGE UNDER CONSTRUCTION)
    • 
 
  • isStandardValidConcept
    • 
 
  • measureValueCompleteness
    • 
 
  • standardConceptRecordCompleteness
    • 
diff --git a/docs/articles/checks/cdmDatatype.html b/docs/articles/checks/cdmDatatype.html
index 0d0020a1..bd765aeb 100644
--- a/docs/articles/checks/cdmDatatype.html
+++ b/docs/articles/checks/cdmDatatype.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
    • 
 
  • 
@@ -184,7 +181,7 @@ 
    cdmDatatype
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/cdmDatatype.Rmd
       
diff --git a/docs/articles/checks/cdmField.html b/docs/articles/checks/cdmField.html
index 64d70791..3464b260 100644
--- a/docs/articles/checks/cdmField.html
+++ b/docs/articles/checks/cdmField.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    cdmField
    
                         
    Heidi Schmidt,
 Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/cdmField.Rmd
       
diff --git a/docs/articles/checks/cdmTable.html b/docs/articles/checks/cdmTable.html
index e5c80c29..03d2cd40 100644
--- a/docs/articles/checks/cdmTable.html
+++ b/docs/articles/checks/cdmTable.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    cdmTable
    
                         
    John Gresh,
 Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/cdmTable.Rmd
       
diff --git a/docs/articles/checks/fkClass.html b/docs/articles/checks/fkClass.html
index 5c6f9d9e..bbfbd439 100644
--- a/docs/articles/checks/fkClass.html
+++ b/docs/articles/checks/fkClass.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    fkClass
    
                         
    Clair
 Blacketer, Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/fkClass.Rmd
       
diff --git a/docs/articles/checks/fkDomain.html b/docs/articles/checks/fkDomain.html
index 8335bb08..bc151de1 100644
--- a/docs/articles/checks/fkDomain.html
+++ b/docs/articles/checks/fkDomain.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    fkDomain
    
                         
    Clair
 Blacketer, Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/fkDomain.Rmd
       
diff --git a/docs/articles/checks/isForeignKey.html b/docs/articles/checks/isForeignKey.html
index 0e24b288..49b25eeb 100644
--- a/docs/articles/checks/isForeignKey.html
+++ b/docs/articles/checks/isForeignKey.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    isForeignKey
    
                         
    Dmytry
 Dymshyts, Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/isForeignKey.Rmd
       
diff --git a/docs/articles/checks/isPrimaryKey.html b/docs/articles/checks/isPrimaryKey.html
index 89b8a58d..cc5d3275 100644
--- a/docs/articles/checks/isPrimaryKey.html
+++ b/docs/articles/checks/isPrimaryKey.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    isPrimaryKey
    
                         
    John Gresh,
 Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/isPrimaryKey.Rmd
       
diff --git a/docs/articles/checks/isRequired.html b/docs/articles/checks/isRequired.html
index 3dab8f3c..3da6dfaa 100644
--- a/docs/articles/checks/isRequired.html
+++ b/docs/articles/checks/isRequired.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    isRequired
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/isRequired.Rmd
       
diff --git a/docs/articles/checks/isStandardValidConcept.html b/docs/articles/checks/isStandardValidConcept.html
index b80a0b53..7f40000a 100644
--- a/docs/articles/checks/isStandardValidConcept.html
+++ b/docs/articles/checks/isStandardValidConcept.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    isStandardValidConcept
    
                         
    Stephanie Hong,
 Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/isStandardValidConcept.Rmd
       
diff --git a/docs/articles/checks/measurePersonCompleteness.html b/docs/articles/checks/measurePersonCompleteness.html
index b0317098..3d68ce34 100644
--- a/docs/articles/checks/measurePersonCompleteness.html
+++ b/docs/articles/checks/measurePersonCompleteness.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    measurePersonCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/measurePersonCompleteness.Rmd
       
diff --git a/docs/articles/checks/measureValueCompleteness.html b/docs/articles/checks/measureValueCompleteness.html
index dbb2554e..114f34ad 100644
--- a/docs/articles/checks/measureValueCompleteness.html
+++ b/docs/articles/checks/measureValueCompleteness.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    measureValueCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/measureValueCompleteness.Rmd
       
diff --git a/docs/articles/checks/plausibleAfterBirth.html b/docs/articles/checks/plausibleAfterBirth.html
index a8e0f1d7..cacfbb0a 100644
--- a/docs/articles/checks/plausibleAfterBirth.html
+++ b/docs/articles/checks/plausibleAfterBirth.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    plausibleAfterBirth
    
                         
    Maxim Moinat,
 Katy Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleAfterBirth.Rmd
       
diff --git a/docs/articles/checks/plausibleBeforeDeath.html b/docs/articles/checks/plausibleBeforeDeath.html
index 771b3883..a2d11704 100644
--- a/docs/articles/checks/plausibleBeforeDeath.html
+++ b/docs/articles/checks/plausibleBeforeDeath.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    plausibleBeforeDeath
    
                         
    Maxim
 Moinat
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleBeforeDeath.Rmd
       
diff --git a/docs/articles/checks/plausibleDuringLife.html b/docs/articles/checks/plausibleDuringLife.html
deleted file mode 100644
index bec0bd41..00000000
--- a/docs/articles/checks/plausibleDuringLife.html
+++ /dev/null
@@ -1,246 +0,0 @@
-
-
-
-
-
-
-
-plausibleDuringLife • DataQualityDashboard
-
-
-
-
-
-
-
-
-
-
-    
-
-    
    
-      
    
-
-
-      
-
-      
    
-  
    
-    
    
-      
    plausibleDuringLife
    
-                        
    
-            
-            
    2024-06-13
    
-      
-      Source: vignettes/checks/plausibleDuringLife.Rmd
-      
-
-    
    
-
-    
-    
-
    
-
    Summary
-
    
-
    Level: FIELD
    Context: Verification
    Category: Plausibility
    Subcategory: Temporal
    Severity:
    
-
    
-
    
-
    Description
-
    
-
    If yes, the number and percent of records with a date value in the
-@cdmFieldName field of the @cdmTableName table that occurs after death.
    
-
    
-
    
-
    Definition
-
    
-
      
-
    • 
-Numerator:
      • 
-
    • 
-Denominator:
      • 
-
    • 
-Related CDM Convention(s):
      • 
-
    • 
-CDM Fields/Tables:
      • 
-
    • 
-Default Threshold Value:
      • 
-
    
-
    
-
    
-
    User Guidance
-
    
-
    
-
    Violated rows query
-
    
-
    
-
    
-
    
-
    ETL Developers
-
    
-
    
-
    
-
    Data Users
-
    
-
    
-
    
-  
    
-
-  
-
-
    
-
-
-
-      
    
-  
    
-
    Developed by Katy Sadowski, Clair Blacketer, Ajit Londhe, Anthony Sena, Anthony Molinaro, Frank DeFalco, Pavel Grafkin.
    
-
    
-
-
    
-  
    
-
    Site built with pkgdown 2.0.9.
    
-
    
-
-      
    
-
    
-
-  
-
-
-  
-
-  
-
diff --git a/docs/articles/checks/plausibleGender.html b/docs/articles/checks/plausibleGender.html
deleted file mode 100644
index a5e62088..00000000
--- a/docs/articles/checks/plausibleGender.html
+++ /dev/null
@@ -1,247 +0,0 @@
-
-
-
-
-
-
-
-plausibleGender • DataQualityDashboard
-
-
-
-
-
-
-
-
-
-
-    
-
-    
    
-      
    
-
-
-      
-
-      
    
-  
    
-    
    
-      
    plausibleGender
    
-                        
    
-            
-            
    2024-06-13
    
-      
-      Source: vignettes/checks/plausibleGender.Rmd
-      
-
-    
    
-
-    
-    
-
    
-
    Summary
-
    
-
    Level: CONCEPT
    Context: Validation
    Category: Plausibility
    Subcategory: Atemporal
    Severity:
    
-
    
-
    
-
    Description
-
    
-
    For a CONCEPT_ID @conceptId (@conceptName), the number and percent of records
-associated with patients with an implausible gender (correct gender =
-@plausibleGender).
    
-
    
-
    
-
    Definition
-
    
-
      
-
    • 
-Numerator:
      • 
-
    • 
-Denominator:
      • 
-
    • 
-Related CDM Convention(s):
      • 
-
    • 
-CDM Fields/Tables:
      • 
-
    • 
-Default Threshold Value:
      • 
-
    
-
    
-
    
-
    User Guidance
-
    
-
    
-
    Violated rows query
-
    
-
    
-
    
-
    
-
    ETL Developers
-
    
-
    
-
    
-
    Data Users
-
    
-
    
-
    
-  
    
-
-  
-
-
    
-
-
-
-      
    
-  
    
-
    Developed by Katy Sadowski, Clair Blacketer, Ajit Londhe, Anthony Sena, Anthony Molinaro, Frank DeFalco, Pavel Grafkin.
    
-
    
-
-
    
-  
    
-
    Site built with pkgdown 2.0.9.
    
-
    
-
-      
    
-
    
-
-  
-
-
-  
-
-  
-
diff --git a/docs/articles/checks/plausibleGenderUseDescendants.html b/docs/articles/checks/plausibleGenderUseDescendants.html
index 23ca231b..2e92d60d 100644
--- a/docs/articles/checks/plausibleGenderUseDescendants.html
+++ b/docs/articles/checks/plausibleGenderUseDescendants.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -183,7 +180,7 @@
       
    plausibleGender
    
                         
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleGenderUseDescendants.Rmd
       
diff --git a/docs/articles/checks/plausibleStartBeforeEnd.html b/docs/articles/checks/plausibleStartBeforeEnd.html
index 66ae6eb9..489a615d 100644
--- a/docs/articles/checks/plausibleStartBeforeEnd.html
+++ b/docs/articles/checks/plausibleStartBeforeEnd.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    plausibleStartBeforeEnd
    
                         
    Maxim
 Moinat
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleStartBeforeEnd.Rmd
       
diff --git a/docs/articles/checks/plausibleTemporalAfter.html b/docs/articles/checks/plausibleTemporalAfter.html
index 7c3cf881..2c11f327 100644
--- a/docs/articles/checks/plausibleTemporalAfter.html
+++ b/docs/articles/checks/plausibleTemporalAfter.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -183,7 +180,7 @@
       
    plausibleTemporalAfter
    
                         
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleTemporalAfter.Rmd
       
diff --git a/docs/articles/checks/plausibleUnitConceptIds.html b/docs/articles/checks/plausibleUnitConceptIds.html
index 71f25bfe..cf0df5fb 100644
--- a/docs/articles/checks/plausibleUnitConceptIds.html
+++ b/docs/articles/checks/plausibleUnitConceptIds.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -183,7 +180,7 @@
       
    plausibleUnitConceptIds
    
                         
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleUnitConceptIds.Rmd
       
diff --git a/docs/articles/checks/plausibleValueHigh.html b/docs/articles/checks/plausibleValueHigh.html
index fdcdc415..7879501f 100644
--- a/docs/articles/checks/plausibleValueHigh.html
+++ b/docs/articles/checks/plausibleValueHigh.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    plausibleValueHigh
    
                         
    Dymytry
 Dymshyts
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleValueHigh.Rmd
       
diff --git a/docs/articles/checks/plausibleValueLow.html b/docs/articles/checks/plausibleValueLow.html
index 86b7bf72..40d10b8a 100644
--- a/docs/articles/checks/plausibleValueLow.html
+++ b/docs/articles/checks/plausibleValueLow.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    plausibleValueLow
    
                         
    Dymytry
 Dymshyts
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/plausibleValueLow.Rmd
       
diff --git a/docs/articles/checks/sourceConceptRecordCompleteness.html b/docs/articles/checks/sourceConceptRecordCompleteness.html
index 7dba571d..2d4effb4 100644
--- a/docs/articles/checks/sourceConceptRecordCompleteness.html
+++ b/docs/articles/checks/sourceConceptRecordCompleteness.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    sourceConceptRecordCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/sourceConceptRecordCompleteness.Rmd
       
diff --git a/docs/articles/checks/sourceValueCompleteness.html b/docs/articles/checks/sourceValueCompleteness.html
index f52bd772..e02e771f 100644
--- a/docs/articles/checks/sourceValueCompleteness.html
+++ b/docs/articles/checks/sourceValueCompleteness.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    sourceValueCompleteness
    
                         
    Jared
 Houghtaling, Clair Blacketer
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/sourceValueCompleteness.Rmd
       
diff --git a/docs/articles/checks/standardConceptRecordCompleteness.html b/docs/articles/checks/standardConceptRecordCompleteness.html
index 4b1e3cbf..ee2794ce 100644
--- a/docs/articles/checks/standardConceptRecordCompleteness.html
+++ b/docs/articles/checks/standardConceptRecordCompleteness.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    standardConceptRecordCompleteness
    
                         
    Katy
 Sadowski
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/standardConceptRecordCompleteness.Rmd
       
diff --git a/docs/articles/checks/withinVisitDates.html b/docs/articles/checks/withinVisitDates.html
index 3089b657..1ca9c9b6 100644
--- a/docs/articles/checks/withinVisitDates.html
+++ b/docs/articles/checks/withinVisitDates.html
@@ -148,9 +148,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
 
  • 
@@ -184,7 +181,7 @@ 
    withinVisitDates
    
                         
    Clair
 Blacketer
    
             
-            
    2024-07-12
    
+            
    2024-07-11
    
       
       Source: vignettes/checks/withinVisitDates.Rmd
       
diff --git a/docs/articles/index.html b/docs/articles/index.html
index 1eadbf9d..a9c50ea9 100644
--- a/docs/articles/index.html
+++ b/docs/articles/index.html
@@ -128,9 +128,6 @@
     
  • 
       withinVisitDates
     
    • 
-    
  • 
-      measureConditionEraCompleteness
-    
    • 
   
 
  • 
   Changelog