Skip to content

Commit

Permalink
Merge branch 'main' into test-186
Browse files Browse the repository at this point in the history
  • Loading branch information
colin-rogers-dbt committed Oct 31, 2024
2 parents 0205e32 + 5812a2e commit 887e5db
Show file tree
Hide file tree
Showing 27 changed files with 356 additions and 61 deletions.
1 change: 1 addition & 0 deletions .changes/1.10.3.md
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
## dbt-adapters 1.10.3 - October 29, 2024
5 changes: 5 additions & 0 deletions .changes/1.7.1.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
## dbt-adapters 1.7.1 - October 15, 2024

### Features

- Enable setting current value of dbt_valid_to ([#320](https://github.com/dbt-labs/dbt-adapters/issues/320))
16 changes: 16 additions & 0 deletions .changes/1.7.2.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
## dbt-adapters 1.7.2 - October 21, 2024

### Breaking Changes

- Drop support for Python 3.8 ([#332](https://github.com/dbt-labs/dbt-adapters/issues/332))

### Features

- Allows unique_key for snapshots to take a list ([#181](https://github.com/dbt-labs/dbt-adapters/issues/181))

### Fixes

- Always validate an incremental model's `incremental_strategy` ([#330](https://github.com/dbt-labs/dbt-adapters/issues/330))

### Contributors
- [@agpapa](https://github.com/agpapa) ([#181](https://github.com/dbt-labs/dbt-adapters/issues/181))
9 changes: 9 additions & 0 deletions .changes/1.8.0.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
## dbt-adapters 1.8.0 - October 29, 2024

### Fixes

- Always make behavior flags available for evaluation ([#338](https://github.com/dbt-labs/dbt-adapters/issues/338))

### Under the Hood

- Add adapter telemetry. ([#301](https://github.com/dbt-labs/dbt-adapters/issues/301))
6 changes: 0 additions & 6 deletions .changes/unreleased/Features-20240927-134248.yaml

This file was deleted.

2 changes: 1 addition & 1 deletion .github/actions/publish-results/action.yml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@ inputs:
description: File type for file name stub (e.g. "unit-tests")
required: true
python-version:
description: Python version for the file name stub (e.g. "3.8")
description: Python version for the file name stub (e.g. "3.9")
required: true
source-file:
description: File to be uploaded
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/unit-tests.yml
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,7 @@ jobs:
strategy:
fail-fast: false
matrix:
python-version: ["3.8", "3.9", "3.10", "3.11", "3.12"]
python-version: ["3.9", "3.10", "3.11", "3.12"]

steps:
- name: Check out repository
Expand Down
8 changes: 4 additions & 4 deletions .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,19 @@ repos:
- id: dbt-core-in-adapters-check

- repo: https://github.com/psf/black
rev: 24.4.0
rev: 24.8.0
hooks:
- id: black
args:
- --line-length=99
- --target-version=py38
- --target-version=py39
- --target-version=py310
- --target-version=py311
- --target-version=py312
- --force-exclude=dbt/adapters/events/adapter_types_pb2.py

- repo: https://github.com/pycqa/flake8
rev: 7.0.0
rev: 7.1.1
hooks:
- id: flake8
exclude: dbt/adapters/events/adapter_types_pb2.py|tests/functional/
Expand All @@ -41,7 +41,7 @@ repos:
- --per-file-ignores=*/__init__.py:F401,*/conftest.py:F401

- repo: https://github.com/pre-commit/mirrors-mypy
rev: v1.9.0
rev: v1.11.2
hooks:
- id: mypy
exclude: dbt/adapters/events/adapter_types_pb2.py|dbt-tests-adapter/dbt/__init__.py
Expand Down
39 changes: 37 additions & 2 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,18 +5,53 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html),
and is generated by [Changie](https://github.com/miniscruff/changie).

## dbt-adapters 1.10.3 - October 29, 2024



## dbt-adapters 1.10.2 - October 01, 2024

### Under the Hood

- dbt-tests-adapters: Add required begin to microbatch model config to BaseMicrobatch test ([#315](https://github.com/dbt-labs/dbt-adapters/issues/315))



## dbt-adapters 1.10.1 - September 16, 2024

## dbt-adapters 1.10.0 - September 12, 2024

## dbt-adapters 1.8.0 - October 29, 2024

### Fixes

- Always make behavior flags available for evaluation ([#338](https://github.com/dbt-labs/dbt-adapters/issues/338))

### Under the Hood

- Add adapter telemetry. ([#301](https://github.com/dbt-labs/dbt-adapters/issues/301))

## dbt-adapters 1.7.2 - October 21, 2024

### Breaking Changes

- Drop support for Python 3.8 ([#332](https://github.com/dbt-labs/dbt-adapters/issues/332))

### Features

- Allows unique_key for snapshots to take a list ([#181](https://github.com/dbt-labs/dbt-adapters/issues/181))

### Fixes

- Always validate an incremental model's `incremental_strategy` ([#330](https://github.com/dbt-labs/dbt-adapters/issues/330))

### Contributors
- [@agpapa](https://github.com/agpapa) ([#181](https://github.com/dbt-labs/dbt-adapters/issues/181))

## dbt-adapters 1.7.1 - October 15, 2024

### Features

- Enable setting current value of dbt_valid_to ([#320](https://github.com/dbt-labs/dbt-adapters/issues/320))

## dbt-adapters 1.7.0 - September 19, 2024

### Features
Expand Down
2 changes: 1 addition & 1 deletion dbt-tests-adapter/dbt/tests/__about__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version = "1.10.2"
version = "1.10.3"
7 changes: 7 additions & 0 deletions dbt-tests-adapter/dbt/tests/adapter/basic/files.py
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,10 @@
{{ config(materialized="incremental") }}
"""

config_materialized_incremental_invalid_strategy = """
{{ config(materialized="incremental", incremental_strategy="bad_strategy") }}
"""

config_materialized_var = """
{{ config(materialized=var("materialized_var", "table"))}}
"""
Expand Down Expand Up @@ -217,3 +221,6 @@
ephemeral_view_sql = config_materialized_view + model_ephemeral
ephemeral_table_sql = config_materialized_table + model_ephemeral
incremental_sql = config_materialized_incremental + model_incremental
incremental_invalid_strategy_sql = (
config_materialized_incremental_invalid_strategy + model_incremental
)
39 changes: 39 additions & 0 deletions dbt-tests-adapter/dbt/tests/adapter/basic/test_incremental.py
Original file line number Diff line number Diff line change
Expand Up @@ -86,6 +86,45 @@ def test_incremental_not_schema_change(self, project):
assert run_result == RunStatus.Success


class BaseIncrementalBadStrategy:
@pytest.fixture(scope="class")
def project_config_update(self):
return {"name": "incremental"}

@pytest.fixture(scope="class")
def models(self):
return {
"incremental.sql": files.incremental_invalid_strategy_sql,
"schema.yml": files.schema_base_yml,
}

@pytest.fixture(scope="class")
def seeds(self):
return {"base.csv": files.seeds_base_csv, "added.csv": files.seeds_added_csv}

@pytest.fixture(autouse=True)
def clean_up(self, project):
yield
with project.adapter.connection_named("__test"):
relation = project.adapter.Relation.create(
database=project.database, schema=project.test_schema
)
project.adapter.drop_schema(relation)

def test_incremental_invalid_strategy(self, project):
# seed command
results = run_dbt(["seed"])
assert len(results) == 2

# try to run the incremental model, it should fail on the first attempt
results = run_dbt(["run"], expect_pass=False)
assert len(results.results) == 1
assert (
'dbt could not find an incremental strategy macro with the name "get_incremental_bad_strategy_sql"'
in results.results[0].message
)


class Testincremental(BaseIncremental):
pass

Expand Down
3 changes: 1 addition & 2 deletions dbt-tests-adapter/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ name = "dbt-tests-adapter"
description = "The set of reusable tests and test fixtures used to test common functionality"
readme = "README.md"
keywords = ["dbt", "adapter", "adapters", "database", "elt", "dbt-core", "dbt Core", "dbt Cloud", "dbt Labs"]
requires-python = ">=3.8.0"
requires-python = ">=3.9.0"
authors = [
{ name = "dbt Labs", email = "info@dbtlabs.com" },
]
Expand All @@ -17,7 +17,6 @@ classifiers = [
"Operating System :: MacOS :: MacOS X",
"Operating System :: Microsoft :: Windows",
"Operating System :: POSIX :: Linux",
"Programming Language :: Python :: 3.8",
"Programming Language :: Python :: 3.9",
"Programming Language :: Python :: 3.10",
"Programming Language :: Python :: 3.11",
Expand Down
2 changes: 1 addition & 1 deletion dbt/adapters/__about__.py
Original file line number Diff line number Diff line change
@@ -1 +1 @@
version = "1.7.0"
version = "1.8.0"
1 change: 1 addition & 0 deletions dbt/adapters/base/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,5 @@
BaseRelation,
RelationType,
SchemaSearchMap,
AdapterTrackingRelationInfo,
)
40 changes: 34 additions & 6 deletions dbt/adapters/base/impl.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from contextlib import contextmanager
from datetime import datetime
from enum import Enum
from importlib import import_module
from multiprocessing.context import SpawnContext
from typing import (
Any,
Expand Down Expand Up @@ -61,12 +62,14 @@
ComponentName,
InformationSchema,
SchemaSearchMap,
AdapterTrackingRelationInfo,
)
from dbt.adapters.cache import RelationsCache, _make_ref_key_dict
from dbt.adapters.capability import Capability, CapabilityDict
from dbt.adapters.contracts.connection import Credentials
from dbt.adapters.contracts.macros import MacroResolverProtocol
from dbt.adapters.contracts.relation import RelationConfig

from dbt.adapters.events.types import (
CacheMiss,
CatalogGenerationError,
Expand Down Expand Up @@ -300,12 +303,13 @@ def behavior(self) -> Behavior:
@behavior.setter # type: ignore
def behavior(self, flags: List[BehaviorFlag]) -> None:
flags.extend(self._behavior_flags)
try:
# we don't always get project flags, for example during `dbt debug`
self._behavior = Behavior(flags, self.config.flags)
except AttributeError:
# in that case, don't load any behavior to avoid unexpected defaults
self._behavior = Behavior([], {})

# we don't always get project flags, for example, the project file is not loaded during `dbt debug`
# in that case, load the default values for behavior flags to avoid compilation errors
# this mimics not loading a project file, or not specifying flags in a project file
user_overrides = getattr(self.config, "flags", {})

self._behavior = Behavior(flags, user_overrides)

@property
def _behavior_flags(self) -> List[BehaviorFlag]:
Expand Down Expand Up @@ -1743,6 +1747,30 @@ def capabilities(cls) -> CapabilityDict:
def supports(cls, capability: Capability) -> bool:
return bool(cls.capabilities()[capability])

@classmethod
def get_adapter_run_info(cls, config: RelationConfig) -> AdapterTrackingRelationInfo:
adapter_class_name, *_ = cls.__name__.split("Adapter")
adapter_name = adapter_class_name.lower()

if adapter_name == "base":
adapter_version = ""
else:
adapter_version = import_module(f"dbt.adapters.{adapter_name}.__version__").version

return AdapterTrackingRelationInfo(
adapter_name=adapter_name,
base_adapter_version=import_module("dbt.adapters.__about__").version,
adapter_version=adapter_version,
model_adapter_details=cls._get_adapter_specific_run_info(config),
)

@classmethod
def _get_adapter_specific_run_info(cls, config) -> Dict[str, Any]:
"""
Adapter maintainers should overwrite this method to return any run metadata that should be captured during a run.
"""
return {}


COLUMNS_EQUAL_SQL = """
with diff_count as (
Expand Down
19 changes: 19 additions & 0 deletions dbt/adapters/base/relation.py
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
Dict,
FrozenSet,
Iterator,
List,
Optional,
Set,
Tuple,
Expand Down Expand Up @@ -341,6 +342,16 @@ def create(
)
return cls.from_dict(kwargs)

@classmethod
def scd_args(cls: Type[Self], primary_key: Union[str, List[str]], updated_at) -> List[str]:
scd_args = []
if isinstance(primary_key, list):
scd_args.extend(primary_key)
else:
scd_args.append(primary_key)
scd_args.append(updated_at)
return scd_args

@property
def can_be_renamed(self) -> bool:
return self.type in self.renameable_relations
Expand Down Expand Up @@ -531,3 +542,11 @@ def flatten(self, allow_multiple_databases: bool = False) -> "SchemaSearchMap":
)

return new


@dataclass(frozen=True, eq=False, repr=False)
class AdapterTrackingRelationInfo(FakeAPIObject, Hashable):
adapter_name: str
base_adapter_version: str
adapter_version: str
model_adapter_details: Any
1 change: 1 addition & 0 deletions dbt/adapters/contracts/connection.py
Original file line number Diff line number Diff line change
Expand Up @@ -226,3 +226,4 @@ class AdapterRequiredConfig(HasCredentials, Protocol):
cli_vars: Dict[str, Any]
target_path: str
log_cache_events: bool
catalogs = Optional[ExternalCatalogConfig]
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,9 @@

{% set to_drop = [] %}

{% set incremental_strategy = config.get('incremental_strategy') or 'default' %}
{% set strategy_sql_macro_func = adapter.get_incremental_strategy_macro(context, incremental_strategy) %}

{% if existing_relation is none %}
{% set build_sql = get_create_table_as_sql(False, target_relation, sql) %}
{% elif full_refresh_mode %}
Expand All @@ -52,9 +55,7 @@
{% endif %}

{#-- Get the incremental_strategy, the macro to use for the strategy, and build the sql --#}
{% set incremental_strategy = config.get('incremental_strategy') or 'default' %}
{% set incremental_predicates = config.get('predicates', none) or config.get('incremental_predicates', none) %}
{% set strategy_sql_macro_func = adapter.get_incremental_strategy_macro(context, incremental_strategy) %}
{% set strategy_arg_dict = ({'target_relation': target_relation, 'temp_relation': temp_relation, 'unique_key': unique_key, 'dest_columns': dest_columns, 'incremental_predicates': incremental_predicates }) %}
{% set build_sql = strategy_sql_macro_func(strategy_arg_dict) %}

Expand Down
Loading

0 comments on commit 887e5db

Please sign in to comment.