Support auto-setting AWS credentials for storage options

[milesgranger]

Will close #72

Also fix #74 along the way.

In [1]: import dask_deltatable as ddt
   ...: import dask.dataframe as dd
   ...: import pandas as pd
   ...:
   ...: df = pd.DataFrame({"x": range(100), "y": pd.Series(range(100)) // 2})
   ...: ddf = dd.from_pandas(df, npartitions=4)
   ...:
   ...: ddt.to_deltalake('s3://dask-deltatable-ci/test-delta', ddf)
   ...: read = ddt.read_deltalake('s3://dask-deltatable-ci/test-delta')

In [2]: read
Out[2]:
Dask DataFrame Structure:
                   x      y
npartitions=4
               int64  int64
                 ...    ...
                 ...    ...
                 ...    ...
                 ...    ...
Dask Name: to_string_dtype, 2 expressions
Expr=ArrowStringConversion(frame=FromMapProjectable(cdd6cff))

@mrocklin, I haven't added tests for this yet. I'm curious if you prefer integration type tests doing actual S3 writes to a protected bucket, so we could add some limited credentials here for like the s3://dask-deltalake-ci bucket. Or if you think mocking it is okay.

[codecov-commenter]

Codecov Report

Attention: Patch coverage is 52.63158% with 18 lines in your changes are missing coverage. Please review.

Project coverage is 71.62%. Comparing base (cbe085a) to head (14e4c93).
Report is 1 commits behind head on main.

Files	Patch %	Lines
dask_deltatable/utils.py	26.08%	17 Missing ⚠️
dask_deltatable/core.py	85.71%	1 Missing ⚠️

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #78      +/-   ##
==========================================
- Coverage   74.06%   71.62%   -2.44%     
==========================================
  Files           6        6              
  Lines         320      356      +36     
==========================================
+ Hits          237      255      +18     
- Misses         83      101      +18     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mrocklin]

No personal preference on testing. Deferring to you all

…

On Fri, Mar 15, 2024 at 6:18 AM Codecov Comments Bot < ***@***.***> wrote: Codecov <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> Report Attention: Patch coverage is 52.63158% with 18 lines in your changes are missing coverage. Please review. Project coverage is 71.62%. Comparing base (cbe085a) <https://app.codecov.io/gh/dask-contrib/dask-deltatable/commit/cbe085a9a7acc66be782024d64df84abc175995e?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> to head (14e4c93) <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?dropdown=coverage&src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> . Report is 1 commits behind head on main. Files <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> Patch % Lines dask_deltatable/utils.py <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib#diff-ZGFza19kZWx0YXRhYmxlL3V0aWxzLnB5> 26.08% 17 Missing ⚠️ <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> dask_deltatable/core.py <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib#diff-ZGFza19kZWx0YXRhYmxlL2NvcmUucHk=> 85.71% 1 Missing ⚠️ <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> ❗ Your organization needs to install the Codecov GitHub app <https://github.com/apps/codecov/installations/select_target> to enable full functionality. Additional details and impacted files @@ Coverage Diff @@## main #78 +/- ## ==========================================- Coverage 74.06% 71.62% -2.44% ========================================== Files 6 6 Lines 320 356 +36 ==========================================+ Hits 237 255 +18 - Misses 83 101 +18 ☔ View full report in Codecov by Sentry <https://app.codecov.io/gh/dask-contrib/dask-deltatable/pull/78?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> . 📢 Have feedback on the report? Share it here <https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=dask-contrib> . — Reply to this email directly, view it on GitHub <#78 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AACKZTGGWX5KDGX6QYL3XFTYYLKIZAVCNFSM6AAAAABEXZLOBOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSOJZGQ2DINZZGM> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[jacobtomlinson]

It's probably best if tests don't rely on external resources. They'll just end up brittle or skipped for most folks. Given how well defined the S3 API is I think mocking is a good approach here.