Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: aws/aws-sdk-pandas
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: 3.7.1
Choose a base ref
...
head repository: aws/aws-sdk-pandas
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: 3.7.2
Choose a head ref
  • 20 commits
  • 94 files changed
  • 6 contributors

Commits on Mar 11, 2024

  1. chore(deps-dev): bump the development-dependencies group with 8 updat… 

    …es (#2717)
    @dependabot
    dependabot[bot] authored Mar 11, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    e8db6e8 View commit details

  2. chore(deps): bump the production-dependencies group with 5 updates (#… 

    …2718)

Bumps the production-dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [botocore](https://github.com/boto/botocore) | `1.34.57` | `1.34.59` |
| [pyarrow](https://github.com/apache/arrow) | `15.0.0` | `15.0.1` |
| [packaging](https://github.com/pypa/packaging) | `23.2` | `24.0` |
| [oracledb](https://github.com/oracle/python-oracledb) | `2.0.1` | `2.1.0` |
| [deltalake](https://github.com/delta-io/delta-rs) | `0.15.3` | `0.16.0` |


Updates `botocore` from 1.34.57 to 1.34.59
- [Changelog](https://github.com/boto/botocore/blob/develop/CHANGELOG.rst)
- [Commits](boto/botocore@1.34.57...1.34.59)

Updates `pyarrow` from 15.0.0 to 15.0.1
- [Commits](apache/arrow@go/v15.0.0...go/v15.0.1)

Updates `packaging` from 23.2 to 24.0
- [Release notes](https://github.com/pypa/packaging/releases)
- [Changelog](https://github.com/pypa/packaging/blob/main/CHANGELOG.rst)
- [Commits](pypa/packaging@23.2...24.0)

Updates `oracledb` from 2.0.1 to 2.1.0
- [Release notes](https://github.com/oracle/python-oracledb/releases)
- [Commits](oracle/python-oracledb@v2.0.1...v2.1.0)

Updates `deltalake` from 0.15.3 to 0.16.0
- [Release notes](https://github.com/delta-io/delta-rs/releases)
- [Changelog](https://github.com/delta-io/delta-rs/blob/main/CHANGELOG.md)
- [Commits](delta-io/delta-rs@python-v0.15.3...rust-v0.16.0)

---
updated-dependencies:
- dependency-name: botocore
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: pyarrow
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: packaging
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: production-dependencies
- dependency-name: oracledb
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
- dependency-name: deltalake
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    @dependabot
    dependabot[bot] authored Mar 11, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    8778f65 View commit details

  3. fix: wr.athena.to_iceberg - Insert query has mismatched column types #… 

    …2678 (#2715)
    @GalVishi
    GalVishi authored Mar 11, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    e2c960b View commit details

Commits on Mar 12, 2024

  1. test: Reduce execution time for the unit tests (#2720)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 12, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    e8a5a96 View commit details

Commits on Mar 14, 2024

  1. test: Add test for S3 access points (#2722)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 14, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    9e33f0b View commit details

  2. test: Add queuing for Neptune bulk load tests (#2724)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 14, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    ef3a419 View commit details

  3. feat: Add support for DeltaLake's DynamoDB lock mechanism (#2705)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 14, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    4418cc4 View commit details

  4. fix: allow s3_output in athena.to_iceberg (#2727) 

    Co-authored-by: Leon Luttenberger <LeonLuttenberger@users.noreply.github.com>
    @jaidisido @LeonLuttenberger
    jaidisido and LeonLuttenberger authored Mar 14, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    0a1cc25 View commit details

Commits on Mar 15, 2024

  1. test: Fix engine switching reset (#2729)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 15, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    b48f45f View commit details

Commits on Mar 18, 2024

  1. chore(deps): bump the production-dependencies group with 3 updates (#… 

    …2732)

Bumps the production-dependencies group with 3 updates: [boto3](https://github.com/boto/boto3), [botocore](https://github.com/boto/botocore) and [setuptools](https://github.com/pypa/setuptools).


Updates `boto3` from 1.34.57 to 1.34.59
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.34.57...1.34.59)

Updates `botocore` from 1.34.59 to 1.34.64
- [Changelog](https://github.com/boto/botocore/blob/develop/CHANGELOG.rst)
- [Commits](boto/botocore@1.34.59...1.34.64)

Updates `setuptools` from 69.1.1 to 69.2.0
- [Release notes](https://github.com/pypa/setuptools/releases)
- [Changelog](https://github.com/pypa/setuptools/blob/main/NEWS.rst)
- [Commits](pypa/setuptools@v69.1.1...v69.2.0)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: botocore
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: setuptools
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    @dependabot
    dependabot[bot] authored Mar 18, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    669d8b6 View commit details

  2. chore(deps-dev): bump the development-dependencies group with 6 updat… 

    …es (#2733)

Bumps the development-dependencies group with 6 updates:

| Package | From | To |
| --- | --- | --- |
| [wheel](https://github.com/pypa/wheel) | `0.42.0` | `0.43.0` |
| [boto3-stubs](https://github.com/youtype/mypy_boto3_builder) | `1.34.59` | `1.34.64` |
| [ruff](https://github.com/astral-sh/ruff) | `0.3.2` | `0.3.3` |
| [pytest-rerunfailures](https://github.com/pytest-dev/pytest-rerunfailures) | `13.0` | `14.0` |
| [bump-my-version](https://github.com/callowayproject/bump-my-version) | `0.18.3` | `0.19.0` |
| [jupyterlab](https://github.com/jupyterlab/jupyterlab) | `4.1.4` | `4.1.5` |


Updates `wheel` from 0.42.0 to 0.43.0
- [Release notes](https://github.com/pypa/wheel/releases)
- [Changelog](https://github.com/pypa/wheel/blob/main/docs/news.rst)
- [Commits](pypa/wheel@0.42.0...0.43.0)

Updates `boto3-stubs` from 1.34.59 to 1.34.64
- [Release notes](https://github.com/youtype/mypy_boto3_builder/releases)
- [Commits](https://github.com/youtype/mypy_boto3_builder/commits)

Updates `ruff` from 0.3.2 to 0.3.3
- [Release notes](https://github.com/astral-sh/ruff/releases)
- [Changelog](https://github.com/astral-sh/ruff/blob/main/CHANGELOG.md)
- [Commits](astral-sh/ruff@v0.3.2...v0.3.3)

Updates `pytest-rerunfailures` from 13.0 to 14.0
- [Changelog](https://github.com/pytest-dev/pytest-rerunfailures/blob/master/CHANGES.rst)
- [Commits](pytest-dev/pytest-rerunfailures@13.0...14.0)

Updates `bump-my-version` from 0.18.3 to 0.19.0
- [Release notes](https://github.com/callowayproject/bump-my-version/releases)
- [Changelog](https://github.com/callowayproject/bump-my-version/blob/master/CHANGELOG.md)
- [Commits](callowayproject/bump-my-version@0.18.3...0.19.0)

Updates `jupyterlab` from 4.1.4 to 4.1.5
- [Release notes](https://github.com/jupyterlab/jupyterlab/releases)
- [Changelog](https://github.com/jupyterlab/jupyterlab/blob/@jupyterlab/lsp@4.1.5/CHANGELOG.md)
- [Commits](https://github.com/jupyterlab/jupyterlab/compare/@jupyterlab/lsp@4.1.4...@jupyterlab/lsp@4.1.5)

---
updated-dependencies:
- dependency-name: wheel
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-dependencies
- dependency-name: boto3-stubs
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-dependencies
- dependency-name: ruff
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-dependencies
- dependency-name: pytest-rerunfailures
  dependency-type: direct:development
  update-type: version-update:semver-major
  dependency-group: development-dependencies
- dependency-name: bump-my-version
  dependency-type: direct:development
  update-type: version-update:semver-minor
  dependency-group: development-dependencies
- dependency-name: jupyterlab
  dependency-type: direct:development
  update-type: version-update:semver-patch
  dependency-group: development-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    @dependabot
    dependabot[bot] authored Mar 18, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    451939f View commit details

Commits on Mar 19, 2024

  1. fix: replace deprecated np.split_array (#2735)

    @jaidisido
    jaidisido authored Mar 19, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    5e140ff View commit details

  2. fix: Athena to_iceberg fails with non-lowercase column names (#2736) 

    Co-authored-by: jaidisido <jaidisido@gmail.com>
    @LeonLuttenberger @jaidisido
    LeonLuttenberger and jaidisido authored Mar 19, 2024
    Loading
    Loading status checks…
    Copy the full SHA
    8dccc8b View commit details

Commits on Mar 21, 2024

  1. chore: Bump to 3.7.2b1 dev version (#2738)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 21, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    f310b36 View commit details

Commits on Mar 25, 2024

  1. chore(deps): bump the github-actions group with 1 update (#2742) 

    Bumps the github-actions group with 1 update: [dependabot/fetch-metadata](https://github.com/dependabot/fetch-metadata).


Updates `dependabot/fetch-metadata` from 1 to 2
- [Release notes](https://github.com/dependabot/fetch-metadata/releases)
- [Commits](dependabot/fetch-metadata@v1...v2)

---
updated-dependencies:
- dependency-name: dependabot/fetch-metadata
  dependency-type: direct:production
  update-type: version-update:semver-major
  dependency-group: github-actions
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    @dependabot
    dependabot[bot] authored Mar 25, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    7d454f8 View commit details

  2. chore(deps): bump the production-dependencies group with 5 updates (#… 

    …2745)

Bumps the production-dependencies group with 5 updates:

| Package | From | To |
| --- | --- | --- |
| [boto3](https://github.com/boto/boto3) | `1.34.59` | `1.34.64` |
| [botocore](https://github.com/boto/botocore) | `1.34.64` | `1.34.69` |
| [pyarrow](https://github.com/apache/arrow) | `15.0.1` | `15.0.2` |
| [oracledb](https://github.com/oracle/python-oracledb) | `2.1.0` | `2.1.1` |
| [deltalake](https://github.com/delta-io/delta-rs) | `0.16.0` | `0.16.2` |


Updates `boto3` from 1.34.59 to 1.34.64
- [Release notes](https://github.com/boto/boto3/releases)
- [Changelog](https://github.com/boto/boto3/blob/develop/CHANGELOG.rst)
- [Commits](boto/boto3@1.34.59...1.34.64)

Updates `botocore` from 1.34.64 to 1.34.69
- [Changelog](https://github.com/boto/botocore/blob/develop/CHANGELOG.rst)
- [Commits](boto/botocore@1.34.64...1.34.69)

Updates `pyarrow` from 15.0.1 to 15.0.2
- [Commits](apache/arrow@go/v15.0.1...go/v15.0.2)

Updates `oracledb` from 2.1.0 to 2.1.1
- [Release notes](https://github.com/oracle/python-oracledb/releases)
- [Commits](oracle/python-oracledb@v2.1.0...v2.1.1)

Updates `deltalake` from 0.16.0 to 0.16.2
- [Release notes](https://github.com/delta-io/delta-rs/releases)
- [Changelog](https://github.com/delta-io/delta-rs/blob/main/CHANGELOG.md)
- [Commits](delta-io/delta-rs@rust-v0.16.0...rust-v0.16.2)

---
updated-dependencies:
- dependency-name: boto3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: botocore
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: pyarrow
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: oracledb
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
- dependency-name: deltalake
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: production-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
    @dependabot
    dependabot[bot] authored Mar 25, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    f22ccef View commit details

Commits on Mar 26, 2024

  1. feat: Ray 2.10 (#2741)

    @kukushking
    kukushking authored Mar 26, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    ce85b46 View commit details

  2. test: Fix quicksight unit tests (#2748)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 26, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    ff445a0 View commit details

  3. chore: Bump version to 3.7.2 (#2746)

    @LeonLuttenberger
    LeonLuttenberger authored Mar 26, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    f05d332 View commit details

Commits on Mar 27, 2024

  1. docs: update layers.rst

    @LeonLuttenberger
    LeonLuttenberger committed Mar 27, 2024
    Loading
    Loading status checks…

    Verified

    This commit was created on GitHub.com and signed with GitHub’s verified signature.
    GPG key ID: B5690EEEBB952194
    Verified
    Learn about vigilant mode
    Copy the full SHA
    cb63a18 View commit details
Showing with 1,092 additions and 969 deletions.
  1. +1 −1 .bumpversion.toml
  2. +1 −1 .github/workflows/dependabot-prs.yml
  3. +40 −40 README.md
  4. +1 −1 VERSION
  5. +1 −1 awswrangler/__metadata__.py
  6. +3 −6 awswrangler/_databases.py
  7. +1 −0 awswrangler/_distributed.py
  8. +1 −0 awswrangler/_sql_formatter.py
  9. +1 −0 awswrangler/_sql_utils.py
  10. +36 −47 awswrangler/_utils.py
  11. +1 −0 awswrangler/athena/_cache.py
  12. +1 −0 awswrangler/athena/_executions.py
  13. +9 −8 awswrangler/athena/_read.py
  14. +1 −0 awswrangler/athena/_spark.py
  15. +12 −1 awswrangler/athena/_write_iceberg.py
  16. +2 −2 awswrangler/catalog/_create.py
  17. +1 −0 awswrangler/catalog/_get.py
  18. +1 −0 awswrangler/catalog/_utils.py
  19. +0 −1 awswrangler/chime.py
  20. +1 −0 awswrangler/cleanrooms/_utils.py
  21. +1 −0 awswrangler/data_api/__init__.py
  22. +1 −0 awswrangler/data_api/_connector.py
  23. +1 −0 awswrangler/data_api/rds.py
  24. +1 −0 awswrangler/data_api/redshift.py
  25. +1 −0 awswrangler/distributed/ray/_core.py
  26. +1 −0 awswrangler/distributed/ray/_register.py
  27. +0 −2 awswrangler/distributed/ray/datasources/__init__.py
  28. +4 −3 awswrangler/distributed/ray/datasources/arrow_csv_datasink.py
  29. +1 −0 awswrangler/distributed/ray/datasources/arrow_csv_datasource.py
  30. +1 −0 awswrangler/distributed/ray/datasources/arrow_json_datasource.py
  31. +4 −3 awswrangler/distributed/ray/datasources/arrow_orc_datasink.py
  32. +1 −0 awswrangler/distributed/ray/datasources/arrow_orc_datasource.py
  33. +1 −0 awswrangler/distributed/ray/datasources/arrow_parquet_base_datasource.py
  34. +34 −9 awswrangler/distributed/ray/datasources/arrow_parquet_datasink.py
  35. +2 −1 awswrangler/distributed/ray/datasources/arrow_parquet_datasource.py
  36. +0 −26 awswrangler/distributed/ray/datasources/block_path_provider.py
  37. +22 −18 awswrangler/distributed/ray/datasources/file_datasink.py
  38. +47 −0 awswrangler/distributed/ray/datasources/filename_provider.py
  39. +8 −7 awswrangler/distributed/ray/datasources/pandas_text_datasink.py
  40. +1 −0 awswrangler/distributed/ray/datasources/pandas_text_datasource.py
  41. +1 −0 awswrangler/distributed/ray/modin/_core.py
  42. +1 −0 awswrangler/distributed/ray/modin/_data_types.py
  43. +1 −0 awswrangler/distributed/ray/modin/_utils.py
  44. +1 −0 awswrangler/distributed/ray/modin/s3/_read_orc.py
  45. +1 −0 awswrangler/distributed/ray/modin/s3/_read_parquet.py
  46. +1 −0 awswrangler/distributed/ray/modin/s3/_read_text.py
  47. +1 −0 awswrangler/distributed/ray/modin/s3/_write_dataset.py
  48. +6 −11 awswrangler/distributed/ray/modin/s3/_write_orc.py
  49. +7 −11 awswrangler/distributed/ray/modin/s3/_write_parquet.py
  50. +4 −25 awswrangler/distributed/ray/modin/s3/_write_text.py
  51. +1 −0 awswrangler/distributed/ray/s3/_list.py
  52. +4 −3 awswrangler/emr.py
  53. +6 −12 awswrangler/mysql.py
  54. +1 −0 awswrangler/neptune/__init__.py
  55. +1 −0 awswrangler/neptune/_gremlin_parser.py
  56. +6 −12 awswrangler/oracle.py
  57. +1 −0 awswrangler/pandas/__init__.py
  58. +6 −12 awswrangler/postgresql.py
  59. +1 −0 awswrangler/redshift/_connect.py
  60. +1 −0 awswrangler/redshift/_utils.py
  61. +1 −0 awswrangler/redshift/_write.py
  62. +2 −2 awswrangler/s3/_read_orc.py
  63. +2 −2 awswrangler/s3/_read_parquet.py
  64. +3 −3 awswrangler/s3/_read_text.py
  65. +1 −0 awswrangler/s3/_read_text_core.py
  66. +32 −9 awswrangler/s3/_write_deltalake.py
  67. +2 −2 awswrangler/s3/_write_orc.py
  68. +3 −3 awswrangler/s3/_write_parquet.py
  69. +6 −6 awswrangler/s3/_write_text.py
  70. +6 −12 awswrangler/sqlserver.py
  71. +0 −1 awswrangler/timestream/__init__.py
  72. +1 −1 docs/source/install.rst
  73. +289 −289 docs/source/layers.rst
  74. +204 −288 poetry.lock
  75. +12 −12 pyproject.toml
  76. +1 −1 test_infra/pyproject.toml
  77. +23 −0 test_infra/stacks/base_stack.py
  78. +10 −0 tests/conftest.py
  79. +5 −5 tests/unit/test_athena.py
  80. +56 −0 tests/unit/test_athena_iceberg.py
  81. +13 −14 tests/unit/test_distributed.py
  82. +1 −1 tests/unit/test_metadata.py
  83. +2 −0 tests/unit/test_neptune.py
  84. +15 −11 tests/unit/test_quicksight.py
  85. +54 −8 tests/unit/test_s3_deltalake.py
  86. +13 −0 tests/unit/test_s3_parquet.py
  87. +1 −0 tox.ini
  88. +10 −10 tutorials/001 - Introduction.ipynb
  89. +15 −15 tutorials/007 - Redshift, MySQL, PostgreSQL, SQL Server, Oracle.ipynb
  90. +3 −3 tutorials/014 - Schema Evolution.ipynb
  91. +1 −1 tutorials/021 - Global Configurations.ipynb
  92. +1 −1 tutorials/022 - Writing Partitions Concurrently.ipynb
  93. +1 −1 tutorials/023 - Flexible Partitions Filter.ipynb
  94. +4 −4 tutorials/030 - Data Api.ipynb
2 changes: 1 addition & 1 deletion .bumpversion.toml
View file
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
[tool.bumpversion]
current_version = "3.7.1"
current_version = "3.7.2"
commit = false
tag = false
tag_name = "{new_version}"
2 changes: 1 addition & 1 deletion .github/workflows/dependabot-prs.yml
View file
Original file line number Diff line number Diff line change
@@ -9,7 +9,7 @@ jobs:
steps:
- name: Fetch Dependabot metadata
id: dependabot-metadata
uses: dependabot/fetch-metadata@v1
uses: dependabot/fetch-metadata@v2
with:
alert-lookup: true
compat-lookup: true
80 changes: 40 additions & 40 deletions README.md
View file
Original file line number Diff line number Diff line change
@@ -100,27 +100,27 @@ FROM "sampleDB"."sampleTable" ORDER BY time DESC LIMIT 3
## At scale
AWS SDK for pandas can also run your workflows at scale by leveraging [Modin](https://modin.readthedocs.io/en/stable/) and [Ray](https://www.ray.io/). Both projects aim to speed up data workloads by distributing processing over a cluster of workers.

The quickest way to get started is to use AWS Glue with Ray. Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/scale.html), our blogs ([1](https://aws.amazon.com/blogs/big-data/scale-aws-sdk-for-pandas-workloads-with-aws-glue-for-ray/)/[2](https://aws.amazon.com/blogs/big-data/advanced-patterns-with-aws-sdk-for-pandas-on-aws-glue-for-ray/)), or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to discover even more features.
The quickest way to get started is to use AWS Glue with Ray. Read our [docs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html), our blogs ([1](https://aws.amazon.com/blogs/big-data/scale-aws-sdk-for-pandas-workloads-with-aws-glue-for-ray/)/[2](https://aws.amazon.com/blogs/big-data/advanced-patterns-with-aws-sdk-for-pandas-on-aws-glue-for-ray/)), or head to our latest [tutorials](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials) to discover even more features.

> ⚠️ **Ray is currently not available for Python 3.12. While AWS SDK for pandas supports Python 3.12, it cannot be used at scale.**
## [Read The Docs](https://aws-sdk-pandas.readthedocs.io/)

- [**What is AWS SDK for pandas?**](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/about.html)
- [**Install**](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html)
- [PyPi (pip)](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#pypi-pip)
- [Conda](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#conda)
- [AWS Lambda Layer](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#aws-lambda-layer)
- [AWS Glue Python Shell Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#aws-glue-python-shell-jobs)
- [AWS Glue PySpark Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#aws-glue-pyspark-jobs)
- [Amazon SageMaker Notebook](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#amazon-sagemaker-notebook)
- [Amazon SageMaker Notebook Lifecycle](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#amazon-sagemaker-notebook-lifecycle)
- [EMR](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#emr)
- [From source](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/install.html#from-source)
- [**At scale**](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/scale.html)
- [Getting Started](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/scale.html#getting-started)
- [Supported APIs](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/scale.html#supported-apis)
- [Resources](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/scale.html#resources)
- [**What is AWS SDK for pandas?**](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/about.html)
- [**Install**](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html)
- [PyPi (pip)](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#pypi-pip)
- [Conda](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#conda)
- [AWS Lambda Layer](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#aws-lambda-layer)
- [AWS Glue Python Shell Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#aws-glue-python-shell-jobs)
- [AWS Glue PySpark Jobs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#aws-glue-pyspark-jobs)
- [Amazon SageMaker Notebook](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#amazon-sagemaker-notebook)
- [Amazon SageMaker Notebook Lifecycle](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#amazon-sagemaker-notebook-lifecycle)
- [EMR](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#emr)
- [From source](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/install.html#from-source)
- [**At scale**](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html)
- [Getting Started](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html#getting-started)
- [Supported APIs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html#supported-apis)
- [Resources](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/scale.html#resources)
- [**Tutorials**](https://github.com/aws/aws-sdk-pandas/tree/main/tutorials)
- [001 - Introduction](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/001%20-%20Introduction.ipynb)
- [002 - Sessions](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/002%20-%20Sessions.ipynb)
@@ -162,30 +162,30 @@ The quickest way to get started is to use AWS Glue with Ray. Read our [docs](htt
- [039 - Athena Iceberg](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/039%20-%20Athena%20Iceberg.ipynb)
- [040 - EMR Serverless](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/040%20-%20EMR%20Serverless.ipynb)
- [041 - Apache Spark on Amazon Athena](https://github.com/aws/aws-sdk-pandas/blob/main/tutorials/041%20-%20Apache%20Spark%20on%20Amazon%20Athena.ipynb)
- [**API Reference**](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html)
- [Amazon S3](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-s3)
- [AWS Glue Catalog](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#aws-glue-catalog)
- [Amazon Athena](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-athena)
- [Amazon Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-redshift)
- [PostgreSQL](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#postgresql)
- [MySQL](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#mysql)
- [SQL Server](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#sqlserver)
- [Oracle](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#oracle)
- [Data API Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#data-api-redshift)
- [Data API RDS](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#data-api-rds)
- [OpenSearch](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#opensearch)
- [AWS Glue Data Quality](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#aws-glue-data-quality)
- [Amazon Neptune](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-neptune)
- [DynamoDB](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#dynamodb)
- [Amazon Timestream](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-timestream)
- [Amazon EMR](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-emr)
- [Amazon CloudWatch Logs](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-cloudwatch-logs)
- [Amazon Chime](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-chime)
- [Amazon QuickSight](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#amazon-quicksight)
- [AWS STS](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#aws-sts)
- [AWS Secrets Manager](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#aws-secrets-manager)
- [Global Configurations](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#global-configurations)
- [Distributed - Ray](https://aws-sdk-pandas.readthedocs.io/en/3.7.1/api.html#distributed-ray)
- [**API Reference**](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html)
- [Amazon S3](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-s3)
- [AWS Glue Catalog](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#aws-glue-catalog)
- [Amazon Athena](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-athena)
- [Amazon Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-redshift)
- [PostgreSQL](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#postgresql)
- [MySQL](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#mysql)
- [SQL Server](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#sqlserver)
- [Oracle](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#oracle)
- [Data API Redshift](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#data-api-redshift)
- [Data API RDS](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#data-api-rds)
- [OpenSearch](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#opensearch)
- [AWS Glue Data Quality](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#aws-glue-data-quality)
- [Amazon Neptune](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-neptune)
- [DynamoDB](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#dynamodb)
- [Amazon Timestream](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-timestream)
- [Amazon EMR](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-emr)
- [Amazon CloudWatch Logs](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-cloudwatch-logs)
- [Amazon Chime](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-chime)
- [Amazon QuickSight](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#amazon-quicksight)
- [AWS STS](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#aws-sts)
- [AWS Secrets Manager](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#aws-secrets-manager)
- [Global Configurations](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#global-configurations)
- [Distributed - Ray](https://aws-sdk-pandas.readthedocs.io/en/3.7.2/api.html#distributed-ray)
- [**License**](https://github.com/aws/aws-sdk-pandas/blob/main/LICENSE.txt)
- [**Contributing**](https://github.com/aws/aws-sdk-pandas/blob/main/CONTRIBUTING.md)

2 changes: 1 addition & 1 deletion VERSION
View file
Original file line number Diff line number Diff line change
@@ -1 +1 @@
3.7.1
3.7.2
2 changes: 1 addition & 1 deletion awswrangler/__metadata__.py
View file
Original file line number Diff line number Diff line change
@@ -7,5 +7,5 @@

__title__: str = "awswrangler"
__description__: str = "Pandas on AWS."
__version__: str = "3.7.1"
__version__: str = "3.7.2"
__license__: str = "Apache License 2.0"
9 changes: 3 additions & 6 deletions awswrangler/_databases.py
View file
Original file line number Diff line number Diff line change
@@ -279,8 +279,7 @@ def read_sql_query(
safe: bool = ...,
timestamp_as_object: bool = ...,
dtype_backend: Literal["numpy_nullable", "pyarrow"] = ...,
) -> pd.DataFrame:
...
) -> pd.DataFrame: ...


@overload
@@ -295,8 +294,7 @@ def read_sql_query(
safe: bool = ...,
timestamp_as_object: bool = ...,
dtype_backend: Literal["numpy_nullable", "pyarrow"] = ...,
) -> Iterator[pd.DataFrame]:
...
) -> Iterator[pd.DataFrame]: ...


@overload
@@ -311,8 +309,7 @@ def read_sql_query(
safe: bool = ...,
timestamp_as_object: bool = ...,
dtype_backend: Literal["numpy_nullable", "pyarrow"] = ...,
) -> pd.DataFrame | Iterator[pd.DataFrame]:
...
) -> pd.DataFrame | Iterator[pd.DataFrame]: ...


def read_sql_query(
1 change: 1 addition & 0 deletions awswrangler/_distributed.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""Distributed engine and memory format configuration."""

from __future__ import annotations

import importlib.util
1 change: 1 addition & 0 deletions awswrangler/_sql_formatter.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""Formatting logic for SQL parameters."""

from __future__ import annotations

import datetime
1 change: 1 addition & 0 deletions awswrangler/_sql_utils.py
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
"""SQL utilities."""

import re

from awswrangler import exceptions
Loading