Skip to content

Commit

Permalink
Support dict as the input of delete_batch (#30241)
Browse files Browse the repository at this point in the history
This smooths the code path of feeding output of list_prefix()
to delete_batch().

Co-authored-by: Shunping Huang <shunping@google.com>
  • Loading branch information
lostluck and shunping committed Feb 7, 2024
1 parent 435eab3 commit 8978c80
Showing 1 changed file with 3 additions and 1 deletion.
4 changes: 3 additions & 1 deletion sdks/python/apache_beam/io/gcp/gcsio.py
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,8 @@ def delete_batch(self, paths):
"""Deletes the objects at the given GCS paths.
Args:
paths: List of GCS file path patterns in the form gs://<bucket>/<name>,
paths: List of GCS file path patterns or Dict with GCS file path patterns
as keys. The patterns are in the form gs://<bucket>/<name>, but
not to exceed MAX_BATCH_OPERATION_SIZE in length.
Returns: List of tuples of (path, exception) in the same order as the
Expand All @@ -217,6 +218,7 @@ def delete_batch(self, paths):
"""
final_results = []
s = 0
if not isinstance(paths, list): paths = list(iter(paths))
while s < len(paths):
if (s + MAX_BATCH_OPERATION_SIZE) < len(paths):
current_paths = paths[s:s + MAX_BATCH_OPERATION_SIZE]
Expand Down

0 comments on commit 8978c80

Please sign in to comment.