You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I searched the issues and found no similar issues.
Component
Other
What happened + What you expected to happen
Getting error messages during ededup section saying there are no input files.
Reproduction script
Download zip from data-prep-kit repo into /Users/dawood/Downloads/data-prep-kit-dev.zip
mkdir /tmp/example
cp data-prep-kit-dev.zip /tmp/example
git clone ...
cd data-prep-kit/examples
make venv
make jupyter
Edit notebook
zip_input_folder = "/tmp/example"
Run notebook through ededup section and get logged messages say no input files
3:26:29 INFO - Running locally
13:26:29 INFO - exact dedup params are {'hash_cpu': 0.5, 'num_hashes': 2, 'doc_column': 'contents'}
13:26:29 INFO - data factory data_ is using local data access: input_folder - test-data/parquet_input output_folder - test-data/ededup_out
13:26:29 INFO - data factory data_ max_files -1, n_sample -1
13:26:29 INFO - data factory data_ Not using data sets, checkpointing False, max files -1, random samples -1, files to use ['.parquet']
13:26:29 INFO - number of workers 3 worker options {'num_cpus': 0.8}
13:26:29 INFO - pipeline id pipeline_id; number workers 3
13:26:29 INFO - job details {'job category': 'preprocessing', 'job name': 'ededup', 'job type': 'ray', 'job id': 'job_id'}
13:26:29 INFO - code location None
13:26:29 INFO - actor creation delay 0
2024-05-14 13:26:31,387 INFO worker.py:1715 -- Started a local Ray instance. View the dashboard at 127.0.0.1:8265
(orchestrate pid=24207) 13:26:32 INFO - orchestrator started at 2024-05-14 13:26:32
(orchestrate pid=24207) 13:26:32 ERROR - No input files to process - exiting
13:26:42 INFO - Completed execution in 0.21104646523793538 min, execution result 0
### Anything else
_No response_
### OS
MacOS (limited support)
### Python
3.10.x
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
The text was updated successfully, but these errors were encountered:
daw3rd
changed the title
[Bug] Example notebook getting finding no input files in ededup
[Bug] Example notebook finding no input files in ededup
May 14, 2024
Search before asking
Component
Other
What happened + What you expected to happen
Getting error messages during ededup section saying there are no input files.
Reproduction script
Download zip from data-prep-kit repo into /Users/dawood/Downloads/data-prep-kit-dev.zip
mkdir /tmp/example cp data-prep-kit-dev.zip /tmp/example git clone ... cd data-prep-kit/examples make venv make jupyter
Edit notebook
Run notebook through ededup section and get logged messages say no input files
The text was updated successfully, but these errors were encountered: