Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

数据分批读取训练问题 #51

Open
shuDaoNan9 opened this issue Mar 5, 2021 · 0 comments
Open

数据分批读取训练问题 #51

shuDaoNan9 opened this issue Mar 5, 2021 · 0 comments
Labels
question Further information is requested

Comments

@shuDaoNan9
Copy link

shuDaoNan9 commented Mar 5, 2021

大数据场景下全量加载到显卡训练显然不现实,鄙人在TF2.4.0中试了下将数据处理好后分批给到DeepMatch模型,发现有以下问题:
1. 通过tf.data.Dataset.from_tensor_slices将数据分批后训练,却发现在tf.compat.v1.disable_eager_execution()后无法使用,报错如下:
D:\Anaconda3\envs\TF2GPU\lib\site-packages\tensorflow\python\keras\backend.py:434: UserWarning: tf.keras.backend.set_learning_phase is deprecated and will be removed after 2020-10-11. To update it, simply pass a True/False value to the training argument of the __call__ method of your layer or model.
warnings.warn('tf.keras.backend.set_learning_phase is deprecated and '
Traceback (most recent call last):
File "F:/python/DeepMatch-master/examples/bp_mind_emr_distr_index_batch.py", line 77, in
for batch_train_model_input,batch_train_label in tf.data.Dataset.from_tensor_slices((test_model_input)).batch(2): #.shuffle(1000):
File "D:\Anaconda3\envs\TF2GPU\lib\site-packages\tensorflow\python\data\ops\dataset_ops.py", line 424, in iter
raise RuntimeError("iter() is only supported inside of tf.function "
RuntimeError: iter() is only supported inside of tf.function or when eager execution is enabled.

2. 通过继承class DataGenerator(keras.utils.Sequence)方式分批读取数据,目前跑单机模式,CPU和GPU利用率上不去

3. 然后尝试了generate_arrays_from_file(data,batch_size=256)分批读取训练,Linux服务器单机正常运行,但CPU和GPU利用率上不去,但win10 GPU运行不行(WIN10报错,后来了解到win10上不能workers=4,use_multiprocessing=True这样多进程)

然后我又看了下DeepCtr那边的code写法,几乎也都是全量数据加载的,没看到分批给到GPU训练,不知路过的各位如何解决此类问题的?

谢谢!

@shuDaoNan9 shuDaoNan9 added the question Further information is requested label Mar 5, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
question Further information is requested
Projects
None yet
Development

No branches or pull requests

1 participant