Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[new] Add the YellowFin optimizer (YFOptimizer) #130

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

ruizewang
Copy link

@ruizewang ruizewang commented Jan 19, 2019

Description:
Add the YellowFin optimizer,called YFOptimizer.

YellowFin is an auto-tuning optimizer based on momentum SGD which requires no manual specification of learning rate and momentum. It measures the objective landscape on-the-fly and tunes momentum as well as learning rate using a local quadratic approximation.

We can use YFOptimizer just like Adam and SGD.
For example:
from fastNLP.core.optimizer import YFOptimizer

trainer = Trainer(model=model, n_epochs=100, optimizer=YFOptimizer(lr=0.01), train_data=train_data, dev_data=dev_data, loss=CrossEntropyLoss(), metrics=AccuracyMetric() )

paper: https://arxiv.org/abs/1706.03471
code: https://github.com/JianGoForIt/YellowFin_Pytorch

Main reason: Add a fancy optimizer, which performs well in many tasks, though has its shortcomings. The biggest advantage is that you can use the default parameters directly, no need to adjust LR manually. (Note: there are also some people say they get a bad performance.)

Checklist 检查下面各项是否完成

Please feel free to remove inapplicable items for your PR.

  • The PR title starts with [$CATEGORY] (例如[bugfix]修复bug,[new]添加新功能,[test]修改测试,[rm]删除旧代码)
  • Changes are complete (i.e. I finished coding on this PR) 修改完成才提PR
  • All changes have test coverage 修改的部分顺利通过测试。对于fastnlp/fastnlp/的修改,测试代码必须提供在fastnlp/test/
  • Code is well-documented 注释写好,API文档会从注释中抽取
  • To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change 修改导致例子或tutorial有变化,请找核心开发人员

Changes: 逐项描述修改的内容

  • 在optimizer.py中新引用了两个库math, numpy
  • 添加了一个优化器YellowFin optimizer (YFOptimizer).

Mention: 找人review你的PR

@修改过这个文件的人
@核心开发人员

Add the YellowFin optimizer,called YFOptimizer.

YellowFin is an auto-tuning optimizer based on momentum SGD which requires no manual specification of learning rate and momentum. It measures the objective landscape on-the-fly and tunes momentum as well as learning rate using a local quadratic approximation.

paper: https://arxiv.org/abs/1706.03471
code: https://github.com/JianGoForIt/YellowFin_Pytorch
@codecov-io
Copy link

Codecov Report

Merging #130 into master will decrease coverage by 2.01%.
The diff coverage is 11.25%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master     #130      +/-   ##
==========================================
- Coverage      68%   65.99%   -2.02%     
==========================================
  Files          90       90              
  Lines        6286     6517     +231     
==========================================
+ Hits         4275     4301      +26     
- Misses       2011     2216     +205
Impacted Files Coverage Δ
fastNLP/core/optimizer.py 19.14% <11.25%> (-72.86%) ⬇️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 3fa95b6...d5e8d61. Read the comment docs.

@ruizewang ruizewang changed the title Update optimizer.py [new] Add the YellowFin optimizer (YFOptimizer) Jan 19, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants