Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add ResNet benchmark #1534

Draft
wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Add ResNet benchmark #1534

wants to merge 4 commits into from

Conversation

laggui
Copy link
Member

@laggui laggui commented Mar 26, 2024

Related Issues/PRs

Closes #1521

Changes

Added ResNet-50 inference in backend-comparison

  • New build script to fetch the resnet-burn implementation and apply a diff patch to be compatible with the current version of Burn on main
  • ResNetBenchmark executes a forward pass on a random input w/ shape [1, 3, 224, 224]
  • Two minor typos fixed

Note: doesn't include autodiff but could be added if required, similar to the current gelu benchmark

Testing

Ran the benchmark with multiple backends

cargo run --release --bin burnbench -- run --benches resnet50 --backends tch-gpu tch-cpu wgpu wgpu-fusion ndarray

Note: it doesn't run on candle because AdaptiveAvgPool2d in the model uses padding which is not supported.

@laggui
Copy link
Member Author

laggui commented Mar 26, 2024

Hmm CI failed but the failing test is unrelated to this PR

/edit: looking for feedback on the benchmark implementation but let's wait until we optimize some stuff before we actually merge it.

This will simply be a helpful tool to evaluate the improvements along the way and once we're satisfied with the results we can merge.

@laggui laggui marked this pull request as draft March 26, 2024 19:04
@syl20bnr
Copy link
Member

LGTM so far, great work!

Copy link

codecov bot commented Mar 26, 2024

Codecov Report

Attention: Patch coverage is 96.77419% with 2 lines in your changes are missing coverage. Please review.

Project coverage is 86.20%. Comparing base (5bac300) to head (cfe65c0).
Report is 96 commits behind head on main.

Files Patch % Lines
backend-comparison/build.rs 98.36% 1 Missing ⚠️
backend-comparison/src/burnbenchapp/base.rs 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #1534      +/-   ##
==========================================
+ Coverage   86.19%   86.20%   +0.01%     
==========================================
  Files         673      674       +1     
  Lines       77545    77606      +61     
==========================================
+ Hits        66837    66898      +61     
  Misses      10708    10708              

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

Copy link
Contributor

This PR has been marked as stale because it has not been updated for over a month

@github-actions github-actions bot added the stale The issue or pr has been open for too long label Apr 28, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
benchmark stale The issue or pr has been open for too long
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add Resnet in backend-comparison
3 participants