Add documentation to burn core nn #1746

jwric · 2024-05-07T21:18:53Z

Pull Request Template

Checklist

Confirmed that run-checks all script has been executed.
Made sure the book is up to date with changes in this PR.

Related Issues/PRs

Related to #594

Changes

Revised rust documentation for burn core neural network modules.

Added links between the struct and the config. Added a link to the related burn_tensor function in the documentation for the forward function.

Removing the formula for relu from the module API to the functional API, citing a paper relevant to relu and mentionning the functional API in the module API

Adding documentation to the Linear module mentionning that LinearConfig struct should be used when creating a Linear Layer Also adding links to the documentation that points people toward the right path

Added links between the struct and the config. Added a link to the struct in the forward function for more info.

Adding documentation stating the RotaryEncoding should be created using a RotaryEncodingConfig

Adding documentation to the prelu module: - Linking forward function documentation to the functional API - Citing the first paper to mention prelu - Adding documentation saying that prelu layer should be created using PReluConfig

Added links for more info. Added shape info at some places.

Provide documentation for the Gru module, including its configuration and usage. Include a link to the paper that introduced the Gated Recurrent Unit (GRU) and specify that the module should be created using GruConfig. Also, mention that the forward function returns a state tensor with specific dimensions.

Adding documentation: - Says to use config to create the layers - Add mathematical formula to the pwff forward pass - Add citation in the pwff to the "Attention is all you need" paper

Provide documentation for the Lstm and BiLstm modules, including their configurations and usage. Include links to the papers that introduced Long Short-Term Memory (LSTM) and Bidirectional LSTM. Specify that the modules should be created using LstmConfig and BiLstmConfig respectively.

…dules

Adding documentation stating to use the config to create the layer

…odules Added references to the burn_tensor associated functions. Added links between the struct and the config.

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

Added references to the burn_tensor associated functions. Added links between the struct and the config.

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

Added references to the burn_tensor associated functions. Added links between the struct and the config.

Added references to the burn_tensor associated functions. Added links between the struct and the config. Removed the backend generic from the config since it's not needed (might be a breaking change).

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

Changed burn_tensor imports to crate::tensor

…ig initialization

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

laggui

Thanks for the major clean-up 🧹 🙏

Some minor comments to address, mainly about the form to be consistent in the changes.

laggui · 2024-05-09T12:15:45Z

crates/burn-core/src/nn/attention/mha.rs

+    // Shape `[batch_size, seq_length_1, d_model]`
    query: Tensor<B, 3>,
+    // Shape `[batch_size, seq_length_2, d_model]`
    key: Tensor<B, 3>,
+    // Shape `[batch_size, seq_length_2, d_model]`


To document struct fields we use a line doc (three forward slashes ///).

laggui · 2024-05-09T12:17:58Z

crates/burn-core/src/nn/conv/conv1d.rs

-use super::checks;
-
-/// Configuration to create an [1D convolution](Conv1d) layer.
+/// Configuration to create an [1D convolution](Conv1d) layer using the [init function](Conv1dConfig::init).


This was already there before.. but if we're fixing the docs it should be "to create a 1D convolution"

laggui · 2024-05-09T12:20:36Z

crates/burn-core/src/nn/conv/conv2d.rs


-/// Configuration to create an [2D convolution](Conv2d) layer.
+/// Configuration to create an [2D convolution](Conv2d) layer, using the [init function](Conv2dConfig::init).


Same as with the conv1d: should be "to create a 2D convolution".

Also, there's no need for the comma in ", using the init function."

laggui · 2024-05-09T12:24:58Z

crates/burn-core/src/nn/leaky_relu.rs

-#[derive(Module, Debug)]
-pub struct LeakyRelu<B: Backend> {
+///
+/// Should be created with [LeakyReluConfig](LeakyReluConfig).
+#[derive(Module, Clone, Debug)]
+pub struct LeakyRelu {
    /// The negative slope.
    pub negative_slope: f64,
-    phantom: PhantomData<B>,
 }


laggui · 2024-05-09T12:31:47Z

crates/burn-core/src/nn/norm/batch.rs

+/// - `X` is the input tensor
+/// - `norm` is the normalization function
+/// - `γ` is the learnable weight
+/// - `β` is the learnable bias
+/// - `Y` is the output tensor


For documentation purposes maybe have X and Y together, either at the end or top of the enumeration:

X is the input tensor

Y is the output tensor

norm is the normalization function

γ is the learnable weight

β is the learnable bias

laggui · 2024-05-09T12:44:14Z

crates/burn-core/src/nn/pool/max_pool2d.rs

-/// Configuration to create an [2D max pooling](MaxPool2d) layer.
+use crate::tensor::module::max_pool2d;
+
+/// Configuration to create an [2D max pooling](MaxPool2d) layer using the [init function](MaxPool2dConfig::init).


Was already there, but should be: "to create a 2D max pooling"..

laggui · 2024-05-09T12:44:50Z

crates/burn-core/src/nn/pos_encoding.rs


 #[cfg(not(feature = "std"))]
 use num_traits::Float;

-/// Configuration to create an [PositionalEncoding](PositionalEncoding) layer.
+/// Configuration to create an [PositionalEncoding](PositionalEncoding) layer using the [init function](PositionalEncodingConfig::init).


Same comment. Should be: "to create a PositionalEncoding"..

laggui · 2024-05-09T12:50:53Z

crates/burn-core/src/nn/rnn/lstm.rs

@@ -186,6 +190,10 @@ pub struct BiLstmConfig {
 }

 /// The BiLstm module. This implementation is for Bidirectional LSTM.
+///
+/// Introduced in the paper: [Framewise phoneme classification with bidirectional LSTM and other neural network architectures](https://www.cs.toronto.edu/~graves/ijcnn_2005.pdf).


Nice literature review 😄 I wasn't aware of the origins for bidirectional LSTM

laggui · 2024-05-09T12:59:42Z

crates/burn-core/src/nn/transformer/decoder.rs

@@ -203,7 +205,7 @@ impl<B: Backend> TransformerDecoderLayer<B> {
            norm_first: config.norm_first,
        }
    }
-
+    /// forward pass for the TransformerDecoder layer


Let's keep a similar capitalization scheme for the line docs.

Suggestion:
Applies the TransformerDecorder forward pass to the input tensor.

laggui · 2024-05-09T13:04:56Z

crates/burn-core/src/nn/transformer/pwff.rs

 ///
 /// # Params
 ///
 /// - linear inner: Linear layer with `d_model` input features and `d_ff` output features.
 /// - linear outer: Linear layer with `d_ff` input features and `d_model` output features.
+///
+/// FFN(x) = max(0, xW1 + b1)W2 + b2


Keep the math in ticks?

FFN(x) = max(0, xW1 + b1)W2 + b2

ThierryCantin-Demers and others added 30 commits May 7, 2024 13:34

Updated documentation for unfold4d

d618c5a

Added links between the struct and the config. Added a link to the related burn_tensor function in the documentation for the forward function.

Changing nn relu module documentation to functional api

098dc9b

Removing the formula for relu from the module API to the functional API, citing a paper relevant to relu and mentionning the functional API in the module API

Linking gelu module API documentation to functional API documentation

2e19b1b

Linear module : adding documentation

7c72ed4

Adding documentation to the Linear module mentionning that LinearConfig struct should be used when creating a Linear Layer Also adding links to the documentation that points people toward the right path

Updated documentation for dropout

e465c2f

Added links between the struct and the config. Added a link to the struct in the forward function for more info.

embedding + swiglu

71244e7

RotaryEncodying : adding documentation

99f9c9c

Adding documentation stating the RotaryEncoding should be created using a RotaryEncodingConfig

prelu: adding documentation

e89d078

Adding documentation to the prelu module: - Linking forward function documentation to the functional API - Citing the first paper to mention prelu - Adding documentation saying that prelu layer should be created using PReluConfig

pos_encoding: adding documentation

f8468b9

Updated documentation for mha

cfc9da9

Added links for more info. Added shape info at some places.

burn-core-nn-transformers: adding documentation

27ea5e3

Adding documentation: - Says to use config to create the layers - Add mathematical formula to the pwff forward pass - Add citation in the pwff to the "Attention is all you need" paper

Updated documentation: ConvTranspose1d and ConvTranspose2d

0c54631

docs: Update documentation for ConvTranspose1d and ConvTranspose2d mo…

fec0572

…dules

loss: Adding documenntation to the loss layers

8810716

Adding documentation stating to use the config to create the layer

chore: Refactor Conv1d module imports and update documentation

b09eee1

docs: Add documentation for AdaptiveAvgPool1d and AdaptiveAvgPool2d m…

76a8ca3

…odules Added references to the burn_tensor associated functions. Added links between the struct and the config.

Merge branch 'add-documentation-to-burn-core-nn' of https://github.co…

7e54f70

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

Merge branch 'add-documentation-to-burn-core-nn' of https://github.co…

5e1df0c

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

Refactor Conv1d module imports and update documentation

5716916

chore: Refactor Conv2d module imports and update documentation

ef7be41

Add documentation for AvgPool1d and AvgPool2d modules

623a57c

Added references to the burn_tensor associated functions. Added links between the struct and the config.

Merge branch 'add-documentation-to-burn-core-nn' of https://github.co…

817153a

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

Add documentation for MaxPool1d and MaxPool2d modules

3d00880

Added references to the burn_tensor associated functions. Added links between the struct and the config.

Add documentation for leaky_relu and removed Config generic

564cba7

Added references to the burn_tensor associated functions. Added links between the struct and the config. Removed the backend generic from the config since it's not needed (might be a breaking change).

refactor: Update BatchNormConfig initialization and add documentation.

574662e

Added link to config in embedding struct documentation

f1019e6

Merge branch 'add-documentation-to-burn-core-nn' of https://github.co…

6c4ad00

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

refactor: Update GroupNormConfig initialization and add documentation

bbc8302

jwric and others added 12 commits May 7, 2024 16:52

refactor: Update InstanceNormConfig initialization and add documentation

3f1b2ff

feat: Update LayerNormConfig initialization and add documentation

5b22141

refactor: Update RmsNormConfig initialization and add documentation

00b0dc4

fixed: removed #derive accidentally

2b4ec0e

Added missing backticks in pools' shapes

221d083

Format nn doc

2ecb142

Make config fields public in nn modules

1144552

Update import statements in nn modules

c05c0c1

Changed burn_tensor imports to crate::tensor

Update import statements in nn modules' tests

dc04bd2

Changed burn_tensor imports to crate::tensor

breaking change refactor: Update GroupNormConfig and InstanceNormConf…

72363ee

…ig initialization

Merge branch 'add-documentation-to-burn-core-nn' of https://github.co…

ac24b64

…m/Les-Informatichiens/burn into add-documentation-to-burn-core-nn

Make SwiGlu fields public

521101a

nathanielsimard requested a review from laggui May 8, 2024 20:37

laggui requested changes May 9, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add documentation to burn core nn #1746

Add documentation to burn core nn #1746

jwric commented May 7, 2024 •

edited

laggui left a comment

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024

laggui May 9, 2024


		/// Configuration to create an [2D convolution](Conv2d) layer.
		/// Configuration to create an [2D convolution](Conv2d) layer, using the [init function](Conv2dConfig::init).

Add documentation to burn core nn #1746

Are you sure you want to change the base?

Add documentation to burn core nn #1746

Conversation

jwric commented May 7, 2024 • edited

Pull Request Template

Checklist

Related Issues/PRs

Changes

laggui left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jwric commented May 7, 2024 •

edited