🔄 Synced file(s) with jhudsl/AnVIL_Template

.github/workflows/anvil-library-build.yml

@@ -0,0 +1,93 @@
+
+name: AnVIL Library Build
+
+on:
+  workflow_call:
+    inputs:
+      render-type:
+        required: true
+        type: string
+      repository:
+        type: string
+        required: true
+    secrets:
+      gh_pat:
+        required: true
+
+jobs:
+  gather-anvil-repos:
+    name: Gather AnVIL Repositories
+    runs-on: ubuntu-latest
+    container:
+      image: jhudsl/course_template:main
+
+    steps:
+      - name: Checkout files
+        uses: actions/checkout@v2
+        with:
+          fetch-depth: 0
+          token: ${{ secrets.gh_pat }}
+
+      - name: Configure git
+        run: |
+          git config --local user.email "itcrtrainingnetwork@gmail.com"
+          git config --local user.name "jhudsl-robot"
+        shell: bash
+
+      - name: Merge if rendering preview
+        if: ${{ inputs.render-type == 'preview' }}
+        run: |
+          branch_name='preview-${{ github.event.pull_request.number }}'
+          git fetch --all
+          git checkout $branch_name
+          git merge -s recursive --strategy-option=theirs origin/${{ github.head_ref }} --allow-unrelated-histories
+        shell: bash
+
+      - name: Checkout if main
+        if: ${{ inputs.render-type == 'main' }}
+        run: |
+          git fetch --all
+          git checkout main
+        shell: bash
+
+      - name: Run AnVIL repo check
+        id: anvil_repos
+        env:
+          GH_PAT: ${{ secrets.gh_pat }}
+        run: |
+          sudo apt-get install libcurl4-openssl-dev
+
+          # Need a clean render of the AnVIL repos library
+          rm -f resources/AnVIL_repos.tsv
+          rm -f docs/index.html
+
+          # Run repo check script
+          Rscript --vanilla "scripts/anvil_repo_check.R" --git_pat "$GH_PAT"
+
+      - name: Archive AnVIL repos
+        uses: actions/upload-artifact@v2
+        with:
+          name: anvil-repo-results
+          path: resources/AnVIL_repos.tsv
+
+      # Commit the rendered bookdown files
+      - name: Commit AnVIL Repos file to main
+        if: ${{ inputs.render-type == 'main' }}
+        env:
+          GH_PAT: ${{ secrets.gh_pat }}
+        run: |
+          git add --force resources/AnVIL_repos.tsv
+          git commit -m 'add AnVIL repos' || echo "No changes to commit"
+          git fetch
+          git merge -s recursive --strategy-option=ours origin/main --allow-unrelated-histories
+          git push -u origin main || echo "No changes to push"
+
+      - name: Commit AnVIL Repos file to preview
+        if: ${{ inputs.render-type == 'preview' }}
+        run: |
+          branch_name='preview-${{ github.event.pull_request.number }}'
+          git add --force resources/AnVIL_repos.tsv || echo "No changes to commit"
+          git commit -m 'Add AnVIL repos file' || echo "No changes to commit"
+          git fetch
+          git merge -s recursive --strategy-option=theirs origin/${{ github.head_ref }} --allow-unrelated-histories
+          git push --force origin $branch_name || echo "No changes to commit"

.github/workflows/pull_request.yml

@@ -57,10 +57,10 @@ jobs:
       rendering_docker_image: "${{ env.RENDERING_DOCKER_IMAGE }}"
       dockerfiles_changed: steps.verify-changed-files.outputs.files_changed
 
-  call-custom-workflow:
-    name: Custom Workflow
+  anvil-library-build:
+    name: AnVIL Library Build
     needs: yaml-check
-    uses: ./.github/workflows/custom-workflow.yml
+    uses: ./.github/workflows/anvil-library-build.yml
     with:
       render-type: 'preview'
       repository: $GITHUB_REPOSITORY
@@ -103,12 +103,12 @@ jobs:
         uses: actions/upload-artifact@v2
         with:
           name: spell-check-results
-          path: resources/spell_check_results.tsv
+          path: check_reports/spell_check_results.tsv
 
       - name: Commit spell check errors
         run: |
           branch_name='preview-${{ github.event.pull_request.number }}'
-          git add --force resources/spell_check_results.tsv || echo "No changes to commit"
+          git add --force check_reports/spell_check_results.tsv || echo "No changes to commit"
           git commit -m 'Add spell check file' || echo "No changes to commit"
           git pull --set-upstream origin $branch_name --allow-unrelated-histories --strategy-option=ours
           git push --force origin $branch_name || echo "No changes to commit"
@@ -119,7 +119,7 @@ jobs:
           GH_PAT: ${{ secrets.GH_PAT }}
         run: |
           branch_name='preview-${{ github.event.pull_request.number }}'
-          sp_error_url=https://raw.githubusercontent.com/${GITHUB_REPOSITORY}/$branch_name/resources/spell_check_results.tsv
+          sp_error_url=https://raw.githubusercontent.com/${GITHUB_REPOSITORY}/$branch_name/check_reports/spell_check_results.tsv
           echo ::set-output name=time::$(date +'%Y-%m-%d')
           echo ::set-output name=commit_id::$GITHUB_SHA
           echo ::set-output name=sp_error_url::$sp_error_url
@@ -207,40 +207,17 @@ jobs:
       shell: bash
 
     - name: URLs checker
-      uses: urlstechie/urlchecker-action@master
-      with:
-        # Specify a branch
-        branch: preview-${{ github.event.pull_request.number }}
-
-        # Cleanup
-        cleanup: false
-
-        # A comma-separated list of file types to cover in the URL checks
-        file_types: .Rmd,.md
-
-        # Choose whether to include file with no URLs in the prints.
-        print_all: false
-
-        exclude_files: .github/PULL_REQUEST_TEMPLATE.md, docs/*
-
-        # A comma separated links to exclude during URL checks
-        exclude_urls: https://jhudatascience.org/{Course_Name}}
-
-        force_pass: true
-
-        # A path to a csv file to save results to
-        save: resources/url_checks.csv
-
-    - name: Count URL errors
       id: url_errors
       run: |
-        results=$(Rscript "scripts/url-check.R")
-        echo "::set-output name=url_results::$results"
+        chk_results=$(Rscript scripts/url-check.R)
+        echo $chk_results
+
+        echo "::set-output name=url_results::$chk_results"
 
     - name: Commit URL check
       run: |
         branch_name='preview-${{ github.event.pull_request.number }}'
-        git add --force resources/url_checks.csv || echo "No changes to commit"
+        git add --force check_reports/url_checks.tsv || echo "No changes to commit"
         git commit -m 'Add URL check file' || echo "No changes to commit"
         git fetch
         git merge -s recursive --strategy-option=theirs origin/${{ github.head_ref }} --allow-unrelated-histories
@@ -252,7 +229,7 @@ jobs:
         GH_PAT: ${{ secrets.GH_PAT }}
       run: |
         branch_name='preview-${{ github.event.pull_request.number }}'
-        url_errors=https://raw.githubusercontent.com/${GITHUB_REPOSITORY}/$branch_name/resources/url_checks.csv
+        url_errors=https://raw.githubusercontent.com/${GITHUB_REPOSITORY}/$branch_name/check_reports/url_checks.tsv
         echo ::set-output name=time::$(date +'%Y-%m-%d')
         echo ::set-output name=commit_id::$GITHUB_SHA
         echo ::set-output name=url_errors::$url_errors
@@ -384,36 +361,31 @@ jobs:
 
   render-preview:
     name: Render preview
-    needs: [yaml-check, call-custom-workflow]
+    needs: [yaml-check, anvil-library-build]
     runs-on: ubuntu-latest
     container:
       image: ${{needs.yaml-check.outputs.rendering_docker_image}}
     if: ${{needs.yaml-check.outputs.toggle_render_preview == 'yes'}}
 
     steps:
-      # Give permission to access the repo
-      - name: safelist repository
-        run: |
-          repo_name='${{ github.event.repository.name }}'
-          git config --global --add safe.directory /__w/$repo_name/$repo_name
-
       - name: Checkout files
-        uses: actions/checkout@v2
+        uses: actions/checkout@v3
         with:
           fetch-depth: 0
 
       # Set up git checkout
       - name: Set up git checkout
         run: |
+          git config --global --add safe.directory "$GITHUB_WORKSPACE"
           git config --local user.email "itcrtrainingnetwork@gmail.com"
           git config --local user.name "jhudsl-robot"
-          
+
           branch_name='preview-${{ github.event.pull_request.number }}'
           git fetch --all
           git checkout $branch_name
           git merge -s recursive --strategy-option=theirs origin/${{ github.head_ref }} --allow-unrelated-histories
         shell: bash
-          
+
       # Run bookdown rendering
       - name: Run bookdown render
         id: bookdown

.github/workflows/render-all.yml

@@ -32,10 +32,10 @@ jobs:
       rendering_docker_image: "${{ env.RENDERING_DOCKER_IMAGE }}"
       toggle_quiz_check: "${{ env.CHECK_QUIZZES }}"
 
-  call-custom-workflow:
-    name: Custom Workflow
+  anvil-library-build:
+    name: AnVIL Library Build
     needs: yaml-check
-    uses: ./.github/workflows/custom-workflow.yml
+    uses: ./.github/workflows/anvil-library-build.yml
     with:
       render-type: 'main'
       repository: $GITHUB_REPOSITORY
@@ -44,19 +44,13 @@ jobs:
 
   render-bookdown:
     name: Render bookdown
-    needs: [yaml-check, call-custom-workflow]
+    needs: [yaml-check, anvil-library-build]
     runs-on: ubuntu-latest
     container:
       image: ${{needs.yaml-check.outputs.rendering_docker_image}}
     if: ${{needs.yaml-check.outputs.toggle_bookdown == 'yes'}}
 
     steps:
-      # Give permission to access the repo
-      - name: safelist repository
-        run: |
-          repo_name='${{ github.event.repository.name }}'
-          git config --global --add safe.directory /__w/$repo_name/$repo_name
-
       - name: checkout
         uses: actions/checkout@v2
         with:
@@ -65,7 +59,6 @@ jobs:
 
       - name: Login as jhudsl-robot
         run: |
-
           git config --local user.email "itcrtrainingnetwork@gmail.com"
           git config --local user.name "jhudsl-robot"
 
@@ -101,19 +94,13 @@ jobs:
 
   render-tocless:
     name: Render TOC-less version for Leanpub or Coursera
-    needs: [yaml-check, call-custom-workflow]
+    needs: [yaml-check, anvil-library-build]
     runs-on: ubuntu-latest
     container:
       image: ${{needs.yaml-check.outputs.rendering_docker_image}}
     if: ${{needs.yaml-check.outputs.toggle_coursera == 'yes' || needs.yaml-check.outputs.toggle_leanpub == 'yes'}}
 
     steps:
-      # Give permission to access the repo
-      - name: safelist repository
-        run: |
-          repo_name='${{ github.event.repository.name }}'
-          git config --global --add safe.directory /__w/$repo_name/$repo_name
-
       - name: checkout
         uses: actions/checkout@v2
         with:
@@ -144,7 +131,7 @@ jobs:
 
   render-leanpub:
     name: Finish Leanpub prep
-    needs: [yaml-check, call-custom-workflow, render-tocless]
+    needs: [yaml-check, anvil-library-build, render-tocless]
     runs-on: ubuntu-latest
     container:
       image: jhudsl/ottrpal:main
@@ -209,19 +196,13 @@ jobs:
 
   render-coursera:
     name: Finish Coursera prep
-    needs: [yaml-check, call-custom-workflow, render-tocless]
+    needs: [yaml-check, anvil-library-build, render-tocless]
     runs-on: ubuntu-latest
     container:
       image: ${{needs.yaml-check.outputs.rendering_docker_image}}
     if: ${{needs.yaml-check.outputs.toggle_coursera == 'yes'}}
 
     steps:
-      # Give permission to access the repo
-      - name: safelist repository
-        run: |
-          repo_name='${{ github.event.repository.name }}'
-          git config --global --add safe.directory /__w/$repo_name/$repo_name
-
       - name: checkout
         uses: actions/checkout@v2
         with:

06-data-cleaning.Rmd

@@ -79,7 +79,7 @@ sce.zeisel.qc <- sce.zeisel[,!qc$discard]
 ```
 
 
-::: {.fyi}
+::: {.reflection}
 QUESTIONS
 
 1. How many samples did you remove as suspected low-quality cells? 
@@ -90,7 +90,7 @@ QUESTIONS
 :::
 
 
-::: {.fyi}
+::: {.dictionary}
 Some researchers mark the suspected low-quality cells instead of removing them from the analysis. Ideally, the low-quality cells form their own cluster that can be ignored when interpreting the scRNA-seq results. This approach prevents the researcher from discarding cells or cell types that represent a true biological state and happen to have poor QC metrics.
 :::
 
@@ -118,7 +118,7 @@ sce.zeisel.qc <- computeSumFactors(sce.zeisel.qc, cluster = clusters)
 plot(librarySizeFactors(sce.zeisel.qc), sizeFactors(sce.zeisel.qc), pch = 16, xlab = "Library size factors", ylab = "Deconvolution factors", log = "xy")
 ```
 
-::: {.fyi}
+::: {.reflection}
 QUESTIONS
 
 1. How do the scaling factors using library size normalization compare to the scaling factors using deconvolution normalization?
@@ -180,7 +180,7 @@ dec.zeisel.qc[c(1,500,1000,2000),1:6]
 
 In this dataframe, the `bio` column represents the excess variation in gene expression (that is, the difference between the observed and expected expression). We are looking at four different genes - the gene with the greatest excess variation, as well as the last genes included in the top 500, top 1000, and top 2000 genes.
 
-::: {.fyi}
+::: {.reflection}
 QUESTIONS
 
 1. What is the range of log-fold expression changes (the excess variation) when you choose the top 500 genes? What about when you choose the top 1000 genes? The top 2000 genes? 

07-clustering.Rmd

@@ -104,7 +104,7 @@ out20
 out.umap
 ```
 
-::: {.fyi}
+::: {.reflection}
 QUESTIONS
 1. How does changing the perplexity parameter affect the t-SNE plot?
 
@@ -138,7 +138,7 @@ plotReducedDim(sce.zeisel.tsne20, "TSNE", colour_by="label")
 ```
 
 
-:::{.fyi}
+:::{.reflection}
 QUESTIONS
 1. How many clusters were identified using graph-based clustering? Which cluster contained the most cells, and how many cells did it have?
 :::

08-annotation.Rmd

@@ -86,7 +86,7 @@ pheatmap(logFCs, breaks = seq(-5, 5, length.out = 101))
 
 Here we see that three genes are generally upregulated in Cluster 1 compared to the other clusters: _Gad1_, _Gad2_, and _Slc6a1_. This is where prior biological knowledge comes in handy, as both _Gad1_ and _Slc6a1_ are known interneuron markers (Zeng et al. 2012).
 
-::: {.fyi}
+::: {.reflection}
 QUESTION
 1. Are there any groups or patterns you see in the second heatmap that look interesting?
 :::
@@ -161,7 +161,7 @@ tab
 
 As you can see, the labels applied by the researchers and by our annotation mostly match. (Pyramidal SS nerves are primarily glutamatergic, so although the categories are labeled differently, those two categories do indeed match!) The major exception is the oligodendrocyte precursor cells, which our annotation called astrocytes. However, this mismatch isn't as concerning as you might think, once you know that both astrocytes and oligodendrocytes come from the same precursor cell lineage.
 
-::: {.fyi}
+::: {.reflection}
 QUESTION
 1. Should we be concerned when 1 or 2 cells are assigned a different cell type identity by our annotation than by the researchers? Why or why not?
 :::