Add support for multiple subdirs each with their own go.mod

[fkautz]

Thank you for creating the issue!

• Yes, I'm using a binary release within 2 latest major releases. Only such installations are supported.
• Yes, I've searched similar issues on GitHub and didn't find any.
• Yes, I've included all information below (version, config, etc).

Please include the following information:

Version of golangci-lint

$ golangci-lint --version
golangci-lint has version 1.20.1 built from 849044b on 2019-10-15T19:11:27Z

Config file

$ cat .golangci.yml
---
run:
  concurrency: 6
  deadline: 15m
  issues-exit-code: 1
  tests: true
  skip-dirs:
    - build
    - conf
    - controlplane/scripts
    - forwarder/vppagent/build
    - forwarder/vppagent/conf
    - forwarder/scripts
    - deployments
    - docks
    - scripts
linters-settings:
  errcheck:
    check-type-assertions: false
    check-blank: false
  govet:
    check-shadowing: true
    settings:
      printf:
        funcs:
          - (github.com/golangci/golangci-lint/pkg/logutils.Log).Infof
          - (github.com/golangci/golangci-lint/pkg/logutils.Log).Warnf
          - (github.com/golangci/golangci-lint/pkg/logutils.Log).Errorf
          - (github.com/golangci/golangci-lint/pkg/logutils.Log).Fatalf
  golint:
    min-confidence: 0.8
  goimports:
    local-prefixes: github.com/networkservicemesh/networkservicemesh
  gocyclo:
    min-complexity: 20
  maligned:
    suggest-new: true
  dupl:
    threshold: 100
  goconst:
    min-len: 3
    min-occurrences: 3
  depguard:
    list-type: blacklist
    include-go-root: false
    packages:
      - github.com/davecgh/go-spew/spew
  misspell:
    locale: US
  unused:
    check-exported: true
  unparam:
    check-exported: true
  nakedret:
    max-func-lines: 30
  prealloc:
    simple: true
    range-loops: true
    for-loops: false
  gocritic:
    enabled-checks:
      - rangeValCopy
      - boolExprSimplify
      - badCond
      - methodExprCall
      - paramTypeCombine
      - ptrToRefParam
      - rangeExprCopy
      - captLocal
      - caseOrder
      - defaultCaseOrder
      - dupBranchBody
      - dupSubExpr
      - elseif
      - emptyFallthrough
      - emptyStringTest
      - equalFold
      - indexAlloc
      - nestingReduce
      - nilValReturn
      - yodaStyleExpr
    settings:
      captLocal:
        paramsOnly: true
      rangeValCopy:
        sizeThreshold: 100
linters:
  disable:
    - depguard
    - wsl
    - lll
    - gofmt
    - varcheck  # deprecated
    - unused  # deprecated
    - goimports
  enable-all: true
issues:
  exclude-use-default: false
  max-issues-per-linter: 0
  max-same-issues: 0
  exclude-rules:  # Use relative path from module
    - path: _test\.go
      linters:
        - golint
    - path: integration/const.go
      linters:
        - deadcode
    - path: cmd/admission-webhook/init.go
      linters:
        - gochecknoglobals
        - gochecknoinits
    - path: cmd/nsm-coredns
      linters:
        - gochecknoglobals
        - gochecknoinits
    - path: tools/socket.go
      linters:
        - gochecknoglobals
        - gochecknoinits
        - gosec
    - path: pkg/monitor/
      linters:
        - dupl
    - path: pkg/nsm.nsm.go
      linters:
        - gocyclo  # TODO: reduce nsm.request() complexity
    - path: cloudtest/pkg/utils/process.go
      linters:
        - gosec
    - path: cloudtest/pkg/commands/main.go
      linters:
        - gosec
        - funlen
    - path: probes/health/serve_mux_health.go
      linters:
        - gosec
    - path: kubetest/log_utils.go
      linters:
        - gosec
  exclude:
    - should not use dot imports
    - \`version\` is a global variable
    - not declared by package utf8
    - lines are duplicate of```

</details>

<details><summary>Go environment</summary>

```bash
$ go version && go env
go version go1.13.1 darwin/amd64
GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/fkautz/Library/Caches/go-build"
GOENV="/Users/fkautz/Library/Application Support/go/env"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GONOPROXY=""
GONOSUMDB=""
GOOS="darwin"
GOPATH="/Users/fkautz/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/Cellar/go/1.13.1/libexec"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/Cellar/go/1.13.1/libexec/pkg/tool/darwin_amd64"
GCCGO="gccgo"
AR="ar"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD="/Users/fkautz/src/networkservicemesh/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/4g/mwsmt1wj5qd5dwdhrshkfwzw0000gn/T/go-build534044314=/tmp/go-build -gno-record-gcc-switches -fno-common"```

</details>

<details><summary>Verbose output of running</summary>

```bash
$ golangci-lint run -v
INFO [config_reader] Config search paths: [./ /Users/fkautz/src/networkservicemesh /Users/fkautz/src /Users/fkautz /Users /] 
INFO [lintersdb] Active 10 linters: [deadcode errcheck gosimple govet ineffassign staticcheck structcheck typecheck unused varcheck] 
INFO [loader] Go packages loading at mode 575 (imports|exports_file|deps|files|name|types_sizes|compiled_files) took 148.047143ms 
INFO [runner/filename_unadjuster] Pre-built 0 adjustments in 252.334µs 
INFO [runner] processing took 2.757µs with stages: max_same_issues: 464ns, autogenerated_exclude: 446ns, nolint: 288ns, max_from_linter: 181ns, skip_dirs: 178ns, path_prettifier: 175ns, filename_unadjuster: 164ns, cgo: 129ns, identifier_marker: 115ns, skip_files: 112ns, source_code: 108ns, path_shortener: 105ns, diff: 101ns, max_per_file_from_linter: 50ns, uniq_by_line: 48ns, exclude: 47ns, exclude-rules: 46ns 
INFO [runner] linters took 79.834664ms with stages: goanalysis_metalinter: 49.130728ms, unused: 30.658834ms 
INFO File cache stats: 0 entries of total size 0B 
INFO Memory: 4 samples, avg is 70.1MB, max is 70.9MB 
INFO Execution took 243.209356ms                  ```

</details>

Network Service Mesh (a CNCF sandbox project) is currently using multiple subdirs each with their own go.mod and go.sum.

E.g.

./go.mod
./go.sum
forwarder/go.mod
forwarder/go.sum
k8s/go.mod
k8s/go.sum

When we run golangci-lint, only the root is loaded and all subdirs with their own go.mod are ignored. I assume this is to avoid analyzing vendored code. Unfortunately, this behavior prevents our repository from being fully analyzed in golangci.com.

Please add support for multiple subdirs to be listed recursively. We are ok adding these to the .golangci.yml if necessary.

Thanks for building such a great tool! Cheers!

[kpurdon]

To add a bit of color here, this is a common issue for anybody with a multi-module monorepo (a reasonably common pattern).

Given the following directory structure:

monorepo/
├── bar
│   ├── bar.go
│   └── go.mod
├── foo
│   └── foo.go
└── go.mod

Running the following from the root of monorepo golangci-lint run ./foo ./bar will fail on ./bar w/ the error ERRO Running error: context loading failed: no go files to analyze.

[twelho]

I came up with the following solution when trying to set up the golangci-lint GitHub Action for a monorepo:

$ cat .github/workflows/golangci-lint.yaml

name: golangci-lint
on: [push, pull_request]
jobs:
  resolve-modules:
    name: Resolve Modules
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - name: Checkout Sources
        uses: actions/checkout@v2
      - id: set-matrix
        run: ./tools/resolve-modules.sh
  golangci:
    name: Linter
    needs: resolve-modules
    runs-on: ubuntu-latest
    strategy:
      matrix: ${{ fromJson(needs.resolve-modules.outputs.matrix) }}
    steps:
      - uses: actions/checkout@v2
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v1
        with:
          version: v1.28
          working-directory: ${{ matrix.workdir }}

$ cat ./tools/resolve-modules.sh

#!/bin/bash
# Recursively finds all directories with a go.mod file and creates
# a GitHub Actions JSON output option. This is used by the linter action.

echo "Resolving modules in $(pwd)"

PATHS=$(find . -mindepth 2 -type f -name go.mod -printf '{"workdir":"%h"},')
echo "::set-output name=matrix::{\"include\":[${PATHS%?}]}"

This uses the newly introduced fromJson() to dynamically define a matrix strategy for each directory that contains a go.mod file (excluding top-level) and runs golangci-lint in each of those directories (submodules) separately. Less ideal than having official support for multi-module monorepos, but at least it doesn't require defining a configuration for each submodule separately.

[matrixik]

Adding to @twelho (thank you for this!)
Use:

PATHS=$(find . -mindepth 2 -not -path "*/vendor/*" -type f -name go.mod -printf '{"workdir":"%h"},')

to exclude vendor folders from linting.

[sakirma]

I still have problems with multiple modules within my repo and @twelho 's solution did solves my problem!
Changing:
uses: golangci/golangci-lint-action@v1
to
uses: golangci/golangci-lint-action@v2
Made it all work again.

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[franklinkim]

Hi, any update if this will be a supported without workarounds?

[gnuletik]

For pre-commit users, you can define multiple entries until a flag into golangci-lint is implemented.

repos:
  - repo: https://github.com/golangci/golangci-lint
    rev: v1.50.1
    hooks:
      # no built-in support for multiple go.mod
      # https://github.com/golangci/golangci-lint/issues/828
      - id: golangci-lint
        name: golangci-lint-root
        entry: bash -c 'golangci-lint run'

      - id: golangci-lint
        name: golangci-lint-subdir
        entry: bash -c 'cd subtest && golangci-lint run --config $OLDPWD/.golangci.yml'

[JoelOtter]

Any movement on this?

[wafer-bw]

This worked for me.

on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        directory: [module1, module2, module3]
    steps:
    - uses: actions/checkout@v3
    - uses: actions/setup-go@v4
      with:
        go-version: '^1.21.1'
    - name: golangci-lint
      uses: golangci/golangci-lint-action@v3
      with:
          version: latest
          working-directory: ${{ matrix.directory }}

[gaby]

@wafer-bw Thank you for the suggestion. I was able to make it dynamic by using the following:

name: golangci-lint

on:
  push:
    branches:
      - "master"
      - "main"
    paths-ignore:
      - "**.md"
      - LICENSE
      - ".github/ISSUE_TEMPLATE/*.yml"
      - ".github/dependabot.yml"
  pull_request:
    branches:
      - "*"
    paths-ignore:
      - "**.md"
      - LICENSE
      - ".github/ISSUE_TEMPLATE/*.yml"
      - ".github/dependabot.yml"

jobs:
  generate-matrix:
    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.set-matrix.outputs.matrix }}
    steps:
      - name: Fetch Repository
        uses: actions/checkout@v4
      - id: set-matrix
        run: |
          DIRECTORIES=$(find . -type d -not -path '*/\.*' | jq -R -s -c 'split("\n")[:-1]')
          echo "matrix=${DIRECTORIES}" >> $GITHUB_OUTPUT

  golangci-lint:
    needs: generate-matrix
    runs-on: ubuntu-latest
    strategy:
      matrix:
        modules: ${{fromJson(needs.generate-matrix.outputs.matrix)}}
    steps:
      - name: Fetch Repository
        uses: actions/checkout@v4
      - uses: actions/setup-go@v4
        with:
          go-version: '^1.21.x'
      - name: golangci-lint
        uses: golangci/golangci-lint-action@v3
        with:
          version: latest
          working-directory: ${{ matrix.modules }}

[stevenh]

I was surprised that this didn't work out the box and was hiding issues because of multiple modules in the repo with no warning, definitely a gotach.