BGZTK0127E Unable to access file for "external" copybook in SearchPathImpactFinder

[FALLAI-Denis]

Hi,

We have been running DBB 1.1.0 for a while and have just upgraded to DBB 1.1.3.
zAppbuild has been upgraded with the 2.5.2 package.

fullBuild and impactBuild worked fine in DBB 1.1.0 and zAppbuid before 2.5.2.
With version DBB 1.1.3 and zAppbuid 2.5.2 the fullBuild works, but the impactBuild triggers an error on the calculation of an access path to an impacting copybook.

In a Repository to build, we have the following configuration:

applicationSrcDirs=${application}/src,${application}/pacbase,${application}/.imports/ENVIRONN,${application}/.imports/GCL
useSearchConfiguration=true
impactSearch=${copybookSearch}${bmsSearch}${linkSearch}
copybookSearch = search:${workspace}/?path=${application}/.imports/**/*.cpy;${application}/src/**/*.cpy;${application}/pacbase/**/*.cpy

The subfolders ENVIRONN and GCL under ${application}/.imports are actually imported Git Repositories containing the copybooks used by the COBOL programs of the main Git Repository corresponding to ${application}.
We are obliged to declare them in applicationSrcDirs so that the impactBuild detects the changes on the contained copybooks.

A copybook located in a subfolder of ${application}/.imports/ENVIRONN is modified.
This copybook impacts a COBOL source located in a subfolder of ${application}/src and which by itself is not modified.

Command executed:
groovyz /var/DBB/MAINT/SCRIPTS/buildBPCE.groovy -w /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master -a ceab3-environn-outinfr-central-mvp2 -h GTEMP.JENKINS.CEAB3.MVP2.M -o /var/DBB/MAINT/LOGS/dbb-logs --impactBuild --verbose

zAppbuild detects the change on the copybook in ${application}/.imports/ENVIRONN:

** Getting current hash for directory /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN
** Storing ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN : dc72cb1bea20e1c6afdbd70cbd8bf34795376213
** Getting baseline hash for directory /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN
** Storing ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN : 5d9ea336f21b2866142a1c0f79b396a33f9a8a98
** Calculating changed files for directory /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN
** Diffing baseline 5d9ea336f21b2866142a1c0f79b396a33f9a8a98 -> current dc72cb1bea20e1c6afdbd70cbd8bf34795376213
*** Changed files for directory /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN :
**** ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy

At this stage, the path to the copybook should have been: .imports/ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy, not ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy?
Is this the cause of the error that occurs afterwards?

A copybook usage impact search is triggered, but it throws an error in the routine SearchPathImpactFinder:

*** Perform impacted analysis for changed files.
** Performing impact analysis on changed file ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy
*** Creating SearchPathImpactFinder with collections [ceab3-environn-outinfr-central-mvp2-master, ceab3-environn-outinfr-central-mvp2-master-outputs] and impactSearch configuration search:/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/?path=ceab3-environn-outinfr-central-mvp2/.imports/**/*.cpy;ceab3-environn-outinfr-central-mvp2/src/**/*.cpy;ceab3-environn-outinfr-central-mvp2/pacbase/**/*.cpysearch:/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/?path=ceab3-environn-outinfr-central-mvp2/bms/*.bmssearch:[:LINK]/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/?path=ceab3-environn-outinfr-central-mvp2/cobol/*.cbl
** Build finished
An error has occured : hudson.AbortException: script returned exit code 1
StackTrace: [org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.handleExit(DurableTaskStep.java:663), org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.check(DurableTaskStep.java:609), org.jenkinsci.plugins.workflow.steps.durable_task.DurableTaskStep$Execution.run(DurableTaskStep.java:553), java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515), java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264), java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304), java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128), java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628), java.base/java.lang.Thread.run(Thread.java:829)]

In the detail of the stack, we see that the access path to the copybook has been incorrectly calculated:
/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy
instead of:
/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/.imports/ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy

Caught: com.ibm.dbb.build.BuildException: java.io.FileNotFoundException: BGZTK0127E Unable to access file, /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy
com.ibm.dbb.build.BuildException: java.io.FileNotFoundException: BGZTK0127E Unable to access file, /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy
	at com.ibm.dbb.dependency.internal.DependencyUtilities.getLogicalFile(DependencyUtilities.java:63)
	at com.ibm.dbb.dependency.SearchPathImpactFinder.findImpactedFiles(SearchPathImpactFinder.java:104)
	at ResolverUtilities.findImpactedFiles(ResolverUtilities.groovy:56)
	at ImpactUtilities$_createImpactBuildList_closure1.doCall(ImpactUtilities.groovy:86)
	at ImpactUtilities.createImpactBuildList(ImpactUtilities.groovy:64)
	at buildBPCE.createBuildList(buildBPCE.groovy:465)
	at buildBPCE.run(buildBPCE.groovy:36)
Caused by: java.io.FileNotFoundException: BGZTK0127E Unable to access file, /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230105-181754/master/ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy
	at com.ibm.dbb.dependency.AbstractDependencyScanner.scan(AbstractDependencyScanner.java:52)
	at com.ibm.dbb.dependency.internal.DependencyUtilities.getLogicalFile(DependencyUtilities.java:52)
	... 6 more

[Frank646]

It seems that the issue is related to the calculation of the access path to the impacting copybook. Have you checked the applicationSrcDirs configuration to ensure that it is correct and points to the correct locations of the copybooks? It is also possible that there is an issue with the impactSearch configuration.

You may want to double-check these configurations and see if they are correctly specified. It might also be helpful to compare the configurations between DBB 1.1.0 and DBB 1.1.3 to see if there are any changes that could be causing the issue.

[dennis-behm]

I share your guess, @FALLAI-Denis, that the root cause is in the identification of the relative path of the modified file.

To calculate the correct offset is tricky - especially when you provide multiple applicationSrcDirs.

We had added another scenario about 10 months ago. Please review the git blame for the fixGitDiffPath method within the ImpactUtilities - you might need to add some additional tracing, where it fails.

It can be a scripting issue with the new scenario or a change in the groovy runtime (DBB 1.1.3 includes Groovy 4.0).

https://github.com/IBM/dbb-zappbuild/blame/2.5.2/utilities/ImpactUtilities.groovy#L993

[FALLAI-Denis]

Hi,

Adding some "println" in fixGitDiffPath:

*** Changed files for directory /SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230106-102324/master/ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN :
DEBUG>>> file=src/bloc/COPYCOB/S9FDI902.cpy, dir=/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230106-102324/master/ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN, mustExist=true, mode=null
DEBUG>>> Scenario 1
DEBUG>>> relPath=ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN/
DEBUG>>> fixedFileName=src/bloc/COPYCOB/S9FDI902.cpy
DEBUG>>> defaultValue=src/bloc/COPYCOB/S9FDI902.cpy
DEBUG>>> props.workspace & fixedFileName=/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230106-102324/master/src/bloc/COPYCOB/S9FDI902.cpy
DEBUG>>> Scenario 2
DEBUG>>> dirName=ENVIRONN
DEBUG>>> dir & file=/SYSTEM/var/tmp/DBB/JENKINS/workspace/CEH49-CEAB3-20230106-102324/master/ceab3-environn-outinfr-central-mvp2/.imports/ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy
DEBUG>>> sortie Scenario 2 no mode en ligne 1038, result=[ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy, 2]
**** ENVIRONN/src/bloc/COPYCOB/S9FDI902.cpy

I think the case where external projects (External Git Repositories) are stored inside the main project tree (Main Git Repository) is not provided for in the xxxx routine.
Scenario 2 is wrongly selected and does not correspond to our use case.

In terms of folder organization we have:

• ${workspace}/${application}: Main Git repository (program)
• ${workspace}/${application}/.imports/xxxx: External Git repository (copybooks)

This organization is similar to that of Git submodules (but with a proprietary implementation): Git Tools - Submodules

[Frank646]

add a check at the beginning of scenario 2 to see if the .imports directory is present in the file path. If it is, you can continue with scenario 2 as it is implemented.

if (dir.contains(".imports")) {
    // Continue with scenario 2 as it is  implemented
} else {
    return [props.workspace + fixedFileName, null]
}

If it is not, you can return the path constructed using props.workspace + fixedFileName instead, as this should be the correct path for a file in the main project tree.

def fixGitDiffPath(file, dir, mustExist, mode) {
    // Check if the .imports directory is present in the file path
    if (dir.contains(".imports")) {
        // Scenario 1
        def relPath = dir.substring(props.workspace.length())
        def fixedFileName = relPath + file
        def defaultValue = file

        // Check if the file exists at the path props.workspace + fixedFileName
        if (new File(props.workspace + fixedFileName).exists()) {
            return [fixedFileName, null]
        }

        // Scenario 2
        def dirName = dir.substring(props.workspace.length() + 1, dir.length() - 1)
        def result = [dirName + '/' + file, 2]

        if (mode != null) {
            result << mode
        }

        return result
    } else {
        // Return the path constructed using props.workspace + fixedFileName
        // This should be the correct path for a file in the main project tree
        return [props.workspace + file, null]
    }
}

[FALLAI-Denis]

Hi,

We applied a solution similar to the one proposed: hard test the presence of ".imports" in the path, which is specific to our use case...
It worked.

    // Scenario 2: Repository name is used as Application Root directory
    // Relativisation du path par rapport au workspace
	String dirName
    if (dir.contains("/.imports/")) {
        dirName = dir - (props.workspace + '/')
    }else{
        dirName = new File(dir).getName()
    }
    if (new File("${dir}/${file}").exists()) {
        return [
            "$dirName/$file" as String,
            2
        ]
    }

However, I think that there should be a thorough reflection and a solution that takes into account the use of several nested Git repositories (use case of submodules in Git).

[Frank646]

@FALLAI-Denis, One approach you could take is to use the git ls-files command to list the files in the Git repository and then parse the output to determine the correct file path.

def fixGitDiffPath(file, dir, mustExist, mode) {
    // Run the 'git ls-files' command to list the files in the repository
    def command = "git ls-files --full-name"
    def process = command.execute()
    process.waitFor()
    def output = process.in.text

    // Parse the output to determine the correct file path
    def lines = output.split("\n")
    def filePath
    for (line in lines) {
        if (line.endsWith(file)) {
            filePath = line
            break
        }
    }

    if (filePath) {
        return [filePath, null]
    } else {
        return [file, mode]
    }
}

[dennis-behm]

@FALLAI-Denis You are correct, Scenario 2 is not correctly selected, but the criteria for this scenario is met.

Your scenario sound actually to be Scenario 4. I think, you would just need to reorder the scenarios and move Scenario 4 before Scenario 2. Would you mind to try this?

[FALLAI-Denis]

Hi,

I will try to make a diagram to explain the organization of our Workspace for DBB by positioning the different variables used by the zAppbuild scripts.
This will allow you to understand our expectation and possibly adapt the fixGitDiffPath function accordingly.