Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spring Batch Job Restart: What should be the correct values for chunk and readerPageSize with respect to grid size. #4589

Open
PSHREYASHOLLA opened this issue May 7, 2024 · 0 comments
Labels
status: waiting-for-triage Issues that we did not analyse yet type: bug

Comments

@PSHREYASHOLLA
Copy link

PSHREYASHOLLA commented May 7, 2024

This is posted here as I did not get a response on https://stackoverflow.com/questions/78199004/restarting-a-failed-job-is-not-processing-the-failed-chunk-data-again-continua

This is my step definition for spring batch partition approach,

@bean public Step step1() { return new StepBuilder("step1", jobRepository) .partitioner(slaveStep().getName(), partitioner()) .step(slaveStep()) .gridSize(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1PartitionGridSize()) .taskExecutor(actStmntTaskExecutor()) .build(); }

// slave step
@Bean
public Step slaveStep() 
{
    
    try {
        return new StepBuilder("slaveStep", jobRepository)
                .<EmployeeDetail, EmployeeTaxDetail>chunk(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1ReaderPageSize(),transactionManager)
                .reader(pagingItemReader(null,null))
                .processor(processor())
                .writer(customerItemWriter())
            .taskExecutor(actStmntTaskExecutor())
                .build();
    } catch (Exception ex) {
        throw new RuntimeException("Error creating slave step: " + ex.getMessage());
    }
}

/**

  • Act stmnt task executor.
  • @return the simple async task executor
    */
    public SimpleAsyncTaskExecutor actStmntTaskExecutor() {
    SimpleAsyncTaskExecutor acctStmtTaskExecuter = new SimpleAsyncTaskExecutor();
    acctStmtTaskExecuter.setConcurrencyLimit(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1TaskExecuterThreadConcurrencyLimit());
    acctStmtTaskExecuter.setThreadPriority(febpEmployeeTaxCalculationBatchProperties.getTaxCalculationStep1TaskExecuterThreadPriority());
    acctStmtTaskExecuter.setThreadNamePrefix("FEBP_TAX_CALCULATION_GEN");
    return acctStmtTaskExecuter;
    }

So here for restart to work,

1)Recommendation for restart is not using TaskExecuter. So should we not use TaskExecuter as mentioned above in step1() and slaveStep()? How to achieve concurrency ?

  1. So only SimpleAsyncTaskExecutor needs to be used? If yes is it at step or slavestep?

3)How should the chunk size, gridSize and readers pagesize and fetchsiz be calculated? As I see if chunk size is set to low values, restart is not re-writting failed data.

@PSHREYASHOLLA PSHREYASHOLLA added status: waiting-for-triage Issues that we did not analyse yet type: bug labels May 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
status: waiting-for-triage Issues that we did not analyse yet type: bug
Projects
None yet
Development

No branches or pull requests

1 participant