Process Does Not Terminate When Processing Multiple Items

[pott-101]

Bug description
I recently upgraded my project's SpringBoot version to 3, along with an upgrade to Spring Batch version 5.

However, I've encountered an issue while executing Batch processes. During the execution of ItemWriter logic, the process unexpectedly halts. Although I've enabled the show-sql option and observed that some queries are being generated, the intended queries do not fully execute and the process remains running.

The point at which it halts varies randomly with each execution, and very occasionally, all intended queries are executed, allowing the process to commit and terminate normally.

TL;DR:

Upgraded to SpringBoot 3.
When ItemReader processes a single item, the commit always executes, and the process terminates normally.
When processing more than one item, the process randomly stops at certain points, and neither commit nor process termination occurs.

Environment
Spring Batch 5.0.3 / Spring Boot 3.1.5 / Java 17 / JPA Hibernate 6

Steps to reproduce
Below is the code I modified while upgrading to version 5 of Spring Batch

1. Remove @EnableBatchProcessing
2. Add a Config class that implements DefaultBatchConfiguration.

@Configuration
public class BatchDBConfiguration extends DefaultBatchConfiguration {

    @BatchDataSource
    @Bean(name = "jobDataSource")
    @ConfigurationProperties(prefix = "spring.datasource")
    public DataSource jobDataSource() {
        return DataSourceBuilder.create().build();
    }

    @Override
    protected DataSource getDataSource() {
        return this.jobDataSource();
    }

    @Override
    protected PlatformTransactionManager getTransactionManager() {
        return this.jdbcTransactionManager();
    }

    @Bean(name = "jdbcTransactionManager")
    public PlatformTransactionManager jdbcTransactionManager() {
        return new JdbcTransactionManager(getDataSource());
    }

    @Override
    protected ExecutionContextSerializer getExecutionContextSerializer() {
        return new Jackson2ExecutionContextStringSerializer();
    }

    @Bean
    @ConditionalOnMissingBean
    @ConditionalOnProperty(prefix = "spring.batch.job", name = "enabled", havingValue = "true", matchIfMissing = true)
    public JobLauncherApplicationRunner jobLauncherApplicationRunner(JobLauncher jobLauncher, JobExplorer jobExplorer,
                                                                     JobRepository jobRepository, BatchProperties properties) {
        JobLauncherApplicationRunner runner = new JobLauncherApplicationRunner(jobLauncher, jobExplorer, jobRepository);
        String jobNames = properties.getJob().getName();

        if (StringUtils.hasText(jobNames)) {
            runner.setJobName(jobNames);
        }
        return runner;
    }
}

Below is an example of one of the jobs where the issue is occurring.

@Configuration
public class ConvertToDormantUserBatchConfig {

    private static final String NAME = "convertToDormantUser";
    private static final String JOB = NAME + "Job";
    private static final String JOB_STEP = JOB + "Step";
    private static final int CHUNK_SIZE = 50;

    private final JobRepository jobRepository;
    private final PlatformTransactionManager platformTransactionManager;

    private final FindPreDormantUsers findPreDormantUsers;
    private final ConvertToDormantUser convertToDormantUser;

    private final ExecutionLoggingListener executionLoggingListener;
    private final TargetDateJobParameter convertToDormantUserJobParameter;
    private final Logger logger = LoggerFactory.getLogger("USER_BATCH");

    public ConvertToDormantUserBatchConfig(JobRepository jobRepository,
                                           @Qualifier("jpaTransactionManager") PlatformTransactionManager platformTransactionManager,
                                           @Qualifier("convertToDormantUserJobParameter") TargetDateJobParameter convertToDormantUserJobParameter,
                                           FindPreDormantUsers findPreDormantUsers,
                                           ConvertToDormantUser convertToDormantUser,
                                           ExecutionLoggingListener executionLoggingListener) {
        this.jobRepository = jobRepository;
        this.platformTransactionManager = platformTransactionManager;
        this.findPreDormantUsers = findPreDormantUsers;
        this.convertToDormantUser = convertToDormantUser;
        this.executionLoggingListener = executionLoggingListener;
        this.convertToDormantUserJobParameter = convertToDormantUserJobParameter;
    }

    @Bean(JOB + "Parameter")
    @JobScope
    public TargetDateJobParameter jobParameter(@Value("#{jobParameters[targetDate]}") String targetDate) {
        return new TargetDateJobParameter(targetDate);
    }

    @Bean
    public Job convertToDormantUserJob() {
        return new JobBuilder(JOB, jobRepository)
                .preventRestart()
                .incrementer(new UniqueRunIdIncrementer())
                .listener(executionLoggingListener)
                .start(convertToDormantUserJobStep())
                .build();
    }

    @Bean
    @JobScope
    public Step convertToDormantUserJobStep() {
        return new StepBuilder(JOB_STEP, jobRepository)
                .<FindPreDormantUsersResult, FindPreDormantUsersResult>chunk(CHUNK_SIZE, platformTransactionManager)
                .reader(convertToDormantUserReader())
                .writer(convertToDormantUserWriter())
                .listener(executionLoggingListener)
                .build();
    }

    @Bean
    @StepScope
    public ListItemReader<FindPreDormantUsersResult> convertToDormantUserReader() {
        var results = findPreDormantUsers.execute(
                new FindPreDormantUsersQuery(
                        CONVERT_DORMANT_USER_CRITERION,
                        convertToDormantUserJobParameter.getTargetDate()
                )
        );
        return new ListItemReader<>(results);
    }

    @Bean
    @StepScope
    public ItemWriter<FindPreDormantUsersResult> convertToDormantUserWriter() {
        return results -> {
            List<String> userTokens = results.getItems().stream().map(FindPreDormantUsersResult::getUserToken).collect(Collectors.toList());
            convertToDormantUser.execute(new ConvertToDormantUserCommand(userTokens));
        };
    }
}

I'm wondering if the TxManager is incorrectly configured. In the job, I'm injecting the jpaTransactionManager bean, which is the TxManager being used effectively throughout the application.

In the BatchConfig, I've created and registered a JdbcTransactionManager as a bean.

Expected behavior
I expect the process to commit successfully and terminate normally when processing multiple items

---

same question in stackOverFlow

[fmbenhassine]

In the code you shared, the job repository is configured with a jdbc transaction manager (returned by the overridden DefaultBatchConfiguration#getTransactionManager) while the step is configured with a jpa transaction manager (autowired in ConvertToDormantUserBatchConfig with @Qualifier("jpaTransactionManager") PlatformTransactionManager platformTransactionManager). This could be the cause of the issue. If you configure the job repository with the same transaction manager as the step (ie the jpa transaction manager), the issue should be fixed. Can you try that and share your feedback?

Moreover, there are many beans that are step/job scoped unnecessarily.

If you still having issues, please provide a minimal complete example that reproduces the problem and we will debug the case. Thank you.