Easy Way of Appending Jobs to a Batch's *Current* Chain?

[zschuessler]

I'm using Batches on Laravel 8 - love'em! In the docs, it shows you how to use chains with them, too:

Chaining Batches
https://laravel.com/docs/8.x/queues#chains-within-batches

Bus::batch([
    // Chain Item 1
    [
        new ReleasePodcast(1),
        new SendPodcastReleaseNotification(1),
    ],
    // Chain Item 2
    [
        new ReleasePodcast(2),
        new SendPodcastReleaseNotification(2),
    ]

Appending Jobs to Batches
The docs even show a use case of appending more jobs to the batch from a "loader"-style job:
https://laravel.com/docs/8.x/queues#adding-jobs-to-batches

public function handle()
{
    // Chain Item 1: Add more jobs!
    $this->batch()->add(Collection::times(1000, function () {
        return new ImportContacts;
    }));
}

The Confusion

I was confused why my jobs were out of sync for admittedly way too long. I dug into what was happening, and it does make sense and isn't a bug.

Expectation

1. Chain Item 1: Add a job that enqueues more child jobs in a chain
2. Those jobs are added to that chain and run before the next item in the chain
3. Chain item 2: The second item in the chain runs after the added jobs

Actual Result

1. Chain Item 1: Add a job that enqueues more jobs in a chain
2. Chain Item 2: The second item in the chain runs before the appended jobs in step 1
3. All jobs are finally run last, added in Chain Item 1

This does make sense, as the demo does show adding to the batch and not a chain. Although I did see some means of interacting with chains in the framework, I wasn't able to determine how to do that with batches.

Something like replacing add with chain below would illustrate the goal:

public function handle()
{
    // Chain Item 1: Add more child jobs!
    $this->batch()->chain(Collection::times(1000, function () {
        return new ImportContacts;
    }));
}

Anyone have a simple solution before I dig some more? Cheers.

[zschuessler]

I came up with a solution that works, but it isn't very eloquent..

// Step 1: Execute a first batch in parallel
$firstBatch = Bus::batch([
    (new EnqueueMoreTestJobsJob())
]);

// Step 2: Execute a second batch in parallel, but only after $firstBatch
$secondBatch = null;
$firstBatch->then(function() use (&$secondBatch) {
    $secondBatch = Bus::batch([
        (new EnqueueEvenMoreTestJobsJob())
    ]);
    $secondBatch->dispatch();
});

// Step 3: Keeping chaining etc etc

This gets unwieldy when working with 5 chain items. Any tips for making this easier on the eyes?

[tomhatzer]

Hey @zschuessler,
would this be something that could work for you?

https://github.com/JustIversen/laravel-job-chainer

You'd somehow have to pass the jobchainer to the jobs inside the chain and/or work with caching to always fetch the latest chain array.

[bretto36]

I came across this whilst looking for a solution to a job in a Batch where i needed to prepend to the current chain in the batch but still maintain batchability

// In the Batchable job i call a new version of itself and increment the page to do whatever processing it needs to do. 
$job = new self(++$this->page);

// If the current job is batching then i add the batchID to the job (this assumes the job has the Batchable trait)
if ($this->batching()) {
    $job->batchId = $this->batchId;
}

// Then i prepend to the chain
$this->prependToChain($job);

[squiaios]

Thank you @bretto36 for your solution.

Is there a better way to append jobs and not prepend please ?

[bretto36]

@squiaios I ended up coming up with a neater solution

$this->addJobToBatchChain(new self(++$this->page));

protected function addJobToBatchChain($job)
    {
        if ($this->batching()) {
            $job->batchId = $this->batchId;
        }

        // Move the chaining to the next job
        $job->chained = $this->chained;

        $this->chained = [];

        $this->batch()->add($job);
    }

But for appending.. that's quite tricky cause you'd technically just need to adjust the chain on the current job, but you wouldn't be able to let the job know there is another job added to the queue, so you progress metrics on the Batch in horizon would be wrong. In my example above i remove the existing chain from the current job and add it to the new next job

If you didn't care about the progress stats then you can always just do $this->appendToChain(new Job());

If you DO care about the progress stats then i tinkered with this but couldn't quite nail it

if ($this->batching()) {
            $job->batchId = $this->batchId;
        }

        // Move the chaining to the next job
        $job->chained = $this->chained;

        // Add the new appended job to the chain
        $newChain = $this->appendToChain($this);

        // Replace the chain on the current job so it doesn't continue to process the new chain
        $this->chained = [];

        // Make an array of the jobs
        $chainToAdd = collect($newChain)->map(function ($job) {
            return unserialize($job);
        })->toArray();

        // I just can't figure out a good way to add it to the batch that doesn't double up on statistics

[squiaios]

Thank you @bretto36 for your solution
I do not care about metrics right now

I will use your first solution, hoping that the excellent team at laravel offers a clean and functional solution for everyone :)