drush_backend_batch_process() and &$context

[ultimike]

I have a batch that runs fine via a form - with the $context being populated and $context['results'] available to my "finished" callback as expected.

I've created a custom Drush command for this batch, but when I run it with drush_backend_batch_process();, the $results array isn't populated as I expect in my "finished" callback. Everything else is working fine - the custom Drush command runs the batch with the expected results, other than the $results array being provided to the "finished" callback.

Is this a known limitation of drush_backend_batch_process(); or am I just missing something?

thanks,
-mike

[halisonfernandes]

Same issue here. I'm on 10.6.1 Drush version.

What I've noticed is that after the first interaction, it's loosing the $context['results'].

[weitzman]

Sounds like a bug to me. Not expected.

[omahm]

Getting the same here on 10.6.2. It's not passing the context between operations and the finish callback.

[weitzman]

FYI Drush 10 is no longer supported https://www.drush.org/latest/install/#drupal-compatibility. @ultimike do you recall what version of Drush you were using?

[omahm]

Thanks @weitzman, I upgraded to 11.0.9 and the issue still persists.

[ultimike]

@weitzman When I posted this issue, I was running 11.0.5.

I have since updated to 11.0.9 and the problem persists 🙁

-mike

[weitzman]

I have added test coverage which shows that $results is populated. $results is the 2nd param in a finished callback. See #5144

Not sure why others are seeing differently. Can you share your batch code? FWIW I looked at https://git.drupalcode.org/project/examples/-/blob/3.x/modules/batch_example/batch_example.module to see how batches are implemented.

[ultimike]

@weitzman Here's the relevant (I believe) code (some names have been changed to protect the innocent):

    // Create a Batch API item for each user.
    foreach ($users as $uid => $user) {
      $operations[] = ['my_module_batch_operation', [$uid]];
    }
    $batch = [
      'operations' => $operations,
      'finished' => 'my_module_finished',
      'file' => $this->extensionListModule->getPath('my_module') . '/my_module.batch.inc',
    ];
    batch_set($batch);
    if ($drush) {
      // For some reason, running via drush leaves the $results array in
      // my_module_finished() empty. See
      // https://github.com/drush-ops/drush/issues/5009
      drush_backend_batch_process();
    }

and

function my_module_batch_operation(int $uid, &$context) {
  if (empty($context['results'])) {
    $context['results'] = [];
    $context['results']['num'] = 0;
  }
  $my_module_batch = \Drupal::service('my_module.batch');
  return $my_module_batch->updateStuffBatch($uid, $context);
}

and

function my_module_finished(bool $success, array $results, array $operations) {
  if ($success) {
    // Note: the $results array isn't properly populated when run via Drush.
    // See: https://github.com/drush-ops/drush/issues/5009
    if (isset($results['num'])) {
      $message = t('Updated the things for @num users.', ['@num' => $results['num']]);
    }
    else {
      $message = t('Zero things updated.');
    }
  }
  else {
    $message = t('Finished with an error.');
  }
  \Drupal::messenger()->addMessage($message);
}

and

  public function updateStuffBatch($uid, &$context) {
    /** @var \Drupal\Core\Entity\EntityStorageInterface $user_storage */
    $user_storage = $this->entityTypeManager->getStorage('user');
    $account = $user_storage->load($uid);
    $this->myService->updateStuff($account);
    $context['results'][] = $uid;
    $context['results']['num']++;
    $context['message'] = $this->t('Updating stuff belonging to "@username".',
      ['@username' => $account->label()]
    );
  }

I think that's everything related - let me know if you need more - and thanks!

-mike

[weitzman]

All I can see return $my_module_batch->updateStuffBatch($uid, $context); doesn't need to return since updateStuffBatch1 has no return value. Don't think that matters though. I dont see any problem in the code. I suggest xdebug to find where $context['results'] is lost or reset.

[ultimike]

@weitzman Thanks for taking a look at it.

I've removed the return as you suggested, and as expected, it has no effect (other than making my code better 😃).

I spent a few minutes stepping through things using xdebug and it is clear that each time my_module_batch_operation() is run, $context['results'] is empty and gets re-initialized.

I have a question and this may be a result of the issue rather than the cause of the issue, but I'm still curious. In _drush_backend_batch_process(), there is a call to $result = $process->getOutputAsJson();. When stepping through the code and I pause right after this call, $result is:

Array
(
    [0] => Array
        (
        )

    [drush_batch_process_finished] => 1
)

At the very least, I am expecting to see the initialized version of the $results array, including $results['num'] = 0;, but the num key is not present.

It seems that whatever I set $context['results'] to never makes it to the Json output.

When I set the values of $context['results'] in updateStuffBatch() do I need to do something special so that they are available to $process->getOutputAsJson();?

thanks,
-mike

[weitzman]

How about putting a breakpoint in https://github.com/drush-ops/drush/blob/11.x/src/Drupal/Commands/core/BatchCommands.php#L20. That is where each batch gets worked. The line after it is where output is printed. Thats whats read by $process->getOutputAsJson()

[ultimike]

Hmm, doing as you suggested, stepping through the code, $return is (copy and paste from VSCode Debug Console):

array(2)
  0: array(0)
  drush_batch_process_finished: true

But, this led me down the path of taking another look at how I initialize the $context['results'] and I found and fixed the issue 😃

In my_module_batch_operation(), I was initializing it like this:

  if (empty($context['results'])) {
    $context['results'] = [];
    $context['results']['num'] = 0;
  }

I went back and looked at the sample code in the Examples module and didn't see anything similar so I suspected that this was the issue. Changing it to the following fixed my issue:

  if (empty($context['results']['num'])) {
    $context['results']['num'] = 0;
  }

thanks!
-mike

[halisonfernandes]

@ultimike @weitzman this is actually an issue. Could you please reopen it?
It keeps the value if you have more than one interaction, but if you have only one it loses the $context['results'].

I'm using one batch process + sandbox to split the operations in my scenario. If the sandbox items are less than the maximum size, I still face the issue - even if I apply something similar to what @ultimike suggested.

I spent some time debugging this issue and I figured out that the problem happens in the following line:

// Magic wrap to catch changes to 'message' key.
$batch_context = new DrushBatchContext($batch_context);

This is basically changing the type of $batch_context from array to DrushBatchContext to handle drush log messages. I noticed that it's not updating the $batch['set'] values after the call_user_func_array - it keeps only the sandbox value because it is not being manipulated by the function offsetSet (don't know why).
That's also the reason why we have this line on batch.inc:

$finished = $batch_context['finished'];

The $finished should be loaded automatically from the $context['finished'], but since we are changing the context type to DrushBatchContext, it's not reflecting on the $finished - btw, the $context['finished'] is handled by the offsetSet function.

A PR has been created to fix it: #5354

[DamienMrtl]

Is there any way to make this work also on Drupal 9. Because it seems, I can't use Drush 12 with drupal9. And I can't update my website to Drupal 10 yet because of some dependencies in my modules.

[halisonfernandes]

@DamienMrtl, there is always an option to add the patch to your composer package.
The patch should work on Drush 11 too.
https://patch-diff.githubusercontent.com/raw/drush-ops/drush/pull/5354.diff

[jcowher]

The patch from #5354 does not apply to 11.6.0 so I'm attaching a re-rolled patch.
drush--5009--11.6.0.patch