Inputs using schedulers can hang when trying to restart pipelines

[robbavey]

When running a pipeline, using an input with a scheduler, using the logstash-mixin-scheduler, such as logstash-input-elasticsearch, if the output is terminally blocked, due to misconfiguration, for example, the input plugin may hang trying to close the input.

This happens when the input plugin cannot write to the queue - typically because a memory queue is full. The act of closing an input, makes a call to shutdown the scheduler. The scheduler shutdown waits indefinitely to join execution threads, one of which may be blocked, attempting to send to the queue. This will block the shutdown of the input, which in turn blocks the reloading of the pipeline, which means the only way to fix is to shutdown Logstash manually.

Sample pipeline - note that the output here is intended to send to a non-listening, incorrectly specified port

input { 
elasticsearch {

                hosts => "https://localhost:9200"
                ssl_verification_mode => 'none'
                index => ".ds-logs-*"
                query => '{ "query": { "bool": { "must_not": { "exists": { "field": "no_field" } } } } }'
                size => 3000
                schedule => "* * * * *"
}}

output { http {
        url => "https://localhost:8003"
}}