lvm_logical_volume is not idempotent because of mount resource

[jwadolowski]

Cookbook version

v4.1.12

Chef-client version

$ chef-client -v
Chef: 14.0.202

Platform Details

AWS EC2 instance running RHEL 7.5

$ cat /etc/redhat-release
Red Hat Enterprise Linux Server release 7.5 (Maipo)

Scenario:

I've been using this cookbook for quite a while to mange LVM volumes and since the moment I switched to chef-client 14.x I notice that every lvm_logical_volume resource gets re-enabled with every chef-client run.

Steps to Reproduce:

# Create PV and VG using lvm_physical_volume and lvm_volume_group respectively 
# (skipped for brevity)

lvm_logical_volume "lv_aem_publish" do
  group "vg_app"
  size "320G"
  filesystem "ext4"
  mount_point "/opt/aem/publish"
end

The code above produces the following output every single time I execute chef-client:

...
Recipe: cognifide-base::lvm
  * lvm_physical_volume[/dev/nvme1n1] action create
    * chef_gem[/dev/nvme1n1_di-ruby-lvm-attrib_removal] action remove (up to date)
    * chef_gem[/dev/nvme1n1_di-ruby-lvm_removal] action remove (up to date)
     (up to date)
  * chef_gem[/dev/nvme1n1_di-ruby-lvm-attrib_removal] action remove (up to date)
  * chef_gem[/dev/nvme1n1_di-ruby-lvm_removal] action remove (up to date)
  * lvm_physical_volume[/dev/nvme2n1] action create (up to date)
  * lvm_volume_group[vg_app] action create (up to date)
  * lvm_volume_group[vg_backups] action create (up to date)
  * lvm_logical_volume[lv_aem_publish] action create
    * directory[/opt/aem/publish] action create (skipped due to not_if)
    * mount[/opt/aem/publish] action mount (up to date)
    * mount[/opt/aem/publish] action enable
      - enable /dev/mapper/vg_app-lv_aem_publish
...

You can clearly see that mount's resource enable action got triggered. I took a look at the mount codebase and noticed that there were some changes between chef-client 13.x and 14.x.

To trigger enable action this condition has to be met.

My findings are as follows:

• current_resource.enabled - that's definitely true (enabled? method sets that and I can see trace messages regarding /etc/fstab match)
• mount_options_unchanged? - sounds like true to me, but read on for more details
• device_unchanged? - 100% that's true, as it verifies device property, which stays untouched

To see what gets assigned to current_resource I wrote a custom mount provider which overwrites load_current_resource.

Test recipe that mimics mount resource from lvm_logical_volume implementation (mount_spec assignment and mount resource execution in particular):

mount_spec = { location: "/opt/aem/publish" }

mount mount_spec[:location] do
  provider Chef::Provider::Mount::Custom

  options mount_spec[:options]
  dump mount_spec[:dump] if mount_spec[:dump]
  pass mount_spec[:pass] if mount_spec[:pass]
  device "/dev/mapper/vg_app-lv_aem_publish"
  fstype "ext4"

  action :enable
end

Custom provider:

class Chef
  class Provider
    class Mount
      class Custom < Chef::Provider::Mount::Mount
        def load_current_resource
          super
          Chef::Log.debug("NR mounted: #{new_resource.mounted}")
          Chef::Log.debug("NR enabled: #{new_resource.enabled}")
          Chef::Log.debug("NR device: #{new_resource.device}")
          Chef::Log.debug("NR mount_point: #{new_resource.mount_point}")
          Chef::Log.debug("NR fstype: #{new_resource.fstype}")
          Chef::Log.debug("NR options: #{new_resource.options}")
          Chef::Log.debug("NR dump: #{new_resource.dump}")
          Chef::Log.debug("NR pass: #{new_resource.pass}")

          Chef::Log.debug("CR mounted: #{current_resource.mounted}")
          Chef::Log.debug("CR enabled: #{current_resource.enabled}")
          Chef::Log.debug("CR device: #{current_resource.device}")
          Chef::Log.debug("CR mount_point: #{current_resource.mount_point}")
          Chef::Log.debug("CR fstype: #{current_resource.fstype}")
          Chef::Log.debug("CR options: #{current_resource.options}")
          Chef::Log.debug("CR dump: #{current_resource.dump}")
          Chef::Log.debug("CR pass: #{current_resource.pass}")
        end
      end
    end
  end
end

The output was as follows:

...
       [2018-04-27T00:51:54+00:00] DEBUG: NR mounted: false
       [2018-04-27T00:51:54+00:00] DEBUG: NR enabled: false
       [2018-04-27T00:51:54+00:00] DEBUG: NR device: /dev/mapper/vg_app-lv_aem_publish
       [2018-04-27T00:51:54+00:00] DEBUG: NR mount_point: /opt/aem/publish
       [2018-04-27T00:51:54+00:00] DEBUG: NR fstype: ext4
       [2018-04-27T00:51:54+00:00] DEBUG: NR options:
       [2018-04-27T00:51:54+00:00] DEBUG: NR dump: 0
       [2018-04-27T00:51:54+00:00] DEBUG: NR pass: 2
       [2018-04-27T00:51:54+00:00] DEBUG: CR mounted: true
       [2018-04-27T00:51:54+00:00] DEBUG: CR enabled: true
       [2018-04-27T00:51:54+00:00] DEBUG: CR device: /dev/mapper/vg_app-lv_aem_publish
       [2018-04-27T00:51:54+00:00] DEBUG: CR mount_point: /opt/aem/publish
       [2018-04-27T00:51:54+00:00] DEBUG: CR fstype: ext4
       [2018-04-27T00:51:54+00:00] DEBUG: CR options: ["defaults"]
       [2018-04-27T00:51:54+00:00] DEBUG: CR dump: 0
       [2018-04-27T00:51:54+00:00] DEBUG: CR pass: 2
       [2018-04-27T00:51:54+00:00] INFO: mount[/opt/aem/publish] enabled

           - enable /dev/mapper/vg_app-lv_aem_publish
...

mount_options_unchaged? compares fstype, options, dump and pass properties. As you can see options is nil for the new_resource and ["defaults"] for the current_resource, hence enable action gets re-triggered.

I guess the root cause is the fact that options accepts nil since chef-client 14.x:

• only Array and String were accepted in 13.8.5 (reference)
• chef-client 14.0.202 accepts Array, String and nil (reference), so default is not assigned

To be honest I don't know whether it should be fixed in the mount resource itself or mitigated in the lvm_logical_volume provider somehow.

Expected Result:

lvm_logical_volume must not apply changes if they're not required.

Actual Result:

mount resource embedded in lvm_logical_volume is executed with every chef-client run.

[stevenoneill]

Same issue here, similar platforms and versions.
RHEL 7.5
chef-client 14.2.0
lvm cookbook 4.1.13

I've worked around by explicitly adding options: 'default' in the mount_point property

  lvm_logical_volume node['some_name'] do
    group "vg_#{node['some_name']}"
    size        '100%VG'
    filesystem  'ext4'
    mount_point location: node['some_location'], options: 'defaults'
  end