only_applicable_if incorrectly records failures when triggered

[wdower]

The only_applicable_if function seems to be generating profile errors instead of the expected skips.

Describe the problem

With the following tests:

control "testing only_applicable_if" do
  impact 0.5
  title "testing only_applicable_if"

  only_applicable_if("this should crash out cleanly") { false }

  describe file("/tmp") do
    it { should be_directory }
  end
end

control "testing only_if" do
  impact 0.5
  title "testing only_if"

  only_if("this should crash out cleanly", impact: 0.0) { false }

  describe file("/tmp") do
    it { should be_directory }
  end
end

control "testing if/else" do
  impact 0.5
  title "testing if/else"
  if true
    impact 0.0
    describe "this should crash out cleanly" do
      skip "N/A control due to if/else condition"
    end
  else
    describe file("/tmp") do
      it { should be_directory }
    end
  end
end

Run the tests:

$> bundle exec inspec -v
6.6.0
$> bundle exec inspec exec . --enhanced-outcomes --reporter json:testing.json cli
Profile:   InSpec Profile (dummy)
Version:   0.1.0
Target:    local://
Target ID: b5944c42-ee71-568d-88fc-4e91a28ff7f7

  N/A  testing only_applicable_if: testing only_applicable_if
     ×  No-op 
     N/A control due to only_applicable_if condition: this should crash out cleanly
  N/A  testing only_if: testing only_if
     ↺  Skipped control due to only_if condition: this should crash out cleanly
  N/A  testing if/else: testing if/else
     ↺  N/A control due to if/else condition


Profile Summary: 0 successful controls, 0 control failures, 0 controls not reviewed, 3 controls not applicable, 0 controls have error
Test Summary: 0 successful, 1 failure, 2 skipped

Note the control using only_applicable_if incorrectly marks its test as a failure, even though the overall control is marked as N/A.

The output of that control is also parsed as a profile error in Heimdall:

I would expect only_applicable_if to be equivalent to only_if.

[Nik08]

So only_applicable_if invokes a failure test in a control. That is how it works.
For more details check this change #6229

The enhanced outcome status Not applicable is entirely based on the impact value of a control, irrespective of the test results within a control.

[aaronlippold]

Yes, I understand that it's what it's doing now, however, that wasn't the original design intention. It's really just syntactic sugar over only_if with impact set to zero for you.

The intention of both of these is that they introduced skip result with the provided message.

And more to the point it's not really introducing a failure result into the control it's Throwing a backtrace which is not something we should be introducing into the final results.

[Nik08]

The intention of only_applicable_if was to introduce a failed result in the control and to set the impact value to zero. That's what I remember and that is why PR also does the same.

And it does not throw a runtime error backtrace on an InSpec run. The backtrace generated because of control failure is only captured via reporters using backtrace key, for example in json reporter.