Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Inconsistency in p_adjust when working with emmeans/emGrid objects #854

Open
Generalized opened this issue Mar 6, 2023 · 0 comments
Open
Labels
enhancement 💥 Implemented features can be improved or revised

Comments

@Generalized
Copy link

Let's assume I have an emmeans/emGrid object already created:

> emg
 Week    emmean    SE  df asymp.LCL asymp.UCL
 Week 1  -0.317 0.132 Inf    -0.577   -0.0573
 Week 2  -0.633 0.136 Inf    -0.900   -0.3656
 Week 4  -0.719 0.178 Inf    -1.068   -0.3696
 Week 8  -0.984 0.166 Inf    -1.309   -0.6587
 Week 16 -1.198 0.162 Inf    -1.514   -0.8810
 Week 56 -1.456 0.290 Inf    -2.025   -0.8879

Covariance estimate used: vbeta 
Confidence level used: 0.95 

Now I want both CI + p-value to be adjusted simultaneously using the mutivariate t distribution:

> set.seed(1000); update(emg, infer = c(TRUE, TRUE), adjust="mvt")
 Week    emmean    SE  df asymp.LCL asymp.UCL z.ratio p.value
 Week 1  -0.317 0.132 Inf    -0.652    0.0181  -2.393  0.0717
 Week 2  -0.633 0.136 Inf    -0.977   -0.2880  -4.643  <.0001
 Week 4  -0.719 0.178 Inf    -1.170   -0.2681  -4.033  0.0003
 Week 8  -0.984 0.166 Inf    -1.404   -0.5642  -5.929  <.0001
 Week 16 -1.198 0.162 Inf    -1.606   -0.7889  -7.413  <.0001
 Week 56 -1.456 0.290 Inf    -2.190   -0.7227  -5.021  <.0001

Covariance estimate used: vbeta 
Confidence level used: 0.95 
Conf-level adjustment: mvt method for 6 estimates 
P value adjustment: mvt method for 6 tests 

Now I'm trying to do the same with parameters (expecting it will pass the parameter to emmeans since it understands this class):

> parameters(emg, p_adjust = "mvt")
Week    | Marginal Means |   SE |         95% CI |     z |      p
-----------------------------------------------------------------
Week 1  |          -0.32 | 0.13 | [-0.58, -0.06] | -2.39 | 0.017 
Week 2  |          -0.63 | 0.14 | [-0.90, -0.37] | -4.64 | < .001
Week 4  |          -0.72 | 0.18 | [-1.07, -0.37] | -4.03 | < .001
Week 8  |          -0.98 | 0.17 | [-1.31, -0.66] | -5.93 | < .001
Week 16 |          -1.20 | 0.16 | [-1.51, -0.88] | -7.41 | < .001
Week 56 |          -1.46 | 0.29 | [-2.02, -0.89] | -5.02 | < .001

p-value adjustment method: mvt
Warning message:
`p_adjust` must be one of holm, hochberg, hommel, bonferroni, bh, by, fdr, none, tukey, scheffe, sidak 

OK, let's try "bh"

> parameters(emg, p_adjust = "bh")
Error in match.arg(method) : 
  'arg' should be one of “holm”, “hochberg”, “hommel”, “bonferroni”, “BH”, “BY”, “fdr”, “none”

Also doesn't work. So maybe the "BH":

> parameters(emg, p_adjust = "BH")
Week    | Marginal Means |   SE |         95% CI |     z |      p
-----------------------------------------------------------------
Week 1  |          -0.32 | 0.13 | [-0.58, -0.06] | -2.39 | 0.017 
Week 2  |          -0.63 | 0.14 | [-0.90, -0.37] | -4.64 | < .001
Week 4  |          -0.72 | 0.18 | [-1.07, -0.37] | -4.03 | < .001
Week 8  |          -0.98 | 0.17 | [-1.31, -0.66] | -5.93 | < .001
Week 16 |          -1.20 | 0.16 | [-1.51, -0.88] | -7.41 | < .001
Week 56 |          -1.46 | 0.29 | [-2.02, -0.89] | -5.02 | < .001

p-value adjustment method: Benjamini & Hochberg (1995)

This one worked.

So, to summarize:

  1. the "mvt" isn't passed to emmeans, even if parameters understand this class (but maybe it was not the goal - then I'm sorry for reporting it)
  2. but even if we pick the other options, the hint is given using all small letters
@strengejacke strengejacke added bug 🐛 Something isn't working enhancement 💥 Implemented features can be improved or revised labels Mar 6, 2023
strengejacke added a commit that referenced this issue Mar 6, 2023
@strengejacke strengejacke removed the bug 🐛 Something isn't working label Mar 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement 💥 Implemented features can be improved or revised
Projects
None yet
Development

No branches or pull requests

2 participants