From bbolker at gmail.com  Mon Jul 18 18:08:25 2016
From: bbolker at gmail.com (Ben Bolker)
Date: Mon, 18 Jul 2016 12:08:25 -0400
Subject: [Rd] summary() dispatch puzzle
In-Reply-To: <22412.61310.487798.494771@stat.math.ethz.ch>
References: <57894B7E.6070508@gmail.com>
	<22412.61310.487798.494771@stat.math.ethz.ch>
Message-ID: <578CFEF9.3030808@gmail.com>


  I'm not doing a good job at explaining.  See inline below ...

On 16-07-18 11:02 AM, Martin Maechler wrote:
>>>>>> Ben Bolker <bbolker at gmail.com>
>>>>>>     on Fri, 15 Jul 2016 16:45:50 -0400 writes:akram
> 
>     > I'm sorry I haven't boiled this down to a more-minimal example yet,
>     > but ...
> 
>     > I'm working on an S3 method (tidy.merMod, in the 'broom' package). It
>     > normally handles 'merMod' objects from the lme4 package, but I'm trying
>     > to make it handle 'merModLmerTest' objects from the lmerTest package too.
> 
>     > The merModLmerTest class inherits (as an S4) class from the merMod
>     > class.  The important difference (for my current purposes) is that
>     > summary.merMod returns an object containing a coefficient table
>     > *without* degrees or freedom or p-values, while the summary method for
>     > merModLmerTest objects returns one *with* both of those columns.
> 
>     > Because merModLmerTest inherits from merMod, calling tidy(obj) on a
>     > merModLmerTest object does get you into broom:::tidy.merMod.  Within
>     > that function, though, calling summary() appears to call summary.merMod
>     > instead of merModLmerTest, so I don't get the additional columns I want.
> 
>     > Ideas/directions for further diagnoses?  Is this an S3/S4 confusion,
>     > or something funny about the way the broom and merModLmerTest package
>     > are talking to each other, or ... ?
> 
> I'm not sure, if I see your problem when I execute the
> reproducible code below ... 
> 
> 
>     > ----
>     > library(lme4)
>     > fm1 <- lmer(Reaction~Days+(Days|Subject),sleepstudy)
>     > library(lmerTest)
>     > fm1B <- as(fm1,"merModLmerTest")
>     > coef(summary(fm1B))               ## table with df and p values
>     > coef(lme4:::summary.merMod(fm1B)) ## table without df and p values
> 
>     > is(fm1B,"merMod")  ## TRUE
> 
>     > library(broom)
>     > tidy(fm1B)
> 
> Part of what I see from above, is
> 
>     > coef(summary(fm1B))               ## table with df and p values
> 		 Estimate Std. Error       df   t value     Pr(>|t|)
>     (Intercept) 251.40510   6.824556 17.00002 36.838310 0.000000e+00
>     Days         10.46729   1.545789 17.00000  6.771485 3.263788e-06
> 
>     > tidy(fm1B)
> 			      term     estimate std.error statistic    group
>     1                  (Intercept) 251.40510485  6.824556 36.838310    fixed
>     2                         Days  10.46728596  1.545789  6.771485    fixed
>     3       sd_(Intercept).Subject  24.74044768        NA        NA  Subject
>     4              sd_Days.Subject   5.92213326        NA        NA  Subject
>     5 cor_(Intercept).Days.Subject   0.06555134        NA        NA  Subject
>     6      sd_Observation.Residual  25.59181589        NA        NA Residual
>     > 
> 
> which seems consistent at least: You get std.error and statistic
> for those two terms that you get 'df' and 'p-value from "the
> good method" of summary(.)

  This is as expected.  What's going on here is that I'm trying to get
the p-value into the tidied object.   You really have to debug *into*
the function to see the problem.  I've tried to remedy this a bit in my
fork by adding a 'debug' argument:

devtools::install_github("bbolker/broom")
## ... re-load package or re-run session ...
tidy(fm1B,debug=TRUE)

produces

output from coef(summary(x)):
             Estimate Std. Error   t value
(Intercept) 251.40510   6.824556 36.838310
Days         10.46729   1.545789  6.771485


that is, when coef(summary(x)) is run *inside* the tidy.merMod function,
it calls summary.merMod rather than the summary method for
merModLmerTest objects ...

> 
> ?
> 
> Of course, I'm using the CRAN versions of the packages, not
> github ones, and I see that we end up using half of tidyVerse
> ... which does not necessarily keep debugging easy..
> 

  Sorry about that.  broom is fairly heavily integrated with the
tidyverse ...


> Martin :
> 
>> sessionInfo()
> R version 3.3.1 Patched (2016-07-16 r70928)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Fedora 22 (Twenty Two)
> 
> locale:
>  [1] LC_CTYPE=de_CH.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8       
>  [4] LC_COLLATE=de_CH.UTF-8     LC_MONETARY=en_US.UTF-8    LC_MESSAGES=de_CH.UTF-8   
>  [7] LC_PAPER=de_CH.UTF-8       LC_NAME=C                  LC_ADDRESS=C              
> [10] LC_TELEPHONE=C             LC_MEASUREMENT=de_CH.UTF-8 LC_IDENTIFICATION=C       
> 
> attached base packages:
> [1] graphics  grDevices datasets  stats     utils     methods   base     
> 
> other attached packages:
> [1] broom_0.4.1     lmerTest_2.0-32 lme4_1.1-12     Matrix_1.2-3    fortunes_1.5-3 
> [6] sfsmisc_1.1-0  
> 
> loaded via a namespace (and not attached):
>  [1] Rcpp_0.12.5         compiler_3.3.1      nloptr_1.0.4        RColorBrewer_1.1-2 
>  [5] plyr_1.8.4          tools_3.3.1         rpart_4.1-10        tibble_1.1         
>  [9] gtable_0.2.0        nlme_3.1-128        lattice_0.20-33     psych_1.6.6        
> [13] DBI_0.4-1           parallel_3.3.1      gridExtra_2.2.1     stringr_1.0.0      
> [17] dplyr_0.5.0         cluster_2.0.3       grid_3.3.1          nnet_7.3-12        
> [21] data.table_1.9.6    R6_2.1.2            survival_2.39-5     foreign_0.8-66     
> [25] latticeExtra_0.6-28 minqa_1.2.4         Formula_1.2-1       tidyr_0.5.1        
> [29] reshape2_1.4.1      ggplot2_2.1.0       magrittr_1.5        Hmisc_3.17-4       
> [33] scales_0.4.0        MASS_7.3-45         splines_3.3.1       assertthat_0.1     
> [37] mnormt_1.5-4        colorspace_1.2-6    stringi_1.1.1       acepack_1.3-3.3    
> [41] munsell_0.4.3       chron_2.3-47       
>>
> 
> 
>     > ## hard to show what's going on without debugging through the function:
>     > ## the issue occurs here:
>     > ## https://github.com/dgrtwo/broom/blob/master/R/lme4_tidiers.R#L94
> 
>     > I can't replicate this behaviour with a trivial method ...
> 
>     > silly <- function(x, ...) {
>     > UseMethod("silly")
>     > }
>     > silly.merMod <- function(object, ...) {
>     > coef(summary(object))
>     > }
>     > silly(fm1B)
>     > environment(silly) <- environment(tidy)
>     > silly(fm1B)
> 
>     > ______________________________________________
>     > R-devel at r-project.org mailing list
>     > https://stat.ethz.ch/mailman/listinfo/r-devel
>



[Rd] summary() dispatch puzzle
Ben Bolker