From murdoch at stats.uwo.ca  Tue Jun 26 19:00:44 2007
From: murdoch at stats.uwo.ca (Duncan Murdoch)
Date: Tue, 26 Jun 2007 13:00:44 -0400
Subject: [Rd] conflict between lme4 and RMySQL packages (PR#9753)
In-Reply-To: <18049.3981.789675.547369@stat.math.ethz.ch>
References: <20070626084353.227DD65971@slim.kubism.ku.dk>	<Pine.LNX.4.64.0706261002260.19805@gannet.stats.ox.ac.uk>
	<18049.3981.789675.547369@stat.math.ethz.ch>
Message-ID: <4681463C.2080106@stats.uwo.ca>

On 6/26/2007 9:07 AM, Martin Maechler wrote:
>>>>>> "BDR" == Prof Brian Ripley <ripley at stats.ox.ac.uk>
>>>>>>     on Tue, 26 Jun 2007 10:51:03 +0100 (BST) writes:
> 
>     BDR> You have three summary() functions here:
>     >> library(RMySQL)
>     BDR> Loading required package: DBI
>     >> library(lme4)
>     BDR> Loading required package: Matrix
>     BDR> Loading required package: lattice
>     >> find("summary")
>     BDR> [1] "package:Matrix" "package:DBI"    "package:base"
> 
>     BDR> and this is simply a namespace issue (note the environments):
> 
>     >> summary
>     BDR> standardGeneric for "summary" defined from package "base"
> 
>     BDR> function (object, ...)
>     BDR> standardGeneric("summary")
>     BDR> <environment: 0x15122d0>
>     BDR> Methods may be defined for arguments: object
> 
>     >> Matrix:::summary
>     BDR> standardGeneric for "summary" defined from package "base"
> 
>     BDR> function (object, ...)
>     BDR> standardGeneric("summary")
>     BDR> <environment: 0x26c6640>
>     BDR> Methods may be defined for arguments: object
> 
>     >> DBI:::summary
>     BDR> standardGeneric for "summary" defined from package "base"
> 
>     BDR> function (object, ...)
>     BDR> standardGeneric("summary")
>     BDR> <environment: 0x15122d0>
>     BDR> Methods may be defined for arguments: object
> 
>     BDR> so the visible summary() from package:Matrix is a copy of that in the DBI 
>     BDR> namespace, not that in the Matrix namespace.  
> 
>     BDR> Since lme4 imports all of 
>     BDR> Matrix, it imports its take-over of summary.  So the issue is not *S4* 
>     BDR> dispatch per se, but how the 'methods' package copes with two or more 
>     BDR> packages presenting the same generic.
> 
> yes.  And that's what I meant.
> 
> The problem is --- not for the first time ---
> that in this case,  from an S4 point of view, you would want to
> keep only one version of summary(), namely ``the generic'',
> and all packages / namespaces loaded and all other
>   setMethod(summary, ...)
> should ``attach'' their methods to that generic.
> {and of also get an error or at least a loud warning when
>  re-setting a method for a signature that already has a method
>  set ``with the generic''. }
> 
> I think we should provide useRs and package writers a way to
> achieve that --- which I think is pretty close to having the S4
> generic behave as if there were no namespaces.

I think a better conceptualization is that when S4 converts a function 
or S3 generic into an S4 generic, it does the conversion in place.

The original summary() is an S3 generic living in base, and everyone 
imports base, so it's not a good example.

A better example would be one of the S3 generics in the stats namespace, 
e.g. start().  If they're importing the generic from stats and 
converting it to an S4 generic, then their code should work with anyone 
else who did the same.  I think the easiest way to do this would be if 
we actually replaced the original S3 generic right in the stats 
namespace, but there might be other ways to implement it.

On the other hand, I can see someone defining a generic named start() 
completely independently of the time series meaning in stats, and 
exporting it from their package.  In this case it should live in their 
namespace, and only people who import that namespace will see it.

Duncan Murdoch





> The other approach, that some of the bioconductor folks
> advocate, is for each package to define it's own version of the
> generic explicitly, by  setGeneric(..).
> But that also breaks functionality, namely that of all toplevel calls {of
> summary() in this case} apart from the ones of the last package loaded,
> where as the namespace-internal calls {e.g., the summary(.)
> inside printMer(.) in the given case} will continue to work.
> 
>     BDR> If you dig a bit deeper,
> 
>     >> ls(environment(summary)$".AllMTable")
>     BDR> [1] "ANY"             "DBIObject"       "MySQLConnection" "MySQLDriver"
>     BDR> [5] "MySQLResult"     "lmer2"           "mer"             "sparseMatrix"
>     BDR> [9] "summary.lmer2"   "summary.mer"
>     >> ls(environment(summary)$".AllMTable")
>     BDR> [1] "ANY"             "DBIObject"       "MySQLConnection" "MySQLDriver"
>     BDR> [5] "MySQLResult"     "lmer2"           "mer"             "sparseMatrix"
>     BDR> [9] "summary.lmer2"   "summary.mer"
>     >> ls(environment(Matrix:::summary)$".AllMTable")
>     BDR> [1] "ANY"          "sparseMatrix"
>     >> ls(environment(DBI:::summary)$".AllMTable")
>     BDR> [1] "ANY"             "DBIObject"       "MySQLConnection" "MySQLDriver"
>     BDR> [5] "MySQLResult"     "lmer2"           "mer"             "sparseMatrix"
>     BDR> [9] "summary.lmer2"   "summary.mer"
> 
>     BDR> shows the issue: the methods do not get stored on the version in the 
>     BDR> Matrix namespace, but rather on the version in package:Matrix which was 
>     BDR> copied from the existing export and not Matrix:::summary.
> 
>     BDR> Clearly there is going to be a namespace problem one
>     BDR> way round: what seems undocumented is which.
> 
> Hmm, haven't we been "here" before and come to the conclusion
> that we'd like a special namespace, ("base4" IIRC ??), for all
> those "global S4 generics" ?
> 
> Martin
> 
>     BDR> On Tue, 26 Jun 2007, maechler at stat.math.ethz.ch wrote:
> 
>     >> Thank you, Dale.
>     >> 
>     >> I think you've revealed a bug in the method dispatch mechanism.
>     >> I'm adding here a script which confirms your finding and add
>     >> slightly more insight:
>     >> 
>     >> library(RMySQL)
>     >> library(lme4)
>     >> 
>     >> data(sleepstudy)
>     >> fm1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
>     >> 
>     >> ## MM: First save, then print -- the error is in "print" i.e. show() :
>     >> s1 <- summary(fm1)
>     >> s1 #-> error in printMer()  which is hidden :
>     >> 
>     >> debug(lme4:::printMer)
>     >> s1 # --> printMer(x, ..)  calls  so <- summary(x)  at the very beginning
>     >> ## and indeed,  summary(x) -- *inside* printMer() dispatches wrongly !!!!
>     >> ## whereas top-level, there is no problem :
>     >> 
>     >> ## maybe here, or then later
>     >> undebug(lme4:::printMer)
>     >> 
>     >> ## This works fine top-level, but gives non-sense when inside  printMer() !!
>     >> ##
>     >> so <- summary(s1) ## does nothing, since s1 *is* already  summary.lmer :
>     >> stopifnot(identical(so, s1)) # ok
>     >> 
>     >> ## one other symptom of the same problem:
>     >> showMethods(summary) ## is fine top-level
>     >> 
>     >> ## whereas it does not see the correct methods when used from inside printMer()
>     >> ## i.e., also when debugging
>     >> 
>     >> 
>     >> 
>     >>>>>>> "db" == dale barr <dale.barr at ucr.edu>
>     >>>>>>> on Tue, 26 Jun 2007 00:57:22 +0200 (CEST) writes:
>     >> 
>     db> Full_Name: Dale Barr
>     db> Version: 2.5.1 (patched)
>     db> OS: Ubuntu linux x86_64
>     db> Submission from: (NULL) (138.23.70.108)
>     >> 
>     >> 
>     db> When RMySQL is loaded in before lme4, the summary() function for lmer objects in
>     db> the lme4 packages produces the following error:
>     >> 
>     db> Error in printMer(object) : no slot of name "status" for this object of class
>     db> "table"
>     >> 
>     db> When RMySQL is loaded AFTER lme4, however, no such error arises.  For example,
>     db> the following code gives the error:
>     >> 
>     >> >> library(RMySQL)
>     db> Loading required package: DBI
>     >> >> library(lme4)
>     db> Loading required package: Matrix
>     db> Loading required package: lattice
>     >> >> data(sleepstudy)
>     >> >> fm1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
>     >> >> summary(fm1)
>     db> Error in printMer(object) : no slot of name "status" for this object of class
>     db> "table"
>     >> 
>     db> Now, here is the exact same code except that lme4 is loaded before RMySQL.  The
>     db> summary function works properly.
>     >> 
>     >> >> library(lme4)
>     db> Loading required package: Matrix
>     db> Loading required package: lattice
>     db> library(RMySQL)> library(RMySQL)
>     db> Loading required package: DBI
>     >> >> data(sleepstudy)
>     >> >> fm1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
>     >> >> summary(fm1)
>     db> Linear mixed-effects model fit by REML
>     db> Formula: Reaction ~ Days + (Days | Subject)
>     db> Data: sleepstudy
>     db> AIC  BIC logLik MLdeviance REMLdeviance
>     db> 1754 1770 -871.8       1752         1744
>     db> Random effects:
>     db> Groups   Name        Variance Std.Dev. Corr
>     db> Subject  (Intercept) 610.835  24.7151
>     db> Days         35.056   5.9208  0.067
>     db> Residual             655.066  25.5943
>     db> number of obs: 180, groups: Subject, 18
>     >> 
>     db> Fixed effects:
>     db> Estimate Std. Error t value
>     db> (Intercept)  251.405      6.820   36.86
>     db> Days          10.467      1.546    6.77
>     >> 
>     db> Correlation of Fixed Effects:
>     db> (Intr)
>     db> Days -0.137
>     >> 
>     >> 
>     db> MY SESSION INFO:
>     db> R version 2.5.1 RC (2007-06-22 r42030)
>     db> x86_64-unknown-linux-gnu
>     >> 
>     db> locale:
>     db> LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C
>     >> 
>     db> attached base packages:
>     db> [1] "stats"     "graphics"  "grDevices" "utils"     "datasets"  "methods"
>     db> [7] "base"
>     >> 
>     db> other attached packages:
>     db> RMySQL         DBI        lme4      Matrix     lattice
>     db> "0.6-0"     "0.2-3" "0.99875-2" "0.99875-2"   "0.15-11"
>     >> 
>     db> ______________________________________________
>     db> R-devel at r-project.org mailing list
>     db> https://stat.ethz.ch/mailman/listinfo/r-devel
>     >> 
>     >> ______________________________________________
>     >> R-devel at r-project.org mailing list
>     >> https://stat.ethz.ch/mailman/listinfo/r-devel
>     >> 
> 
>     BDR> -- 
>     BDR> Brian D. Ripley,                  ripley at stats.ox.ac.uk
>     BDR> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
>     BDR> University of Oxford,             Tel:  +44 1865 272861 (self)
>     BDR> 1 South Parks Road,                     +44 1865 272866 (PA)
>     BDR> Oxford OX1 3TG, UK                Fax:  +44 1865 272595
> 
>     BDR> ______________________________________________
>     BDR> R-devel at r-project.org mailing list
>     BDR> https://stat.ethz.ch/mailman/listinfo/r-devel
> 
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



[Rd] conflict between lme4 and RMySQL packages (PR#9753)
Duncan Murdoch