From maechler at stat.math.ethz.ch  Tue Jun 26 15:07:25 2007
From: maechler at stat.math.ethz.ch (Martin Maechler)
Date: Tue, 26 Jun 2007 15:07:25 +0200
Subject: [Rd] conflict between lme4 and RMySQL packages (PR#9753)
In-Reply-To: <Pine.LNX.4.64.0706261002260.19805@gannet.stats.ox.ac.uk>
References: <20070626084353.227DD65971@slim.kubism.ku.dk>
	<Pine.LNX.4.64.0706261002260.19805@gannet.stats.ox.ac.uk>
Message-ID: <18049.3981.789675.547369@stat.math.ethz.ch>

>>>>> "BDR" == Prof Brian Ripley <ripley at stats.ox.ac.uk>
>>>>>     on Tue, 26 Jun 2007 10:51:03 +0100 (BST) writes:

    BDR> You have three summary() functions here:
    >> library(RMySQL)
    BDR> Loading required package: DBI
    >> library(lme4)
    BDR> Loading required package: Matrix
    BDR> Loading required package: lattice
    >> find("summary")
    BDR> [1] "package:Matrix" "package:DBI"    "package:base"

    BDR> and this is simply a namespace issue (note the environments):

    >> summary
    BDR> standardGeneric for "summary" defined from package "base"

    BDR> function (object, ...)
    BDR> standardGeneric("summary")
    BDR> <environment: 0x15122d0>
    BDR> Methods may be defined for arguments: object

    >> Matrix:::summary
    BDR> standardGeneric for "summary" defined from package "base"

    BDR> function (object, ...)
    BDR> standardGeneric("summary")
    BDR> <environment: 0x26c6640>
    BDR> Methods may be defined for arguments: object

    >> DBI:::summary
    BDR> standardGeneric for "summary" defined from package "base"

    BDR> function (object, ...)
    BDR> standardGeneric("summary")
    BDR> <environment: 0x15122d0>
    BDR> Methods may be defined for arguments: object

    BDR> so the visible summary() from package:Matrix is a copy of that in the DBI 
    BDR> namespace, not that in the Matrix namespace.  

    BDR> Since lme4 imports all of 
    BDR> Matrix, it imports its take-over of summary.  So the issue is not *S4* 
    BDR> dispatch per se, but how the 'methods' package copes with two or more 
    BDR> packages presenting the same generic.

yes.  And that's what I meant.

The problem is --- not for the first time ---
that in this case,  from an S4 point of view, you would want to
keep only one version of summary(), namely ``the generic'',
and all packages / namespaces loaded and all other
  setMethod(summary, ...)
should ``attach'' their methods to that generic.
{and of also get an error or at least a loud warning when
 re-setting a method for a signature that already has a method
 set ``with the generic''. }

I think we should provide useRs and package writers a way to
achieve that --- which I think is pretty close to having the S4
generic behave as if there were no namespaces.

The other approach, that some of the bioconductor folks
advocate, is for each package to define it's own version of the
generic explicitly, by  setGeneric(..).
But that also breaks functionality, namely that of all toplevel calls {of
summary() in this case} apart from the ones of the last package loaded,
where as the namespace-internal calls {e.g., the summary(.)
inside printMer(.) in the given case} will continue to work.

    BDR> If you dig a bit deeper,

    >> ls(environment(summary)$".AllMTable")
    BDR> [1] "ANY"             "DBIObject"       "MySQLConnection" "MySQLDriver"
    BDR> [5] "MySQLResult"     "lmer2"           "mer"             "sparseMatrix"
    BDR> [9] "summary.lmer2"   "summary.mer"
    >> ls(environment(summary)$".AllMTable")
    BDR> [1] "ANY"             "DBIObject"       "MySQLConnection" "MySQLDriver"
    BDR> [5] "MySQLResult"     "lmer2"           "mer"             "sparseMatrix"
    BDR> [9] "summary.lmer2"   "summary.mer"
    >> ls(environment(Matrix:::summary)$".AllMTable")
    BDR> [1] "ANY"          "sparseMatrix"
    >> ls(environment(DBI:::summary)$".AllMTable")
    BDR> [1] "ANY"             "DBIObject"       "MySQLConnection" "MySQLDriver"
    BDR> [5] "MySQLResult"     "lmer2"           "mer"             "sparseMatrix"
    BDR> [9] "summary.lmer2"   "summary.mer"

    BDR> shows the issue: the methods do not get stored on the version in the 
    BDR> Matrix namespace, but rather on the version in package:Matrix which was 
    BDR> copied from the existing export and not Matrix:::summary.

    BDR> Clearly there is going to be a namespace problem one
    BDR> way round: what seems undocumented is which.

Hmm, haven't we been "here" before and come to the conclusion
that we'd like a special namespace, ("base4" IIRC ??), for all
those "global S4 generics" ?

Martin

    BDR> On Tue, 26 Jun 2007, maechler at stat.math.ethz.ch wrote:

    >> Thank you, Dale.
    >> 
    >> I think you've revealed a bug in the method dispatch mechanism.
    >> I'm adding here a script which confirms your finding and add
    >> slightly more insight:
    >> 
    >> library(RMySQL)
    >> library(lme4)
    >> 
    >> data(sleepstudy)
    >> fm1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
    >> 
    >> ## MM: First save, then print -- the error is in "print" i.e. show() :
    >> s1 <- summary(fm1)
    >> s1 #-> error in printMer()  which is hidden :
    >> 
    >> debug(lme4:::printMer)
    >> s1 # --> printMer(x, ..)  calls  so <- summary(x)  at the very beginning
    >> ## and indeed,  summary(x) -- *inside* printMer() dispatches wrongly !!!!
    >> ## whereas top-level, there is no problem :
    >> 
    >> ## maybe here, or then later
    >> undebug(lme4:::printMer)
    >> 
    >> ## This works fine top-level, but gives non-sense when inside  printMer() !!
    >> ##
    >> so <- summary(s1) ## does nothing, since s1 *is* already  summary.lmer :
    >> stopifnot(identical(so, s1)) # ok
    >> 
    >> ## one other symptom of the same problem:
    >> showMethods(summary) ## is fine top-level
    >> 
    >> ## whereas it does not see the correct methods when used from inside printMer()
    >> ## i.e., also when debugging
    >> 
    >> 
    >> 
    >>>>>>> "db" == dale barr <dale.barr at ucr.edu>
    >>>>>>> on Tue, 26 Jun 2007 00:57:22 +0200 (CEST) writes:
    >> 
    db> Full_Name: Dale Barr
    db> Version: 2.5.1 (patched)
    db> OS: Ubuntu linux x86_64
    db> Submission from: (NULL) (138.23.70.108)
    >> 
    >> 
    db> When RMySQL is loaded in before lme4, the summary() function for lmer objects in
    db> the lme4 packages produces the following error:
    >> 
    db> Error in printMer(object) : no slot of name "status" for this object of class
    db> "table"
    >> 
    db> When RMySQL is loaded AFTER lme4, however, no such error arises.  For example,
    db> the following code gives the error:
    >> 
    >> >> library(RMySQL)
    db> Loading required package: DBI
    >> >> library(lme4)
    db> Loading required package: Matrix
    db> Loading required package: lattice
    >> >> data(sleepstudy)
    >> >> fm1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
    >> >> summary(fm1)
    db> Error in printMer(object) : no slot of name "status" for this object of class
    db> "table"
    >> 
    db> Now, here is the exact same code except that lme4 is loaded before RMySQL.  The
    db> summary function works properly.
    >> 
    >> >> library(lme4)
    db> Loading required package: Matrix
    db> Loading required package: lattice
    db> library(RMySQL)> library(RMySQL)
    db> Loading required package: DBI
    >> >> data(sleepstudy)
    >> >> fm1 <- lmer(Reaction ~ Days + (Days|Subject), sleepstudy)
    >> >> summary(fm1)
    db> Linear mixed-effects model fit by REML
    db> Formula: Reaction ~ Days + (Days | Subject)
    db> Data: sleepstudy
    db> AIC  BIC logLik MLdeviance REMLdeviance
    db> 1754 1770 -871.8       1752         1744
    db> Random effects:
    db> Groups   Name        Variance Std.Dev. Corr
    db> Subject  (Intercept) 610.835  24.7151
    db> Days         35.056   5.9208  0.067
    db> Residual             655.066  25.5943
    db> number of obs: 180, groups: Subject, 18
    >> 
    db> Fixed effects:
    db> Estimate Std. Error t value
    db> (Intercept)  251.405      6.820   36.86
    db> Days          10.467      1.546    6.77
    >> 
    db> Correlation of Fixed Effects:
    db> (Intr)
    db> Days -0.137
    >> 
    >> 
    db> MY SESSION INFO:
    db> R version 2.5.1 RC (2007-06-22 r42030)
    db> x86_64-unknown-linux-gnu
    >> 
    db> locale:
    db> LC_CTYPE=en_US.UTF-8;LC_NUMERIC=C;LC_TIME=en_US.UTF-8;LC_COLLATE=en_US.UTF-8;LC_MONETARY=en_US.UTF-8;LC_MESSAGES=en_US.UTF-8;LC_PAPER=en_US.UTF-8;LC_NAME=C;LC_ADDRESS=C;LC_TELEPHONE=C;LC_MEASUREMENT=en_US.UTF-8;LC_IDENTIFICATION=C
    >> 
    db> attached base packages:
    db> [1] "stats"     "graphics"  "grDevices" "utils"     "datasets"  "methods"
    db> [7] "base"
    >> 
    db> other attached packages:
    db> RMySQL         DBI        lme4      Matrix     lattice
    db> "0.6-0"     "0.2-3" "0.99875-2" "0.99875-2"   "0.15-11"
    >> 
    db> ______________________________________________
    db> R-devel at r-project.org mailing list
    db> https://stat.ethz.ch/mailman/listinfo/r-devel
    >> 
    >> ______________________________________________
    >> R-devel at r-project.org mailing list
    >> https://stat.ethz.ch/mailman/listinfo/r-devel
    >> 

    BDR> -- 
    BDR> Brian D. Ripley,                  ripley at stats.ox.ac.uk
    BDR> Professor of Applied Statistics,  http://www.stats.ox.ac.uk/~ripley/
    BDR> University of Oxford,             Tel:  +44 1865 272861 (self)
    BDR> 1 South Parks Road,                     +44 1865 272866 (PA)
    BDR> Oxford OX1 3TG, UK                Fax:  +44 1865 272595

    BDR> ______________________________________________
    BDR> R-devel at r-project.org mailing list
    BDR> https://stat.ethz.ch/mailman/listinfo/r-devel



[Rd] conflict between lme4 and RMySQL packages (PR#9753)
Martin Maechler