From osklyar at maninvestments.com  Tue Sep  9 19:04:37 2008
From: osklyar at maninvestments.com (Sklyar, Oleg (London))
Date: Tue, 9 Sep 2008 18:04:37 +0100
Subject: [Rd] 'xtfrm' performance (influences 'order' performance) in R
 devel
In-Reply-To: <48C69942.8070305@r-project.org>
References: <1A68FCB28DE72F4BA3B967E6506CCE4303049734@mildnpexmb01.maninvestments.ad.man.com>
	<48C683DC.1060707@r-project.org> 
	<1A68FCB28DE72F4BA3B967E6506CCE4303049737@mildnpexmb01.maninvestments.ad.man.com>
	<48C69942.8070305@r-project.org>
Message-ID: <1A68FCB28DE72F4BA3B967E6506CCE430304973A@mildnpexmb01.maninvestments.ad.man.com>


Aha, it works if I do

setGeneric("order", signature="...")

However the problem with that is that it generates a warning which I
cannot suppress on install:

Creating a generic for "order" in package  "AHLCalendar"
    (the supplied definition differs from and overrides the implicit
generic in package "base": Signatures differ:  (...), (na.last,
decreasing))

and it generates a warning about masking order from base on load:

  AHLCalendar [0.2.42] (9 Sep 2008). ?AHLCalendar or
vignette('AHLCalendar') to get started

Attaching package: 'AHLCalendar'

        The following object(s) are masked from package:base :

         order 

The package exports (excerpt):

exportPattern("^[^\\.]")
exportMethods(order)

The reason for these messages is that the signature is different and I
particularly dislike the masking thing (as I cannot predict if it leads
to other problems somewhere). As I understand the current dotsMethods
does not allow mixing dots and other types, so I cannot really define a
matching signature. Is that right? Is there a way around it?

As for the rest, yes, I meant generic and it works nicely for xtfrm. But
as I wrote later, the problem is in 'rank' and rank is not generic so
defining a method would not help in calling a different implementation.

Thanks,
Oleg

Dr Oleg Sklyar
Research Technologist
AHL / Man Investments Ltd
+44 (0)20 7144 3107
osklyar at maninvestments.com 

> -----Original Message-----
> From: John Chambers [mailto:jmc at r-project.org] 
> Sent: 09 September 2008 16:42
> To: Sklyar, Oleg (London)
> Cc: R-devel at r-project.org
> Subject: Re: [Rd] 'xtfrm' performance (influences 'order' 
> performance) in R devel
> 
> Sklyar, Oleg (London) wrote: 
> 
> 	Ha, defined xtfrm for TimeDate, works instantly (xtfrm 
> is already a
> 	method). However, it won't be taken up by order as it 
> is not in the
> 	imported namespace, so order falls back to xtfrm.default.
> 	  
> 
> By "method" you mean "generic"?  xtfrm is an S3 generic.  I'm 
> not clear what happens if you define an S3 method for it.  
> Yes, there is a problem defining an S4 generic & it would be 
> good to deal with that.  Nontrivial, however.
> 
> 
> 	
> 	Moreover, defining order (which is not a method 
> unfortunately, *any
> 	chance of changing this*?):
> 	
> 	setGeneric("order")
> 	setMethod("order", "TimeDate", 
> 		function (..., na.last = TRUE, decreasing = FALSE) 
> 			order(list(...)[[1]]@.Data,na.last=na.last,
> 	decreasing=decreasing))
> 	
> 	does not help either as it won't be taken up, order 
> still calls the
> 	default one, what am I doing wrong?
> 	  
> 
> I'm skeptical that this is true.  I did a simple example:
> 
> > setClass("foo", contains = "numeric", representation(flag = 
> "logical"))
> [1] "foo"
> > xx = new("foo", rnorm(5))
> > setGeneric("order", sig = "...")
> Creating a generic for "order" in package  ".GlobalEnv"
>     (the supplied definition differs from and overrides the 
> implicit generic in package "base": Signatures differ:  
> (...), (na.last, decreasing))
> [1] "order"
> > setMethod("order", "foo", function (..., na.last = TRUE, 
> decreasing = FALSE){message("Method called"); order(..1 at .Data)})
> [1] "order"
> > order(xx)
> Method called
> [1] 2 4 3 1 5
> 
> You do need to be calling order() directly from one of your 
> functions, and have it in your namespace, if your package has one.
> 
> 
> 	
> 	
> 	
> 	Dr Oleg Sklyar
> 	Research Technologist
> 	AHL / Man Investments Ltd
> 	+44 (0)20 7144 3107
> 	osklyar at maninvestments.com 
> 	
> 	  
> 
> 		-----Original Message-----
> 		From: John Chambers [mailto:jmc at r-project.org] 
> 		Sent: 09 September 2008 15:11
> 		To: Sklyar, Oleg (London)
> 		Cc: R-devel at r-project.org
> 		Subject: Re: [Rd] 'xtfrm' performance 
> (influences 'order' 
> 		performance) in R devel
> 		
> 		No definitive answers, but here are a few observations.
> 		
> 		In the call to order() code, I notice that you 
> have dropped 
> 		into the branch
> 		    if (any(unlist(lapply(z, is.object))))
> 		where the alternative in your case would seem 
> to have been 
> 		going directly to the internal code.
> 		
> 		You can consider a method for xtfrm(), which 
> would help but 
> 		won't get you completely back to a trivial 
> computation.  
> 		Alternatively,  order() should be eligible for the new 
> 		mechanism of defining methods for "...".
> 		
> 		(Individual existing methods may not be the 
> issue, and one 
> 		can't infer anything definite from the evidence 
> given,  but a 
> 		plausible culprit is the "[" method.  Because 
> [] expressions 
> 		appear so often, it's always chancy to define a 
> nontrivial 
> 		method for this function.)
> 		
> 		John
> 		
> 		Sklyar, Oleg (London) wrote: 
> 		
> 			Hello everybody,
> 			
> 			it looks like the presense of some (do 
> know know which) 
> 		S4 methods for a
> 			given S4 class degrades the performance 
> of xtfrm (used 
> 		in 'order' in new
> 			R-devel) by a factor of millions. This 
> is for classes 
> 		that ARE derived
> 			from numeric directly and thus should 
> be quite trivial 
> 		to convert to
> 			numeric.
> 			
> 			Consider the following example:
> 			
> 			setClass("TimeDateBase", 
> 			    representation("numeric", mode="character"),
> 			    prototype(mode="posix")
> 			)
> 			setClass("TimeDate",
> 			    representation("TimeDateBase", 
> tzone="character"),
> 			    prototype(tzone="London")
> 			)
> 			x = new("TimeDate", 1220966224 + runif(1e5))
> 			
> 			system.time({ z = order(x) })
> 			## > system.time({ z = order(x) })
> 			##   user  system elapsed 
> 			##  0.048   0.000   0.048 
> 			
> 			getClass("TimeDate")
> 			## Class "TimeDate"
> 			
> 			## Slots:
> 			                                    
> 			## Name:      .Data     tzone      mode
> 			## Class:   numeric character character
> 			
> 			## Extends: 
> 			## Class "TimeDateBase", directly
> 			## Class "numeric", by class 
> "TimeDateBase", distance 2
> 			## Class "vector", by class 
> "TimeDateBase", distance 3
> 			
> 			
> 			Now, if I load a library that not only 
> defines these 
> 		same classes, but
> 			also a bunch of methods for those, then 
> I have the 
> 		following result:
> 			
> 			library(AHLCalendar)
> 			x = now() + runif(1e5) ## just random 
> times in POSIXct format
> 			x[1:5]
> 			## TimeDate [posix] object in 
> 'Europe/London' of length 5:
> 			## [1] "2008-09-09 14:19:35.218" 
> "2008-09-09 14:19:35.672"
> 			## [3] "2008-09-09 14:19:35.515" 
> "2008-09-09 14:19:35.721"
> 			## [5] "2008-09-09 14:19:35.657"
> 			
> 			  
> 		
> 				system.time({ z = order(x) })
> 				    
> 		
> 			
> 			
> 			Enter a frame number, or 0 to exit   
> 			
> 			 1: system.time({
> 			 2: order(x)
> 			 3: lapply(z, function(x) if 
> (is.object(x)) xtfrm(x) else x)
> 			 4: FUN(X[[1]], ...)
> 			 5: xtfrm(x)
> 			 6: xtfrm.default(x)
> 			 7: as.vector(rank(x, ties.method = 
> "min", na.last = "keep"))
> 			 8: rank(x, ties.method = "min", 
> na.last = "keep")
> 			 9: switch(ties.method, average = , min 
> = , max =
> 			.Internal(rank(x[!nas], ties.
> 			10: .gt(c(1220966375.21811, 
> 1220966375.67217, 1220966375.51470,
> 			1220966375.7211
> 			11: x[j]
> 			12: x[j]
> 			
> 			Selection: 0
> 			Timing stopped at: 47.618 13.791 66.478 
> 			
> 			At the same time:
> 			
> 			system.time({ z = as.numeric(x) }) ## 
> same as x at .Data
> 			##   user  system elapsed 
> 			##  0.001   0.000   0.001 
> 			
> 			The only difference between the two is 
> that I have the 
> 		following methods
> 			defined for TimeDate (full listing below). 
> 			
> 			Any idea why this could be happenning. 
> And yes, it is 
> 		down to xtfrm
> 			function, 'order' was just a place 
> where the problem 
> 		occured. Should
> 			xtfrm function be smarter with respect 
> to classes that 
> 		are actually
> 			derived from 'numeric'?
> 			
> 			  
> 		
> 				showMethods(class="TimeDate")
> 				    
> 		
> 			Function: + (package base)
> 			e1="TimeDate", e2="TimeDate"
> 			e1="TimeDate", e2="numeric"
> 			    (inherited from: e1="TimeDateBase", 
> e2="numeric")
> 			
> 			Function: - (package base)
> 			e1="TimeDate", e2="TimeDate"
> 			
> 			Function: Time (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: TimeDate (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: TimeDate<- (package AHLCalendar)
> 			x="TimeSeries", value="TimeDate"
> 			
> 			Function: TimeSeries (package AHLCalendar)
> 			x="data.frame", ts="TimeDate"
> 			x="matrix", ts="TimeDate"
> 			x="numeric", ts="TimeDate"
> 			
> 			Function: [ (package base)
> 			x="TimeDate", i="POSIXt", j="missing"
> 			x="TimeDate", i="Time", j="missing"
> 			x="TimeDate", i="TimeDate", j="missing"
> 			x="TimeDate", i="integer", j="missing"
> 			    (inherited from: x="TimeDateBase", 
> i="ANY", j="missing")
> 			x="TimeDate", i="logical", j="missing"
> 			    (inherited from: x="TimeDateBase", 
> i="ANY", j="missing")
> 			x="TimeSeries", i="TimeDate", j="missing"
> 			x="TimeSeries", i="TimeDate", j="vector"
> 			
> 			Function: [<- (package base)
> 			x="TimeDate", i="ANY", j="ANY", value="ANY"
> 			x="TimeDate", i="ANY", j="ANY", value="numeric"
> 			x="TimeDate", i="missing", j="ANY", value="ANY"
> 			x="TimeDate", i="missing", j="ANY", 
> value="numeric"
> 			
> 			Function: add (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: addMonths (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: addYears (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: align (package AHLCalendar)
> 			x="TimeDate", to="character"
> 			x="TimeDate", to="missing"
> 			
> 			Function: as.POSIXct (package base)
> 			x="TimeDate"
> 			
> 			Function: as.POSIXlt (package base)
> 			x="TimeDate"
> 			
> 			Function: coerce (package methods)
> 			from="TimeDate", to="TimeDateBase"
> 			
> 			Function: coerce<- (package methods)
> 			from="TimeDate", to="numeric"
> 			
> 			Function: dates (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: format (package base)
> 			x="TimeDate"
> 			
> 			Function: fxFwdDate (package AHLCalendar)
> 			x="TimeDate", country="character"
> 			
> 			Function: fxSettleDate (package AHLCalendar)
> 			x="TimeDate", country="character"
> 			
> 			Function: holidays (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: index (package AHLCalendar)
> 			x="TimeDate", y="POSIXt"
> 			x="TimeDate", y="Time"
> 			x="TimeDate", y="TimeDate"
> 			
> 			Function: initialize (package methods)
> 			.Object="TimeDate"
> 			    (inherited from: .Object="ANY")
> 			
> 			Function: leapYear (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: mday (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: mode (package base)
> 			x="TimeDate"
> 			    (inherited from: x="TimeDateBase")
> 			
> 			Function: mode<- (package base)
> 			x="TimeDate", value="character"
> 			    (inherited from: x="TimeDateBase", 
> value="character")
> 			
> 			Function: month (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: pretty (package base)
> 			x="TimeDate"
> 			
> 			Function: prettyFormat (package AHLCalendar)
> 			x="TimeDate", munit="character"
> 			x="TimeDate", munit="missing"
> 			
> 			Function: print (package base)
> 			x="TimeDate"
> 			
> 			Function: show (package methods)
> 			object="TimeDate"
> 			    (inherited from: object="TimeDateBase")
> 			
> 			Function: summary (package base)
> 			object="TimeDate"
> 			
> 			Function: td2tz (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: times (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: tojulian (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: toposix (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: tots (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: tzone (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: tzone<- (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: wday (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: yday (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			Function: year (package AHLCalendar)
> 			x="TimeDate"
> 			
> 			
> 			
> 			Dr Oleg Sklyar
> 			Research Technologist
> 			AHL / Man Investments Ltd
> 			+44 (0)20 7144 3107
> 			osklyar at maninvestments.com
> 			
> 			
> 			
> 		
> **********************************************************************
> 			The contents of this email are for the named 
> 		addressee(s...{{dropped:22}}
> 			
> 			______________________________________________
> 			R-devel at r-project.org mailing list
> 			https://stat.ethz.ch/mailman/listinfo/r-devel
> 			
> 			  
> 		
> 		
> 		    
> 
> 	
> 	
> 	
> **********************************************************************
> 	The contents of this email are for the named 
> addressee(s...{{dropped:22}}
> 	
> 	______________________________________________
> 	R-devel at r-project.org mailing list
> 	https://stat.ethz.ch/mailman/listinfo/r-devel
> 	
> 	  
> 
> 


**********************************************************************
The contents of this email are for the named addressee(s...{{dropped:22}}



[Rd] 'xtfrm' performance (influences 'order' performance) in R devel
Sklyar, Oleg (London