From murdoch@dunc@n @end|ng |rom gm@||@com  Thu Oct 10 00:23:53 2019
From: murdoch@dunc@n @end|ng |rom gm@||@com (Duncan Murdoch)
Date: Wed, 9 Oct 2019 18:23:53 -0400
Subject: [Rd] S3 lookup rules changed in R 3.6.1
In-Reply-To: <CAM2gKPZfigRJL1yfyMq-hC3K6QVcswx05NM8HaJ41HBzom4fOQ@mail.gmail.com>
References: <CAM2gKPZfigRJL1yfyMq-hC3K6QVcswx05NM8HaJ41HBzom4fOQ@mail.gmail.com>
Message-ID: <004fe769-cfbd-73e5-a567-cbe7928c1229@gmail.com>

On 09/10/2019 3:22 p.m., Konrad Rudolph wrote:
> tl;dr: S3 lookup no longer works in custom non-namespace environments as of
> R 3.6.1. Is this a bug?

I don't know whether this was intentional or not, but a binary search 
through the svn commits finds that the errors started in this one:

------------------------------------------------------------------------
r75127 | hornik | 2018-08-13 09:58:47 -0400 (Mon, 13 Aug 2018) | 2 lines
Changed paths:
    M /trunk/src/main/objects.c
    M /trunk/tests/reg-tests-1a.R

Have S3 methods lookup by default look for the S3 registry in the topenv
of the generic.
------------------------------------------------------------------------

Duncan Murdoch

> 
> I am implementing S3 dispatch for generic methods in environments that are
> not
> packages. I am trying to emulate the R package namespace mechanism by
> having a
> ?namespace? environment that defines generics and methods, but only exposes
> the
> generics themselves, not the methods.
> 
> To make S3 lookup work when using the generics, I am using
> `registerS3method`.
> While this method itself has no extensive documentation, the documentation
> of
> `UseMethod` contains this relevant passage:
> 
>> Namespaces can register methods for generic functions. To support this,
>> ?UseMethod? and ?NextMethod? search for methods in two places: in the
>> environment in which the generic function is called, and in the
> registration
>> data base for the environment in which the generic is defined (typically a
>> namespace). So methods for a generic function need to be available in the
>> environment of the call to the generic, or they must be registered. (It
> does
>> not matter whether they are visible in the environment in which the
> generic is
>> defined.) As from R 3.5.0, the registration data base is searched after
> the
>> top level environment (see ?topenv?) of the calling environment (but
> before
>> the parents of the top level environment).
> 
> This used to work but it stopped working in R 3.6.1 and I cannot figure out
> (a)
> why, and (b) how to fix it. Unfortunately I am unable to find the relevant
> information by reading the R source code, even when ?diff?ing what seem to
> be
> the only even remotely relevant changes [1].
> 
> The R NEWS merely list the following change for R 3.6.0:
> 
>> * S3method() directives in ?NAMESPACE? can now also be used to perform
> delayed
>>    S3 method registration.
>> [?]
>> * Method dispatch uses more relevant environments when looking up class
>>    definitions.
> 
> Unfortunately it is not clear to me what exactly this means.
> 
> Here?s a minimal example code that works under R 3.5.3 but breaks under
> R 3.6.1
> (I don?t know about 3.6.0).
> 
> ```
> # Define ?package namespace?:
> ns = new.env(parent = .BaseNamespaceEnv)
> local(envir = ns, {
>      test = function (x) UseMethod('test')
>      test.default = function (x) message('test.default')
>      test.foo = function (x) message('test.foo')
> 
>      .__S3MethodsTable__. = new.env(parent = .BaseNamespaceEnv)
>      .__S3MethodsTable__.$test.default = test.default
>      .__S3MethodsTable__.$test.foo = test.foo
> 
>      # Or, equivalently:
>      # registerS3method('test', 'default', test.default)
>      # registerS3method('test', 'foo', test.foo)
> })
> 
> # Expose generic publicly:
> test = ns$test
> 
> # Usage:
> test(1)
> test(structure(1, class = 'foo'))
> ```
> 
> Output in R up to 3.5.3:
> 
> ```
> test.default
> test.foo
> ```
> 
> Output in R 3.6.1:
> 
> ```
> Error in UseMethod("test") :
>    no applicable method for 'test' applied to an object of class
> "c('double', 'numeric')"
> ```
> 
> It?s worth noting that the output of `.S3methods` is the same for all R
> versions, and from my understanding of its output, this *should* indicate
> that
> S3 lookup should behave identically, too. Furthermore, lookup via
> `getS3method`
> succeeds in all R versions, and (again, in my understanding) the logic of
> this
> function should be identical to the logic of R?s internal S3 dispatch:
> 
> ```
> getS3method('test', 'default')(1)
> getS3method('test', 'foo')(1)
> ```
> 
> Conversely, specialising an existing generic from a loaded package works.
> E.g.:
> 
> ```
> local(envir = ns, {
>      print.foo = function (x) message('print.foo')
>      registerS3method('print', 'foo', print.foo)
> })
> 
> print(structure(1, class = 'foo'))
> ```
> 
> This prints ?print.foo? in all R versions as expected.
> 
> So my question is: Why do the `test(?)` calls in R 3.6.1 no longer trigger
> S3
> method lookup in the generic function?s environment? Is this behaviour by
> design
> or is it a bug? If it?s by design, why does `getS3method` still use the old
> behaviour? And, most importantly, how can I fix my definition of `ns` to
> make
> S3 dispatch for non-exposed methods work again?
> 
> ? actually I just found a workaround:
> 
> ```
> ns$.packageName = 'not important'
> ```
> 
> This marks `ns` as a package namespace. To me, the documentation seems to
> imply
> that this shouldn?t be necessary (and it previously wasn?t). Furthermore,
> the
> code for `registerS3method` explicitly supports non-package namespace
> environments. Unfortunately this workaround is not satisfactory because
> pretending that the environment is a package namespace, when it really
> isn?t,
> might break other things.
> 
> [1] See r75273; there?s also r74625, which changes the actual lookup
> mechanism
>      used by `UseMethod`, but that seems even less relevant, because it is
>      disabled unless a specific environment variable is set.
>



[Rd] S3 lookup rules changed in R 3.6.1
Duncan Murdoch