From krivitsky at stat.psu.edu  Sat May 26 19:02:34 2012
From: krivitsky at stat.psu.edu (Pavel N. Krivitsky)
Date: Sat, 26 May 2012 13:02:34 -0400
Subject: [Rd] [patch] Behavior of .C() and .Fortran() when given
 double(0) or integer(0).
In-Reply-To: <4FB4C900.6070204@stats.ox.ac.uk>
References: <1336153356.24356.332.camel@Navi.krivitsky.homeip.net>
	<4FB4C900.6070204@stats.ox.ac.uk>
Message-ID: <1338051754.4646.165.camel@Navi.krivitsky.homeip.net>

Dear Professor Ripley and R-devel,

Thank you for taking the time to look into this. I understand that the
2.15.0 behavior in question was undocumented and ambiguous (at least as
of that release), and it should not have been relied upon, intuitiveness
or not. My suggestion is that in the next release, it ought to be the
standard, documented behavior, not just because it's historical, but
because it's more convenient and safer.

>From the point of view of programmer convenience, a having a 0-length
vector on the R side always map to a NULL pointer on the C side provides
a useful bit of information that the programmer can use, while a
non-NULL pointer to no data isn't useful, and the current R-devel
behavior requires the programmer to pass the information about whether
it's empty through an additional argument (of which there is an upper
limit). For example, if a procedure implemented in C takes optional
weights, passing a double(0) that was translated to NULL could be used
to signal that there are no weights. Also, while the .Call() interface
allows an R vector passed to it to be resized, the .C() and .Fortran()
interfaces don't, so a 0-length R vector passed via .C() or .Fortran()
can be neither read nor written to, and nothing is lost by passing it as
NULL.

On the issue of safety, while dereferencing a NULL pointer is not an
instant segfault on absolutely every system, it is the case for the
overwhelming majority of modern systems on which anyone is likely to run
a recent version of R. For those systems for which it is not the case,
the behavior is no worse than dereferencing a non-NULL pointer to no
data. On the contrary, while it's easy to check if a pointer is NULL,
there is no general way to check whether a non-NULL pointer is valid, so
if the 0-length->NULL behavior is made the standard and documented,
package developers may be more likely to make use of it to check.

On the issue of instrumentation and debugging, again, I think it comes
down to programmer convenience. Segmentation faults caused by NULL
dereferencing can be caught and debugged interactively with a debugger
like GDB, while non-NULL memory errors have less predictable
consequences and require tools like the slower and non-interactive
Valgrind. Perhaps R's new guard bytes will change that somewhat, but,
from what I've read, they only check for invalid writes, not invalid
reads, which can cause almost as much trouble. On the other hand, both
trying to read from a NULL pointer and trying to write to it will be
detected on most systems. And, having 0-length vectors be passed as NULL
does not preclude using guard bytes on non-0-length vectors.

To summarize, I think that on both safety and convenience, standardizing
on the 0-length->NULL behavior dominates the 0-length->invalid-pointer
behavior: in each scenario that either of us has brought up so far, it
behaves no worse and often better. My patch does not include changes to
documentation, and, if you like, I am willing to write one that does. If
my patch can be improved in some other way, please let me know and I
will try to improve it.

                            Sincerely,
                            Pavel Krivitsky



On Thu, 2012-05-17 at 10:46 +0100, Prof Brian Ripley wrote:
> On 04/05/2012 18:42, Pavel N. Krivitsky wrote:
> > Dear R-devel,
> >
> > While tracking down some hard-to-reproduce bugs in a package I maintain,
> > I stumbled on a behavior change between R 2.15.0 and the current R-devel
> > (or SVN trunk).
> >
> > In 2.15.0 and earlier, if you passed an 0-length vector of the right
> > mode (e.g., double(0) or integer(0)) as one of the arguments in a .C()
> > call with DUP=TRUE (the default), the C routine would be passed NULL
> > (the C pointer, not R NULL) in the corresponding argument. The current
> 
> Where did you get that from?  The documentation says it passes an (e.g.) 
> double* pointer to a copy of the data area of the R vector.  There is no 
> change in the documented behaviour ....  Now, of course a zero-length 
> area can be at any address, but none is stated anywhere that I am aware of.
> 
> > development version instead passes it a pointer to what appears to be
> > memory location immediately following the the SEXP that holds the
> > metadata for the argument. If the argument has length 0, this is often
> > memory belonging to a different R object. (DUP=FALSE in 2.15.0
> > appears to have the same behavior as R-devel.)
> >
> > .C() documentation and Writing R Extensions don't explicitly specify a
> > behavior for 0-length vectors, so I don't know if this change is
> > intentional, or whether it was a side-effect of the following news item:
> >
> >        .C() and .Fortran() do less copying: arguments which are raw,
> >        logical, integer, real or complex vectors and are unnamed are not
> >        copied before the call, and (named or not) are not copied after
> >        the call.  Lists are no longer copied (they are supposed to be
> >        used read-only in the C code).
> >
> > Was the change in the empty vector behavior intentional?
> >
> > It seems to me that standardizing on the behavior of giving the C
> > routine NULL is safer, more consistent with other memory-related
> > routines, and more convenient: whereas dereferencing a NULL pointer is
> > an immediate (and therefore easily traced) segfault, dereferencing an
> 
> That's not true, in general.
> 
> > invalid pointer that is nevertheless in the general memory area
> > allocated to the program often causes subtle errors down the line;
> > R_alloc asked to allocate 0 bytes returns NULL, at least on my platform;
> 
> Again, undocumented and should not be relied on.
> 
> > and the C routine can easily check if a pointer is NULL, but with the
> > R-devel behavior, the programmer has to add an explicit way of telling
> > that an empty vector was passed.
> 
> It's no different from any other vector length: it is easy for careless 
> programmers to read/write off the ends of the allocated area, and this 
> is why in R-devel we have an option to check for that (and of course 
> also what valgrind is good at finding in an instrumented version of R).
> 
> > I've attached a small test case (dotC_NULL.* files) that shows the
> > difference. The C file should be built with R CMD SHLIB, and the R file
> > calls the functions in the library with a variety of arguments. Output I
> > get from running
> > R CMD BATCH --no-timing --vanilla --slave dotC_NULL.R
> > on R 2.15.0, R trunk, and R trunk with my patch (described below) are attached.
> >
> > The attached patch (dotC_NULL.patch) against the current trunk
> > (affecting src/main/dotcode.c) restores the old behavior for DUP=TRUE
> > (i.e., 0-length vector ->  NULL pointer) and extends it to the DUP=FALSE
> > case. It does so by checking if an argument --- if it's of mode raw,
> > integer, real, or complex --- to a .C() or .Fortran() call has length 0,
> > and, if so, sets the pointer to be passed to NULL and then skips the
> > copying of the C routine's changes back to the R object for that
> > argument. The additional computing cost should be negligible (i.e.,
> > checking if vector length equals 0 and break-ing out of a switch
> > statement if so).
> >
> > The patch appears to work, at least for my package, and R CMD check
> > passes for all recommended packages (on my 64-bit Linux system), but
> > this is my first time working with R's internals, so handle with care.
> 
> That's easy: we will not be changing this.  In particular, the new 
> checks I refer to above rely on passing the address of an in-process 
> memory area with guard bytes.
> 
> >                                     Best,
> >                                     Pavel Krivitsky
> >
> >
> >
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
> 
>



[Rd] [patch] Behavior of .C() and .Fortran() when given double(0) or integer(0).
Pavel N. Krivitsky