From tr@ver@c @end|ng |rom gm@||@com  Wed Jan 23 19:27:22 2019
From: tr@ver@c @end|ng |rom gm@||@com (Travers Ching)
Date: Wed, 23 Jan 2019 10:27:22 -0800
Subject: [Rd] 
 Objectsize function visiting every element for alt-rep strings
In-Reply-To: <b43ad328-4342-dfdd-319b-9b21cf05aecb@gmail.com>
References: <CAPLMX9GKS9QQs9WRBYQQc8f1xc3B2OmXu5URLP=BDa6QFHHVVw@mail.gmail.com>
 <23621.35467.558760.525093@stat.math.ethz.ch>
 <alpine.DEB.2.21.1901220950150.2972@luke-Latitude-7480>
 <CAJXgQP1YwQc0GyxFsw4WVMKntjUYi8chC+hX+BJsFH7wh612=A@mail.gmail.com>
 <b43ad328-4342-dfdd-319b-9b21cf05aecb@gmail.com>
Message-ID: <CAPLMX9HE+gGF-AaO6T1XBxiyRt+gXn=UQ+gv+Drg+h60SqViQA@mail.gmail.com>

It should be possible to calculate object.size in the presence of
sharing, at least with respect to all sub-nodes of a SEXP.  E.g.,
during calculation, keep a hash of all SEXP pointers visited.  If a
pointer has already been visited, add only the size of the pointer to
the total object size.

Travers

On Wed, Jan 23, 2019 at 1:33 AM Tomas Kalibera <tomas.kalibera at gmail.com> wrote:
>
> On 1/22/19 6:17 PM, Kevin Ushey wrote:
> > I think that object.size() is most commonly used to answer the question,
> > "what R objects are consuming the most memory currently in my R session?"
> > and for that reason I think returning the size of the internal
> > representations of objects (for e.g. ALTREP objects; unevaluated promises)
> > is the right default behavior.
>
> I don't think one could answer that question at all in the presence of
> sharing (of objects with value semantics due to copy on write, string
> cache or other caches, sharing of objects with referential semantics
> such as environments, etc). Also the mapping from R objects (SEXPs) to
> what users might understand as objects would not be clear (which SEXPs
> belong to which "object", which SEXPs are too low-level for the user to
> be considered, etc). In principle, there could be a memory profiler
> working at SEXP level and exposing all the intricacies of the memory
> layout, answering reachability questions on a heap dump (so one could
> find out about a 1G integer vector and then list all bindings say in
> namespace environments from which it is reachable), but of course that
> would be a lot of work to implement and to maintain. The problem is not
> unique to R (e.g. see Java with the same problems of sharing that
> prevent meaningful definition for object size). I am not persuaded it
> makes sense to add more options to a function that does not have and
> cannot have a well defined user-level semantics, and I would discourage
> writing code that is trying to build on that function as I think that it
> might lead to confusion and frustration. I think equality for example is
> easier to define (just that one could come up with multiple meaningful
> definitions, so it makes sense to have multiple options).
>
> Best
> Tomas
> >
> > I also agree it would be worth considering adding arguments that control
> > how object.size() is computed for different kinds of R objects, since users
> > might want to use object.size() to answer different types of questions.
> >
> > All that said, if the ultimate goal here is to avoid having RStudio
> > materialize ALTREP objects in the background, then perhaps that change
> > should happen in RStudio :-)
> >
> > Best,
> > Kevin
> >
> > On Tue, Jan 22, 2019 at 8:21 AM Tierney, Luke <luke-tierney at uiowa.edu>
> > wrote:
> >
> >> On Mon, 21 Jan 2019, Martin Maechler wrote:
> >>
> >>>>>>>> Travers Ching
> >>>>>>>>      on Tue, 15 Jan 2019 12:50:45 -0800 writes:
> >>>     > I have a toy alt-rep string package that generates
> >>>     > randomly seeded strings.  example: library(altstringisode)
> >>>     > x <- altrandomStrings(1e8) head(x) [1]
> >>>     > "2PN0bdwPY7CA8M06zVKEkhHgZVgtV1"
> >>>     > "5PN2qmWqBlQ9wQj99nsQzldVI5ZuGX" ... etc object.size(1e8)
> >>>
> >>>     > Object.size will call the set_altstring_Elt_method for
> >>>     > every single element, materializing (slowly) every element
> >>>     > of the vector.  This is a problem mostly in R-studio since
> >>>     > object.size is called automatically, defeating the purpose
> >>>     > of alt-rep.
> >> There is no sensible way in general to figure out how large the
> >> strings would be without computing them. There might be specifically
> >> for a deferred sequence conversion but it would require a fair bit of
> >> effort to figure out that would be better spent elsewhere.
> >>
> >> I've never been a big fan of object.size since what it is trying to
> >> compute isn't very well defined in the context of sharing and possible
> >> internal state changes (even before ALTREP byte code compilation could
> >> change the internals of a function [which object.size sees] and
> >> assigning into environments or evaluating promises can change
> >> environments [which object.size ignores]). The issue is not unlike the
> >> one faced by identical(), which has a bunch of options for the
> >> different ways objects can be identical, and might need even more.
> >>
> >> We could in general have object.size for and ALTREP return the
> >> object.size results of the current internal representation, but that
> >> might not always be appropriate. Again, what object.size is trying to
> >> compute isn't very well defined.
> >>
> >> RStudio does seem to call object.size on every assignment to
> >> .GlobalEnv. That might be worth revisiting.
> >>
> >>
> >> Best,
> >>
> >> luke
> >>
> >>> Hmm.  But still, the idea had been that object.size()  *shuld*
> >>> return the size of the "de-ALTREP'ed" object *but* should not
> >>> de-ALTREP it.
> >>> That's what happens for integers, but indeed fails to happen for
> >>> such as.character(.)ed integers.
> >>>
> >>>  From my eRum presentation (which took from the official ALTREP
> >> documentation
> >>> https://svn.r-project.org/R/branches/ALTREP/ALTREP.html ) :
> >>>
> >>>   > x <- 1:1e15
> >>>   > object.size(x) # 8000'000'000'000'048 bytes : 8000 TBytes -- ok, not
> >> really
> >>>   8000000000000048 bytes
> >>>   > is.unsorted(x) # FALSE : i.e., R's *knows* it is sorted
> >>>   [1] FALSE
> >>>   > xs <- sort(x)  #
> >>>   > .Internal(inspect(x))
> >>>   @80255f8 14 REALSXP g0c0 [NAM(7)]  1 : 1000000000000000 (compact)
> >>>   >
> >>>
> >>>   > cx <- as.character(x)
> >>>   > .Internal(inspect(cx))
> >>>   @80485d8 16 STRSXP g0c0 [NAM(1)]   <deferred string conversion>
> >>>     @80255f8 14 REALSXP g1c0 [MARK,NAM(7)]  1 : 1000000000000000 (compact)
> >>>   > system.time( print(object.size(x)), gc=FALSE)
> >>>   8000000000000048 bytes
> >>>      user  system elapsed
> >>>     0.000   0.000   0.001
> >>>   > system.time( print(object.size(cx)), gc=FALSE)
> >>>   Error: cannot allocate vector of size 8388608.0 Gb
> >>>   Timing stopped at: 11.43 0 11.46
> >>>   >
> >>>
> >>> One could consider it a bug that object.size(cx) is indeed
> >>> inspecting every string, i.e., accessing cx[i] for all i.
> >>> Note that it is *not*  deALTREPing cx  itself :
> >>>
> >>>> x <- 1:1e6
> >>>> cx <- as.character(x)
> >>>> .Internal(inspect(cx))
> >>> @7f5b1a0 16 STRSXP g0c0 [NAM(1)]   <deferred string conversion>
> >>>   @7f5adb0 13 INTSXP g0c0 [NAM(7)]  1 : 1000000 (compact)
> >>>> system.time( print(object.size(cx)), gc=FALSE)
> >>> 64000048 bytes
> >>>    user  system elapsed
> >>>   0.369   0.005   0.374
> >>>> .Internal(inspect(cx))
> >>> @7f5b1a0 16 STRSXP g0c0 [NAM(7)]   <deferred string conversion>
> >>>   @7f5adb0 13 INTSXP g0c0 [NAM(7)]  1 : 1000000 (compact)
> >>>     > Is there a way to avoid the problem of forced
> >>>     > materialization in rstudio?
> >>>
> >>>     > PS: Is there a way to tell if a post has been received by
> >>>     > the mailing list?  How long does it take to show up in the
> >>>     > archives?
> >>>
> >>> [ that (waiting time) distribution is quite right skewed... I'd
> >>>   guess it's median to be less than 10 minutes... but we had
> >>>   artificially delayed it somewhat in the past to fight
> >>>   spammers, and ETH (the hosting instituttion) and others have
> >>>   increased spam and virus filtering so everything has become
> >>>   quite a bit slower ]
> >>>
> >>> ______________________________________________
> >>> R-devel at r-project.org mailing list
> >>> https://stat.ethz.ch/mailman/listinfo/r-devel
> >>>
> >> --
> >> Luke Tierney
> >> Ralph E. Wareham Professor of Mathematical Sciences
> >> University of Iowa                  Phone:             319-335-3386
> >> Department of Statistics and        Fax:               319-335-3017
> >>      Actuarial Science
> >> 241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
> >> Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu
> >>
> >> ______________________________________________
> >> R-devel at r-project.org mailing list
> >> https://stat.ethz.ch/mailman/listinfo/r-devel
> >>
> >       [[alternative HTML version deleted]]
> >
> > ______________________________________________
> > R-devel at r-project.org mailing list
> > https://stat.ethz.ch/mailman/listinfo/r-devel
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel



[Rd] Objectsize function visiting every element for alt-rep strings
Travers Ching