From luke-tier@ey m@ili@g off uiow@@edu  Tue Aug  7 17:07:28 2018
From: luke-tier@ey m@ili@g off uiow@@edu (luke-tier@ey m@ili@g off uiow@@edu)
Date: Tue, 7 Aug 2018 10:07:28 -0500 (CDT)
Subject: [Rd] 
 [R-pkg-devel] Run garbage collector when too many open files
In-Reply-To: <da92aa42-00eb-dbbe-5267-1bed89d75a4b@eoos.dds.nl>
References: <627f7758-c5ae-0ab8-b87b-10c6f5a1edc1@eoos.dds.nl>
 <ccb2f0f9-ce61-510f-6f6f-6e41d5aae6b4@statistik.tu-dortmund.de>
 <da92aa42-00eb-dbbe-5267-1bed89d75a4b@eoos.dds.nl>
Message-ID: <alpine.DEB.2.20.1808071004180.3237@luke-Latitude-7480>

In R 3.5 and later you should not need to gc() -- that should happen
automatically within the connections code.

Nevertheless, I would recommend redesigning your approach to avoid
hanging onto open file connections as these are a scarce resource.
You can keep around your temporary files without having them open and
only open/close them on access, with the close run in an on.exit or a
tryCatch/finally clause.

Best,

luke

On Tue, 7 Aug 2018, Jan van der Laan wrote:

> Dear Uwe,
>
> (When replying to your message, I sent the reply to r-devel and not 
> r-package-devel, as Martin Meachler suggested that this thread would be a 
> better fit for r-devel.)
>
> Thanks. In the example below I used rm() explicitly, but in general users 
> wouldn't do that.
>
> One of the reasons for the large number of file handles is that sometimes 
> unnamed temporary objects are created. For example:
>
>> library(ldat)
>> libraty(lvec)
>>
>> a <- lvec(10, "integer")
> OPENFILE '/tmp/RtmpVqkDsw/file32145169fb06/lvec3214753f2af0'
>> b <- as_rvec(a[1:3])
> OPENFILE '/tmp/RtmpVqkDsw/file32145169fb06/lvec32146a50f383'
> OPENFILE '/tmp/RtmpVqkDsw/file32145169fb06/lvec3214484b652c'
>> print(b)
> [1] 0 0 0
>>
>>
>> gc()
> CLOSEFILE '/tmp/RtmpVqkDsw/file32145169fb06/lvec3214484b652c'
> CLOSEFILE '/tmp/RtmpVqkDsw/file32145169fb06/lvec32146a50f383'
>          used (Mb) gc trigger (Mb) max used (Mb)
> Ncells  796936 42.6    1442291 77.1  1168576 62.5
> Vcells 1519523 11.6    4356532 33.3  4740854 36.2
>
>
> For debugging, I log when files are opened and closed. The call a[1:3] (which 
> creates a slice of a) creates two temporary objects [1]. These are only 
> deleted when I explicitly call gc() or on some other random moment in time.
>
> I hope this illustrates the problem better.
>
>
> Best,
> Jan
>
>
> [1] One improvement would be to create less temporary files; often these 
> contain only very little information that is better kept in memory. But that 
> is only a partial solution.
>
>
>
>
> On 07-08-18 15:24, Uwe Ligges wrote:
>> Why not add functionality that allows to delete object + runs cleanup code?
>> 
>> Best,
>> Uwe Ligges
>> 
>> 
>> 
>> On 07.08.2018 14:26, Jan van der Laan wrote:
>>> 
>>> 
>>> In my package I open handles to temporary files from c++, handles to them 
>>> are returned to R through vptr objects. The files are deleted then the 
>>> corresponding R-object is deleted and the garbage collector runs:
>>> 
>>> a <- lvec(10, "integer")
>>> rm(a)
>>> 
>>> Then when the garbage collector runs the file is deleted. However, on some 
>>> platforms (probably with lower limits on the maximum number of file 
>>> handles a process can have open), I run into the problem that the garbage 
>>> collector doesn't run often enough. In this case that means that another 
>>> package of mine using this package generates an error when its tests are 
>>> run.
>>> 
>>> The simplest solution is to add some calls to gc() in my tests. But a more 
>>> general/automatic solution would be nice.
>>> 
>>> I thought about something in the lines of
>>> 
>>> robust_lvec <- function(...) {
>>> ?? tryCatch({
>>> ???? lvec(...)
>>> ?? }, error = function(e) {
>>> ???? gc()
>>> ???? lvec(...) # duplicated code
>>> ?? })
>>> }
>>> 
>>> e.g. try to open a file, when that fails call the garbage collector and 
>>> try again. However, this introduces duplicated code (in this case only one 
>>> line, but that can be more), and doesn't help if it is another function 
>>> that tries to open a file.
>>> 
>>> Is there a better solution?
>>> 
>>> Thanks!
>>> 
>>> Jan
>>> 
>>> ______________________________________________
>>> R-package-devel at r-project.org mailing list
>>> https://stat.ethz.ch/mailman/listinfo/r-package-devel
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>

-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu


[Rd] [R-pkg-devel] Run garbage collector when too many open files
luke-tier@ey m@ili@g off uiow@@edu