From tom@@@k@||ber@ @end|ng |rom gm@||@com  Thu Apr 25 13:57:32 2019
From: tom@@@k@||ber@ @end|ng |rom gm@||@com (Tomas Kalibera)
Date: Thu, 25 Apr 2019 13:57:32 +0200
Subject: [Rd] Questions/suggestions about new staged installation
In-Reply-To: <ba06c79e-757e-0414-d977-8a5b41f46755@fredhutch.org>
References: <ba06c79e-757e-0414-d977-8a5b41f46755@fredhutch.org>
Message-ID: <cefb09df-c755-f753-a780-986b7e38aa05@gmail.com>

On 4/25/19 3:11 AM, Pages, Herve wrote:
> Hi,
>
> I was playing around with inotifywait (great tool!) to see the new
> staged installation of source packages in action. In one terminal I'm
> monitoring the create/delete/move events of the installation library with:
>
>   ? inotifywait -m --timefmt '%F %T' --format '%T -- %w %e %f' -e create
> -e delete -e move path/to/R/library/
>
> While in another terminal I install CRAN package abc with:
>
>   ? R CMD INSTALL abc_2.1.tar.gz
>
> All deps are already installed.
>
> ## With R 3.5
>
> When installing abc the first time with R 3.5 inotifywait reports the
> following events:
>
> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 16:57:42 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR abc
> 2019-04-24 16:57:45 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
> 00LOCK-
>
> Then on subsequent times:
>
> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/
> MOVED_FROM,ISDIR abc
> 2019-04-24 16:58:14 -- /home/hpages/R/R-3.5.r75051/library/ CREATE,ISDIR abc
> 2019-04-24 16:58:17 -- /home/hpages/R/R-3.5.r75051/library/ DELETE,ISDIR
> 00LOCK-abc
>
> IIUC the additional 'MOVED_FROM,ISDIR abc'? event on subsequent installs
> is triggered by the backing up of the earlier installation in case the
> new installation fails.
>
> ## With R 3.6
>
> First installation:
>
> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 17:09:04 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR abc
> 2019-04-24 17:09:08 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_TO,ISDIR abc
> 2019-04-24 17:09:09 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
> 00LOCK-abc
>
> Subsequent installations:
>
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_FROM,ISDIR abc
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR abc
> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_TO,ISDIR abc
> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
> 00LOCK-abc
>
> IIUC the new 'MOVED_TO,ISDIR abc' event is triggered by the package
> installation folder being moved from the temporary location to the final
> location.
>
> However I wonder about the CREATE,ISDIR abc event. It looks like even
> for a staged installation tools:::.install_packages() still creates the
> empty abc dir. This should no longer be necessary because this empty
> folder is replaced later by the 'MOVED_TO,ISDIR abc' event. Am I missing
> somethin

Yes, the final directory location gets created before the installation. 
I did this to minimize the added cognitive complexity to the code. I 
could explore if this could be changed, if the installation could work 
without that directory (which would mean implementing and testing on all 
CRAN/BIOC, installing in different modes, so certainly some effort).? Is 
the current implementation causing any problem?

> Another thing: the times indicate that the 'MOVED_TO,ISDIR abc' event
> (move from temp to final location) happens about 3-4 seconds after the
> 'MOVED_FROM,ISDIR abc' event (backing up). In this interval of time, the
> abc package is missing. Wouldn't it be safer to leave the earlier
> installation where it is and to replace it when the new installation
> folder is moved from the temporary location to the final location? Said
> otherwise, the backing up step no longer seems necessary with staged
> installs. So inotifywait would report something like this:
>
> 2019-04-24 17:09:59 -- /home/hpages/R/R-3.6.r76385/library/ CREATE,ISDIR
> 00LOCK-abc
> 2019-04-24 17:10:02 -- /home/hpages/R/R-3.6.r76385/library/
> MOVED_TO,ISDIR abc
> 2019-04-24 17:10:03 -- /home/hpages/R/R-3.6.r76385/library/ DELETE,ISDIR
> 00LOCK-abc
>
> That is, only 3 events. On first and subsequent installation.

I don't think that would be possible. The package is test-loaded also 
from the final installation directory, and if that fails, restored from 
a back-up. This test-loading is useful as it can detect some problems 
with hard-coded directories possibly not detected before on test-loading 
from the temporary installation directory.

Best,
Tomas

>
> Cheers,
>
> H.
>
>
>   > sessionInfo()
> R version 3.6.0 beta (2019-04-12 r76385)
> Platform: x86_64-pc-linux-gnu (64-bit)
> Running under: Ubuntu 16.04.5 LTS
>
> Matrix products: default
> BLAS:?? /home/hpages/R/R-3.6.r76385/lib/libRblas.so
> LAPACK: /home/hpages/R/R-3.6.r76385/lib/libRlapack.so
>
> locale:
>   ?[1] LC_CTYPE=en_US.UTF-8?????? LC_NUMERIC=C
>   ?[3] LC_TIME=en_US.UTF-8??????? LC_COLLATE=en_US.UTF-8
>   ?[5] LC_MONETARY=en_US.UTF-8??? LC_MESSAGES=en_US.UTF-8
>   ?[7] LC_PAPER=en_US.UTF-8?????? LC_NAME=C
>   ?[9] LC_ADDRESS=C?????????????? LC_TELEPHONE=C
> [11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C
>
> attached base packages:
> [1] stats???? graphics? grDevices utils???? datasets? methods base
>
> loaded via a namespace (and not attached):
> [1] compiler_3.6.0
>
>



[Rd] Questions/suggestions about new staged installation
Tomas Kalibera