From ligges at statistik.tu-dortmund.de  Thu Mar 20 23:29:46 2014
From: ligges at statistik.tu-dortmund.de (Uwe Ligges)
Date: Thu, 20 Mar 2014 23:29:46 +0100
Subject: [Rd] [RFC] A case for freezing CRAN
In-Reply-To: <532B6A7D.3070800@fhcrc.org>
References: <129902082.964468.1395296156508.JavaMail.root@fhcrc.org>	<532AC885.1090305@gmail.com>	<532B3E0A.4030907@fhcrc.org>	<CAOTG1hVzjPBhw8yZ5egftKf6dBXKzgUZ_xDBw7fMz+ASk10xTQ@mail.gmail.com>
	<532B6A7D.3070800@fhcrc.org>
Message-ID: <532B6BDA.8070006@statistik.tu-dortmund.de>



On 20.03.2014 23:23, Herv? Pag?s wrote:
>
>
> On 03/20/2014 01:28 PM, Ted Byers wrote:
>> On Thu, Mar 20, 2014 at 3:14 PM, Herv? Pag?s <hpages at fhcrc.org
>> <mailto:hpages at fhcrc.org>> wrote:
>>
>>     On 03/20/2014 03:52 AM, Duncan Murdoch wrote:
>>
>>         On 14-03-20 2:15 AM, Dan Tenenbaum wrote:
>>
>>
>>
>>             ----- Original Message -----
>>
>>                 From: "David Winsemius" <dwinsemius at comcast.net
>>                 <mailto:dwinsemius at comcast.net>>
>>                 To: "Jeroen Ooms" <jeroen.ooms at stat.ucla.edu
>>                 <mailto:jeroen.ooms at stat.ucla.edu>>
>>                 Cc: "r-devel" <r-devel at r-project.org
>>                 <mailto:r-devel at r-project.org>>
>>                 Sent: Wednesday, March 19, 2014 11:03:32 PM
>>                 Subject: Re: [Rd] [RFC] A case for freezing CRAN
>>
>>
>>                 On Mar 19, 2014, at 7:45 PM, Jeroen Ooms wrote:
>>
>>                     On Wed, Mar 19, 2014 at 6:55 PM, Michael Weylandt
>>                     <michael.weylandt at gmail.com
>>                     <mailto:michael.weylandt at gmail.com>> wrote:
>>
>>                         Reading this thread again, is it a fair summary
>>                         of your position
>>                         to say "reproducibility by default is more
>>                         important than giving
>>                         users access to the newest bug fixes and
>>                         features by default?"
>>                         It's certainly arguable, but I'm not sure I'm
>>                         convinced: I'd
>>                         imagine that the ratio of new work being done vs
>>                         reproductions is
>>                         rather high and the current setup optimizes for
>>                         that already.
>>
>>
>>                     I think that separating development from released
>>                     branches can give
>>                     us
>>                     both reliability/reproducibility (stable branch) as
>>                     well as new
>>                     features (unstable branch). The user gets to pick
>>                     (and you can pick
>>                     both!). The same is true for r-base: when using a
>>                     'released'
>>                     version
>>                     you get 'stable' base packages that are up to 12
>>                     months old. If you
>>                     want to have the latest stuff you download a nightly
>>                     build of
>>                     r-devel.
>>                     For regular users and reproducible research it is
>>                     recommended to
>>                     use
>>                     the stable branch. However if you are a developer
>>                     (e.g. package
>>                     author) you might want to develop/test/check your
>>                     work with the
>>                     latest
>>                     r-devel.
>>
>>                     I think that extending the R release cycle to CRAN
>>                     would result
>>                     both
>>                     in more stable released versions of R, as well as
>>                     more freedom for
>>                     package authors to implement rigorous change in the
>>                     unstable
>>                     branch.
>>                     When writing a script that is part of a production
>>                     pipeline, or
>>                     sweave
>>                     paper that should be reproducible 10 years from now,
>>                     or a book on
>>                     using R, you use stable version of R, which is
>>                     guaranteed to behave
>>                     the same over time. However when developing packages
>>                     that should be
>>                     compatible with the upcoming release of R, you use
>>                     r-devel which
>>                     has
>>                     the latest versions of other CRAN and base packages.
>>
>>
>>
>>                 As I remember ... The example demonstrating the need for
>>                 this was an
>>                 XML package that cause an extract from a website where
>>                 the headers
>>                 were misinterpreted as data in one version of pkg:XML
>>                 and not in
>>                 another. That seems fairly unconvincing. Data cleaning
>> and
>>                 validation is a basic task of data analysis. It also
>>                 seems excessive
>>                 to assert that it is the responsibility of CRAN to
>>                 maintain a synced
>>                 binary archive that will be available in ten years.
>>
>>
>>
>>             CRAN already does this, the bin/windows/contrib directory has
>>             subdirectories going back to 1.7, with packages dated
>>             October 2004. I
>>             don't see why it is burdensome to continue to archive these.
>>             It would
>>             be nice if source versions had a similar archive.
>>
>>
>>         The bin/windows/contrib directories are updated every day for
>>         active R
>>         versions.  It's only when Uwe decides that a version is no
>>         longer worth
>>         active support that he stops doing updates, and it "freezes".  A
>>         consequence of this is that the snapshots preserved in those
>> older
>>         directories are unlikely to match what someone who keeps up to
>>         date with
>>         R releases is using.  Their purpose is to make sure that those
>> older
>>         versions aren't completely useless, but they aren't what
>> Jeroen was
>>         asking for.
>>
>>
>>     But it is almost completely useless from a reproducibility point of
>>     view to get random package versions. For example if some people try
>>     to use R-2.13.2 today to reproduce an analysis that was published
>>     2 years ago, they'll get Matrix 1.0-4 on Windows, Matrix 1.0-3 on
>> Mac,
>>     and Matrix 1.1-2-2 on Unix.

Not true, since Matrix 1.1-2-2 has

Depends: 	R (? 2.15.2)


Best,
Uwe Ligges


  And none of them of course is what was
>> used
>>     by the authors of the paper (they used Matrix 1.0-1, which is what
>> was
>>     current when they ran their analysis).
>>
>> Initially this discussion brought back nightmares of DLL hell on
>> Windows.  Those as ancient as I will remember that well.  But now, the
>> focus seems to be on reproducibility, but with what strikes me as a
>> seriously flawed notion of what reproducibility means.
>>
>> Herve Pages mentions the risk of irreproducibility across three minor
>> revisions of version 1.0 of Matrix.
>
> If you use R-2.13.2, you get Matrix 1.1-2-2 on Linux. AFAIK this is
> the most recent version of Matrix, aimed to be compatible with the most
> current version of R (i.e. R 3.0.3). However, it has never been tested
> with R-2.13.2. I'm not saying that it should, that would be a big waste
> of resources of course. All I'm saying it that it doesn't make sense to
> serve by default a version that is known to be incompatible with the
> version of R being used. It's very likely to not even install properly.
>
> For the apparently small differences between the versions you get on
> Windows and Mac, the Matrix package was just an example. With other
> packages you get (again if you use R-2.13.2):
>
>                src   win    mac
>    abc         1.8   1.5    1.4
>    ape       3.1-1 3.0-1    2.8
>    BaSTA     1.9.3   1.1    1.0
>    bcrm      0.4.3   0.2    0.1
>    BMA    3.16.2.3  3.15 3.14.1
>    Boruta    3.0.0   1.6    1.5
>    ...
>
> Are the differences big enough?
>
> Also note that back in October 2011, people using R-2.13.2 would get
> e.g. ape 2.7-3 on Linux, Windows and Mac. Wouldn't it make sense that
> people using R-2.13.2 today get the same? Why would anybody use
> R-2.13.2 today if it's not to run again some code that was written
> and used two years ago to obtain some important results?
>
> Cheers,
> H.
>
>
>> My gut reaction would be that if
>> the results are not reproducible across such minor revisions of one
>> library, they are probably just so much BS.  I am trained in
>> mathematical ecology, with more than a couple decades of post-doc
>> experience working with risk assessment in the private sector.  When I
>> need to do an analysis, I will repeat it myself in multiple products, as
>> well as C++ or FORTRAN code I have hand-crafted myself (and when I wrote
>> number crunching code myself, I would do so in multiple programming
>> languages - C++, Java, FORTRAN, applying rigorous QA procedures to each
>> program/library I developed).  Back when I was a grad student, I would
>> not even show the results to my supervisor, let alone try to publish
>> them, unless the results were reproducible across ALL the tools I used.
>> If there was a discrepancy, I would debug that before discussing them
>> with anyone.  Surely, it is the responsibility of the journals' editors
>> and reviewers to apply a similar practice.
>>
>> The concept of reproducibility used to this point in this discussion
>> might be adequate from a programmers perspective (except in my lab), it
>> is wholly inadequate from a scientist's perspective.  I maintain that if
>> you have the original data, and repeat the analysis using the latest
>> version of R and the available, relevant packages, the original results
>> are probably due to a bug either in the R script or in R or the packages
>> used IF the results obtained using the latest versions of these are not
>> consistent with the originally reported results.  Therefore, of the
>> concerns I see raised in this discussion, the principle one of concern
>> is that of package developers who fail to pay sufficient attention to
>> backwards compatibility: a new version ought not break any code that
>> executes fine using previous versions.  That is not a trivial task, and
>> may require contributors obtaining the assistance of a software
>> engineer.  I am sure anyone in this list who programs in C++ knows how
>> the ANSI committees handle change management.  Introduction of new
>> features is something that is largely irrelevant for backwards
>> compatibility (but there are exceptions), but features to be removed
>> are handled by declaring them deprecated, and leaving them in that
>> condition for years.  That tells anyone using the language that they
>> ought to plan to adapt their code to work when the deprecated feature is
>> finally removed.
>>
>> I am responsible for maintaining code (involving distributed computing)
>> to which many companies integrate their systems, and I am careful to
>> ensure that no change I make breaks their integration into my system,
>> even though I often have to add new features.  And I don't add features
>> lightly, and have yet to remove features.  When that eventually happens,
>> the old feature will be deprecated, so that the other companies have
>> plenty of time to adapt their integration code.  I do not know whether
>> CRAN ought to have any responsibility for this sort of change
>> management, or if they have assumed some responsibility for some of it,
>> but I would argue that the package developers have the primary
>> responsibility for doing this right.
>>
>> Just my $0.05 (the penny no longer exists in Canada)
>>
>> Cheers
>>
>> Ted
>> R.E. (Ted) Byers, Ph.D., Ed.D.
>



[Rd] [RFC] A case for freezing CRAN
Uwe Ligges