From kasperdanielhansen at gmail.com  Tue Sep  4 23:46:06 2012
From: kasperdanielhansen at gmail.com (Kasper Daniel Hansen)
Date: Tue, 4 Sep 2012 17:46:06 -0400
Subject: [Rd] if(--as-cran)?
In-Reply-To: <50467387.3030508@gmail.com>
References: <50465D7F.5060004@gmail.com>
	<0CAEE2B1235DF643B0F5E1EAE1487B530946D655@023-CH1MPN1-043.023d.mgd.msft.net>
	<CAC2h7utRwFeb5_z5_gLUF=6b_cMw+CAgPnfqU_utMR7TPwu7zg@mail.gmail.com>
	<000301cd8ae3$44a362a0$cdea27e0$@mcmaster.ca>
	<50467387.3030508@gmail.com>
Message-ID: <CAC2h7uvSU_K6wu4i7C7Px1v-VSJ4bT_C2C9eeZ+mJF=eaxdX2w@mail.gmail.com>

On Tue, Sep 4, 2012 at 5:32 PM, Duncan Murdoch <murdoch.duncan at gmail.com> wrote:
> On 04/09/2012 5:21 PM, John Fox wrote:
>>
>> Dear all,
>>
>> I'd like to second this fairly simple request. I currently enclosed some
>> of
>> the examples in the effects package in \donttest{} blocks to satisfy the
>> CRAN timing requirements for examples. It would be nice to have something
>> like a \donttestcran{} block that suppresses the tests when --as-cran is
>> set
>> (and on CRAN itself).
>>
>> I'm sure that I've missed many of the nuances in this discussion, but this
>> seems like a simple solution to me.
>
>
> That would work for examples, but not tests.  Many packages have scripts
> that are in the tests directory, not just test code in the .Rd files.  What
> I think you should use is my suggested HowMuchTimeLeftBeforeTimeout()
> function, but if that's not writeable, then simply having a TIMELIMIT
> environment variable or similar, so your code could see if the there's a
> timeout pending.

I don't like this idea, because then I need to think about the order
in which the tests are run.

You are right that we could simulate this by having an environment
variable, and if that variable is set, we could do the full tests.
Like Dirk is mentioning, the (sensible, I might add) requirement that
the tests should be "quick" suddenly makes the quick testing the
default.

What I - and I think a lot of other people - would like, is to do this
in a standard way so it would be uniform across packages.  I don't
think it would be good if I start using R_LONG_TESTS and Dirk starts
to use __NON_CRAN_TESTS or whatever.  Then all of us need to know the
package-specific variables when we test other people packages (and
that happens occasionally, for example when you have a package that
other packages depend upon).

In this use case, standardization would be hugely beneficial in my
opinion.  In my reading of the discussion, this is really what all of
us are saying.

>
> Duncan Murdoch
>
>
>>
>> Best,
>>   John
>>
>> > -----Original Message-----
>> > From: r-devel-bounces at r-project.org [mailto:r-devel-bounces at r-
>> > project.org] On Behalf Of Kasper Daniel Hansen
>> > Sent: Tuesday, September 04, 2012 5:12 PM
>> > To: Warnes, Gregory
>> > Cc: Terry Therneau; r-devel at r-project.org
>> > Subject: Re: [Rd] if(--as-cran)?
>> >
>> > On Tue, Sep 4, 2012 at 4:53 PM, Warnes, Gregory
>> > <gregory.warnes at novartis.com> wrote:
>> > >
>> > > On 9/4/12 3:58 PM, "Duncan Murdoch" <murdoch.duncan at gmail.com> wrote:
>> > >
>> > >>On 04/09/2012 3:44 PM, Terry Therneau wrote:
>> > >>>ly in
>> > >>> On 09/04/2012 01:57 PM, Duncan Murdoch wrote:
>> > >>> > On 04/09/2012 2:36 PM, Warnes, Gregory wrote:
>> > >>> >> On 9/4/12 8:38 AM, "Duncan Murdoch" <murdoch.duncan at gmail.com>
>> > wrote:
>> > >>> >>
>> > >>> >>
>> > >>> >> >On 04/09/2012 8:20 AM, Terry Therneau wrote:
>> > >>> >> >>
>> > >>> >> >> On 09/04/2012 05:00 AM, M
>> > >>><mailto:r-devel-request at r-project.org>artin wrote:
>> > >>> >> >> > The issue is not just about "CRAN" vs "off CRAN".
>> > >>> >> >> > It is good to think about a more general scheme of
>> > >>> >> >> > "light testing" vs "normal testing" vs "extensive testing",
>> > >>> >> >> > e.g., for the situation where the package implements
>> > >>> >> >> > (simulation/bootstrap/ ..) based inference, and the
>> > developer
>> > >>> >> >> > (but not only) should be able to run the extensive tests.
>> > >>> >> >> >
>> > >>> >> >> > Martin
>> > >>> >> >>
>> > >>> >> >> I agree with Martin. A mechanism to specify testing level
>> > would
>> > >>>be the
>> > >>> >> >> best. Then CRAN can choose to set that variable to "3" say,
>> > with
>> > >>>level
>> > >>> >> >> 1 for extensive and 2 for usual.
>> > >>>>> >>
>> > >>>
>> > >>>[snip]
>> > >
>> > >>The testingLevel() function is supposed to be a way to know that a
>> > >>certain level of testing is being done, to allow such tailoring.
>> > >>However, I don't think it's practical.  I think you can ask whether a
>> > >>specific test is being run (my "D" %in% tests() example), but you
>> > can't
>> > >>reasonably convert the set of tests chosen by a tester into a single
>> > >>number.
>> > >>
>> > >>What I think you and Greg are talking about is something different.
>> > You
>> > >>are asking that we set up more suites of tests, corresponding to
>> > >>numerical levels.  Currently we have two suites:  the default, and the
>> > >>--as-cran suite.  But we also have completely customized suites, set
>> > by
>> > >>users who want to check specific things.  They can do that the way you
>> > >>do (by calling the tests explicitly), or by setting environment
>> > >>variables (as described in the Tools chapter of the R Internals
>> > manual).
>> > >
>> > > No!  We're not asking for the r-core to create more test suites, or
>> > even
>> > > to do anything different based on the test intensity level.
>> > >
>> > > We're just asking for a standard way to control the intensity of the
>> > tests
>> > > *we* write to prevent us from duplicating this functionality in our
>> > own
>> > > packages, probably in incompatible ways.
>> >
>> > And given that CRAN recently put down timing requirements (and
>> > Bioconductor has had them for a long time), it could be extremely
>> > useful to have one system.  It is not clear to me whether it needs
>> > more than 2 levels ("slow" and "fast"), but I'll leave that up to
>> > people who have thought longer about this.
>> >
>> > I could certainly use it in several packages to differentiate between
>> > slow and quick tests.
>> >
>> > Kasper
>> >
>> > ______________________________________________
>> > R-devel at r-project.org mailing list
>> > https://stat.ethz.ch/mailman/listinfo/r-devel
>>
>



[Rd] if(--as-cran)?
Kasper Daniel Hansen