From hpages at fredhutch.org  Tue Nov  8 18:59:37 2016
From: hpages at fredhutch.org (=?UTF-8?B?SGVydsOpIFBhZ8Oocw==?=)
Date: Tue, 8 Nov 2016 09:59:37 -0800
Subject: [Rd] Running package tests and not stop on first fail
In-Reply-To: <22561.39997.738866.706206@stat.math.ethz.ch>
References: <CABE2sp66ywzHW9LFOJ-Z0mRWWro62ZFGXppiFxPVqJC0gpssdQ@mail.gmail.com>
	<22555.5435.945673.692470@stat.math.ethz.ch>
	<4527a200-d3a1-bbed-6687-697aabef5b89@fredhutch.org>
	<CABE2sp5Da4vc8Qksk8nD_bn-pv2+MJcrYaB-2J44HGo5eow3pg@mail.gmail.com>
	<22556.43073.490072.801815@stat.math.ethz.ch>
	<22556.43796.305280.635245@max.nulle.part>
	<22556.45002.81234.741856@stat.math.ethz.ch>
	<CADRwj9-eK4tufYNnKazU_cpVuKb=ac9FXCTKx63DFGm+wqPM1A@mail.gmail.com>
	<20161105205308.2E1FB140692@ion-3.math.ethz.ch>
	<e7b32111-aa37-4470-8b54-b752a9a914ae@fredhutch.org>
	<22561.39997.738866.706206@stat.math.ethz.ch>
Message-ID: <0d678886-6690-88e6-d791-3fdfa9484a01@fredhutch.org>

Thanks Martin.

These changes are great and improve the usefulness of 'R CMD check'.
Especially in the context of the Bioconductor daily builds where
we'll use --no-stop-on-test-error so developers will get a full
picture of all the errors in their package at once.

Cheers,
H.

To provide some context to my special interest for this,

On 11/08/2016 01:34 AM, Martin Maechler wrote:
>>>>>> Herv? Pag?s <hpages at fredhutch.org>
>>>>>>     on Mon, 7 Nov 2016 14:37:15 -0800 writes:
>
>     > On 11/05/2016 01:53 PM, Martin Maechler wrote:
>     >>>>>>> Oliver Keyes <ironholds at gmail.com>
>     >>>>>>> on Fri, 4 Nov 2016 12:42:54 -0400 writes:
>     >>
>     >> > On Friday, 4 November 2016, Martin Maechler
>     >> > <maechler at stat.math.ethz.ch> wrote:
>     >>
>     >> >> >>>>> Dirk Eddelbuettel <edd at debian.org <javascript:;>>
>     >> >> >>>>> on Fri, 4 Nov 2016 10:36:52 -0500 writes:
>     >> >>
>     >> >> > On 4 November 2016 at 16:24, Martin Maechler wrote: |
>     >> >> My > proposed name '--no-stop-on-error' was a quick shot;
>     >> >> if | > somebody has a more concise or better "English
>     >> >> style" > wording | (which is somewhat compatible with all
>     >> >> the other > options you see | from 'R CMD check --help'),
>     >> >> | please > speak up.
>     >> >>
>     >> >> > Why not keep it simple?  The similar feature this most
>     >> >> > resembles is 'make -k' and its help page has
>     >> >>
>     >> >> > -k, --keep-going
>     >> >>
>     >> >> > Continue as much as possible after an > error.  While
>     >> >> the target that failed, and those that > depend on it,
>     >> >> cannot be remade, the other dependencies of > these
>     >> >> targets can be processed all the same.
>     >> >>
>     >> >> Yes, that would be quite a bit simpler and nice in my
>     >> >> view.  One may think it to be too vague,
>     >>
>     >> > Mmn, I would agree on vagueness (and it breaks the pattern
>     >> > set by other flags of human-readability). Deep familiarity
>     >> > with make is probably not something we should ask of
>     >> > everyone who needs to test a package, too.
>     >>
>     >> > I quite like stop-on-error=true (exactly the same as the
>     >> > previous suggestion but shaves off some characters by
>     >> > inverting the Boolean)
>     >>
>     >> Thank you, Brian, Dirk and Oliver for these (and some offline)
>     >> thoughts and suggestions!
>     >>
>     >> My current summary:
>     >>
>     >> 1) I really don't want a  --<option-key>=value
>     >> but rather stay with logical/binary variables that "express
>     >> themselves"... in the same way I strongly prefer
>     >>
>     >> if (A_is_special)       ....
>     >> to
>     >> if (A_special == TRUE)  ....
>     >>
>     >> for a logical variable A_* .   Yes, this is mostly a matter
>     >> of taste,.. but related to how R style itself "works"
>     >>
>     >> 2) Brian mentioned that this is only about ./tests/ tests which
>     >> are continued, not about the Examples which are treated separately.
>     >> That's why we had contemplated additionally using 'tests' (because that's
>     >> the directory name used for unit/regression/.. tests) in the option
>     >> name.
>     >>
>     >> Even though Brian is correct, ideally we *would* want to also influence the
>     >> examples' running to *not* stop on a first error..   However that would
>     >> need more work, reorganizing how the examples are run and that may not be
>     >> worth the pain.   However it should be considered a goal in the long run.
>
>     > My name is Herv?, and I was not suggesting that what happens with the
>     > examples should be changed. I was just preaching consistency (again
>     > sorry) between what happens with the examples and what happens with
>     > the tests.
>
> Thank you, Herv? and excuse me for not answering more focused on
> what you said.
> I think I do understand what you say (at least by now :-)) and
> agree that consistency is something important and to be strived for,
> also with these options.
>
>     > Why not simply change the latter?
>     > Do we really need an option to control this?
>
> Very good questions.  If the change could be made much better,
> I'd agree we'd not need a new option because the change could be
> considerided uniformly better than the current (R 3.3.2, say) behavior.
> However the change as it is currently, is not good enough to be
> the only option (see below).
>
>     > The behavior was changed for the examples a couple of
>     > years ago and nobody felt the need to introduce an option
>     > to control this at the time.
>
> Yes, that change was made very nicely (not by me) and I'd say
> the result *was* uniformly better than the previous behavior, so
> there did not seem much of a reason to still provide the old behavior.
>
>       >> After all that, I tend to remain with the original proposed name. It is at
>       >> least easy to pronounce and spell correctly...
>
>     > Unless --no-stop-on-error controls both (i.e. examples and tests), and
>     > both behave the same way by default, this is a misnomer.
>
> Well, if you choose the option, there *is* no stop on errors anymore
> ... because the examples nowadays never stop the (R CMD check Pkg)
> from running on an error as we've mentioned above.
>
> [[--- Digression on  Examples' checking
>
>   However / on the other hand: Because of the way the examples
>   are run --- efficiently from one single R source file --- it
>   is not so easy there, to let them run further: The first error
>   from all the examples stops running the example-checks, i.e.,
>   the <pkg>-Ex.R script, but at least *does* continue to run the
>   next 'R CMD check ...' steps.
>
>   To change this without incurring considerable drawbacks, or
>   involve a lot of changes does not seem easy:
>
>   - If each example would be run in a separate R process, R has to
>     be restarted once for every man/*.Rd file.. which easily adds
>     one minute or rather several minutes to the total testing time
>     (and a I think quite a bit of the checking code had to be re-written).
>
>   - Alternatively, all examples are parsed (fail-safe!) first and put together
>     into an R list (or environment), and then each run via tryCatch(.).
>     This is nicer and more efficient conceptually, but needs even
>     more code changes and tends to lose the transparent *-Ex.R -> *-Ex.Rout
>     interface we have now.
>
>  --- end of digression ---]]
>
>     > If this option controls the tests only, it should be more something like
>     > --no-stop-on-test-error.  If in the long run, another option is added to
>     > control the examples, then could be --no-stop-on-example-error.
>
>     > But again, maybe all this is not needed at all. Maybe changing what
>     > happens with the tests would be enough.
>
> I agree: it would be enough, if the change was better than what we currently have
> (in R-devel):  On
>     https://bugs.r-project.org/bugzilla/show_bug.cgi?id=17176,
> Jan mentions that the patch is simple but suffers a deficiency
> (my wording) in that the error *reporting* is not correct:
>
> If you have three  tests/*.R  test scripts producing an error,
> still only the first of these is reported (and counted in the
> final 'Status: ..' line), and the package checker has to
> manually look at the   tests/*.Rout.fail files to see which of
> the checks failed exactly.... and the reporting of the first
> error is *after* running all the tests/*.R scripts which is also
> far from ideal.
>
>
> Consequently, for now, it does need an option there, and it
> should not be default because of the "inconsistent" report
> output.
>
> I don't mind to  add  the extra  "-test" to the option string,
> i.e., change
> from
>       --no-stop-on-error
> to    --no-stop-on-test-error
>
> Others?
>
>
>     > Cheers,
>     > H.
>
>     > --
>     > Herv? Pag?s
>
>     > Program in Computational Biology
>     > Division of Public Health Sciences
>     > Fred Hutchinson Cancer Research Center
>     > 1100 Fairview Ave. N, M1-B514
>     > P.O. Box 19024
>     > Seattle, WA 98109-1024
>
>     > E-mail: hpages at fredhutch.org
>     > Phone:  (206) 667-5791
>     > Fax:    (206) 667-1319
>

-- 
Herv? Pag?s

Program in Computational Biology
Division of Public Health Sciences
Fred Hutchinson Cancer Research Center
1100 Fairview Ave. N, M1-B514
P.O. Box 19024
Seattle, WA 98109-1024

E-mail: hpages at fredhutch.org
Phone:  (206) 667-5791
Fax:    (206) 667-1319



[Rd] Running package tests and not stop on first fail
=?UTF-8?B?SGVydsOpIFBhZ8Oocw==?=