From pgilbert at bank-banque-canada.ca  Mon Mar 31 21:09:51 2008
From: pgilbert at bank-banque-canada.ca (Paul Gilbert)
Date: Mon, 31 Mar 2008 15:09:51 -0400
Subject: [Rd] tests Rin and Rout
In-Reply-To: <18416.44977.144597.422309@stat.math.ethz.ch>
References: <47F014DB.6040000@u-paris10.fr>	<18416.40443.244387.473214@stat.math.ethz.ch>	<20080331105833.hg2cj67g6c84c8k0@icare.u-paris10.fr>
	<18416.44977.144597.422309@stat.math.ethz.ch>
Message-ID: <47F136FF.9060407@bank-banque-canada.ca>

Martin Maechler wrote:
>>>>>> "ChrG" == cgenolin  <cgenolin at u-paris10.fr>
>>>>>>     on Mon, 31 Mar 2008 10:58:33 +0200 writes:
>>>>>>             
>
>     ChrG> Martin Maechler <maechler at stat.math.ethz.ch> a ??crit?
>     ChrG> :
>     >>>>>>> "CG" == Christophe Genolini <cgenolin at u-paris10.fr>
>     >>>>>>> on Mon, 31 Mar 2008 00:31:55 +0200 writes:
>     >> 
>     >> >>
>     >> >> Generally I find it's good to look at examples that
>     >> work.  >> For examples of packages using tests, look at
>     >> source >> packages on CRAN.  Run the tests on them (using
>     >> R CMD >> check), and see what gets produced.
>     >> >>
>     CG> Do you have the name of a package that use it ? I try
>     CG> the 10 first package, and 10 other at random, but none
>     CG> of them use tests...
>     >> 
>     >> hmm, I see 219 out 1378 CRAN packages having a 'tests'
>     >> subdirectory, so it seems you have been a bit
>     >> unlucky. ;-)
>   
Hmmm. Well, since I have several of the 219, and I suspect a handful of 
other developers also individually have several,  it looks a bit like 
most developers are not using them. Perhaps there needs to be more 
encouragement.

More specifically on the question:
- You might look at the package numDeriv, which is fairly small and does 
testing in the way I have more recently found to be most convenient. In 
some of my older packages my mechanism was a bit awkward until the 
package settled down.

-I now find it is generally simpler to have several small test files 
rather than one big one.

-If there is a logical order for running the tests then this can be 
arranged using a naming convention. I think tests are run in lexical 
order by filename, which can be different on different systems but, it 
can be made to work by putting numbers n the beginning of the filenames.

-If one of the functions that you run actually fails to exit properly, 
then the test will fail.

-If numerical results rather than printed output is what you want to 
test, then .R files are probably preferred.  In the test file you should 
check a value and stop() if it is not good, something like
   fuzz <- 1e-14
   if (fuzz < abs(sq(2) -  4)) stop("sq(2) test failed.")

-If you just check the printed value using the Rin / Rout mechanism, the 
rounding in print() will hide errors that you should catch. If you try 
to fix this by printing to higher precision, the tests will fail on 
different architectures and with different compilers.

- For some calculations, fuzz may need to be somewhat larger than you 
might expect, in order to accommodate different architectures and 
compilers. (This can be an indication that the algorithm or the specific 
problem is a bit unstable).

- It is nice to have test values (4 in the above example) that are known 
theoretically, but I usually have to use a value that came from an 
initial run of the program (so I am just testing if the program results 
have changed). In this case it is useful to print the test value to a 
high precision, say digits=18, just before you do the test, so you know 
what the new/wrong result is when the test fails. This is also useful 
for tracing down how big the fuzz needs to be to accommodate different 
architectures.

- If you use random numbers then you need to set the random number 
generator to a known state in each test file. The package setRNG may be 
helpful for this. (The logical here is that you are testing the code for 
changes and errors, not running a random experiment.)

-If you use random numbers then you may want to run the program 
random.number.test() in setRNG as one of your first tests. This function 
checks that the RNG and normal generator have not been changed.  (They 
have not changed in a long time, but I am a bit sensitive to this point, 
having wasted a lot of time after a small change between two versions of 
Splus.)

-If  you use the Rin / Rout mechanism, be careful about what you print, 
otherwise you may have a high maintenance problem because the printed 
output can change a bit for reasons you may not anticipate. (It may be 
possible  to make this robust in ways I don't understand - so if someone 
can add to this I would appreciate hearing. I'm especially curious how 
language support works with this.)

-You can test error checking and warnings in your functions using try() 
in the test files, but you might consider leaving this until you have 
mastered some of the other pieces.

- Graphics output can be put in files (I think in Rplot.ps by default) 
but I don't know of any way to automatically check the graphs to see if 
they are good.  As far as I know, you have to actually look at them. If 
anyone knows of an automatic mechanism I would really like to hear about it.

-Each test file needs to attach any needed packages, including the one 
being tested.

-Be a bit careful about what gets found in your default search path. 
This can be especially important if you are checking the tests files 
outside of the  R CMD check environment, and after you have installed 
your package and are testing a new revised version.

-If you are testing multiple packages that are inter-related, then you 
may need to be careful about some additional things, like not picking up 
one of the test packages from the site library rather than your testing 
library.

HTH
Paul Gilbert
>     ChrG> Do you imply that I say I try and I did not ?  
>
> not at all!  I am sorry if you (or anyone) got this impression!
>
>     ChrG> Well, lets check that :
>
>     ChrG> P_unluck_1 <- 1-(219/1379) 
>     ChrG> (P_unluck_10 <- P_unluck_1^10) [1] 0.1773933
>
>     ChrG> So P>0.05, once can not say that I am not a liar.
>     ChrG> Further investigation are needed :-)
>
> :-)
>
>     >> Isn't all this is explained nicely in the "Writing R
>     >> Extensions" Manual?
>
>     ChrG> NO ! There is 8 lines on it, and the structure of the
>     ChrG> Rin and Rout is not given. The writing R Ext. say "the
>     ChrG> Test code can be in .R or in .Rin" but I do not know
>     ChrG> what "test code" is suppose to be. I try simple things
>     ChrG> like some function with argument, but it does not
>     ChrG> works. I try more complicated things, I did not manage
>     ChrG> to find the right things to do.
>
> Ok; so we should expand the part in the manual.
>
> First:  '.Rin' are typically only used very rarely; so, normally
> the input file is 'foo.R' and hence (well ``hence'') must be
> valid R code, typically starting by 
> library(<mypackage>).
>
>     ChrG> It also say ".Rin file containing code which in turn
>     ChrG> in .R", but there is also no information about the
>     ChrG> what the .Rin should be. May be it is obvious for some
>     ChrG> people, but not for me.
>
>     >> Or are there sections we should expand ?
>
>     ChrG> I do not know, if there is only few people using it,
>     ChrG> may be it is not worth. I guess you have so many other
>     ChrG> thinks to do...  On the other hand, if some tool are
>     ChrG> to difficult to understand, it is normal that people
>     ChrG> do not use them... I don't know.
>
>     ChrG> I will work on package example that Pr Ripley point to
>     ChrG> me. Then when I will have understand the way it works,
>     ChrG> would you like me to add little paragraph in "Writing
>     ChrG> R extention" ? I mean, not an paragraph by expert but
>     ChrG> a paragraph by a regular medium level user for some
>     ChrG> other regular medium level user.
>
> That maybe very useful, thank you for the offer!
> Note that the manual source is in
>   https://svn.r-project.org/R/trunk/doc/manual/R-exts.texi
>
>     >> For answering your subsequent questions, you should
>     >> probably both look at an example package *and* read the
>     >> 'Writing R Extensions' manual a bit more "closely".
>
>     ChrG> Well, I try to not disturb people for little obvious
>     ChrG> thinks. I can tell you I did work hard before asking,
>     ChrG> specialy before asking on r-devel...
>
> Okay, I apologize for having misjudged your efforts.
>
> Martin
>
> ______________________________________________
> R-devel at r-project.org mailing list
> https://stat.ethz.ch/mailman/listinfo/r-devel
>   
====================================================================================

La version fran?aise suit le texte anglais.

------------------------------------------------------------------------------------

This email may contain privileged and/or confidential in...{{dropped:26}}



[Rd] tests Rin and Rout
Paul Gilbert