From spencer.graves at prodsyse.com  Tue Dec 20 18:14:58 2016
From: spencer.graves at prodsyse.com (Spencer Graves)
Date: Tue, 20 Dec 2016 11:14:58 -0600
Subject: [Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c
In-Reply-To: <22617.25258.765788.878896@max.nulle.part>
References: <CAAVP=akpmJ2gDdDXNPTOh-fFNAy-B52v0QRp6FaNRfhW9F_ZKw@mail.gmail.com>
	<CAFDcVCTOtaX0cxAZo8hh3mZdp75Xx7UZ4Kry5C+EUxZoqhA8Dw@mail.gmail.com>
	<CAAVP=an9yD_E22KF3q-Bh-XxVzyaEYaic-iTpzCZVxv25G2QnA@mail.gmail.com>
	<22617.24288.140314.653221@stat.math.ethz.ch>
	<22617.25258.765788.878896@max.nulle.part>
Message-ID: <d140e7e2-4a76-31c8-1940-17c1ad17c4ad@prodsyse.com>

Hi, Dirk:


On 12/20/2016 10:56 AM, Dirk Eddelbuettel wrote:
> On 20 December 2016 at 17:40, Martin Maechler wrote:
> | >>>>> Steve Bronder <sbronder at stevebronder.com>
> | >>>>>     on Tue, 20 Dec 2016 01:34:31 -0500 writes:
> |
> |     > Thanks Henrik this is very helpful! I will try this out on our tests and
> |     > see if gcDLLs() has a positive effect.
> |
> |     > mlr currently has tests broken down by learner type such as classification,
> |     > regression, forecasting, clustering, etc.. There are 83 classifiers alone
> |     > so even when loading and unloading across learner types we can still hit
> |     > the MAX_NUM_DLLS error, meaning we'll have to break them down further (or
> |     > maybe we can be clever with gcDLLs()?). I'm CC'ing Lars Kotthoff and Bernd
> |     > Bischl to make sure I am representing the issue well.
> |
> | This came up *here* in May 2015
> | and then May 2016 ... did you not find it when googling.
> |
> | Hint:  Use
> |        site:stat.ethz.ch MAX_NUM_DLLS
> | as search string in Google, so it will basically only search the
> | R mailing list archives
> |
> | Here's the start of that thread :
> |
> |   https://stat.ethz.ch/pipermail/r-devel/2016-May/072637.html
> |
> | There was not a clear conclusion back then, notably as
> | Prof Brian Ripley noted that 100 had already been an increase
> | and that a large number of loaded DLLs decreases look up speed.
> |
> | OTOH (I think others have noted that) a large number of DLLs
> | only penalizes those who *do* load many, and we should probably
> | increase it.
> |
> | Your use case of "hyper packages" which load many others
> | simultaneously is somewhat convincing to me... in so far as the
> | general feeling is that memory should be cheap and limits should
> | not be low.
> |
> | (In spite of Brian Ripleys good reasons against it, I'd still
> |  aim for a *dynamic*, i.e. automatically increased list here).
>
> Yes.  Start with 10 or 20, add 10 as needed.  Still fast in the 'small N'
> case and no longer a road block for the 'big N' case required by mlr et al.
>
> As a C++ programmer, I am now going to hug my std::vector and quietly retreat.


May I humbly request a translation of "std::vector" for people like me 
who are not familiar with C++?


I got the following:


 > install.packages('std')
Warning in install.packages :
   package ?std? is not available (for R version 3.3.2)


       Thanks,
       Spencer Graves
>
> Dirk
>
>   
> | Martin Maechler
> |
> |     > Regards,
> |
> |     > Steve Bronder
> |     > Website: stevebronder.com
> |     > Phone: 412-719-1282
> |     > Email: sbronder at stevebronder.com
> |
> |
> |     > On Tue, Dec 20, 2016 at 1:04 AM, Henrik Bengtsson <
> |     > henrik.bengtsson at gmail.com> wrote:
> |
> |     >> On reason for hitting the MAX_NUM_DLLS (= 100) limit is because some
> |     >> packages don't unload their DLLs when they being unloaded themselves.
> |     >> In other words, there may be left-over DLLs just sitting there doing
> |     >> nothing but occupying space.  You can remove these, using:
> |     >>
> |     >> R.utils::gcDLLs()
> |     >>
> |     >> Maybe that will help you get through your tests (as long as you're
> |     >> unloading packages).  gcDLLs() will look at base::getLoadedDLLs() and
> |     >> its content and compare to loadedNamespaces() and unregister any
> |     >> "stray" DLLs that remain after corresponding packages have been
> |     >> unloaded.
> |     >>
> |     >> I think it would be useful if R CMD check would also check that DLLs
> |     >> are unregistered when a package is unloaded
> |     >> (https://github.com/HenrikBengtsson/Wishlist-for-R/issues/29), but of
> |     >> course, someone needs to write the code / a patch for this to happen.
> |     >>
> |     >> /Henrik
> |     >>
> |     >> On Mon, Dec 19, 2016 at 6:01 PM, Steve Bronder
> |     >> <sbronder at stevebronder.com> wrote:
> |     >> > This is a request to increase MAX_NUM_DLLS in Rdynload.c in from 100 to
> |     >> 500.
> |     >> >
> |     >> > On line 131 of Rdynload.c, changing
> |     >> >
> |     >> > #define MAX_NUM_DLLS 100
> |     >> >
> |     >> >  to
> |     >> >
> |     >> > #define MAX_NUM_DLLS 500
> |     >> >
> |     >> >
> |     >> > In development of the mlr package, there have been several episodes in
> |     >> the
> |     >> > past where we have had to break up unit tests because of the "maximum
> |     >> > number of DLLs reached" error. This error has been an inconvenience that
> |     >> is
> |     >> > going to keep happening as the package continues to grow. Is there more
> |     >> > than meets the eye with this error or would everything be okay if the
> |     >> above
> |     >> > line changes? Would that have a larger effect in other parts of R?
> |     >> >
> |     >> > As R grows, we are likely to see more 'meta-packages' such as the
> |     >> > Hadley-verse, caret, mlr, etc. need an increasing amount of DLLs loaded
> |     >> at
> |     >> > any point in time to conduct effective unit tests. If  MAX_NUM_DLLS is
> |     >> set
> |     >> > to 100 for a very particular reason than I apologize, but if it is
> |     >> possible
> |     >> > to increase MAX_NUM_DLLS it would at least make the testing at mlr much
> |     >> > easier.
> |     >> >
> |     >> > I understand you are all very busy and thank you for your time.
> |     >> >
> |     >> >
> |     >> > Regards,
> |     >> >
> |     >> > Steve Bronder
> |     >> > Website: stevebronder.com
> |     >> > Phone: 412-719-1282
> |     >> > Email: sbronder at stevebronder.com
> |     >> >
> |     >> >         [[alternative HTML version deleted]]
> |     >> >
> |     >> > ______________________________________________
> |     >> > R-devel at r-project.org mailing list
> |     >> > https://stat.ethz.ch/mailman/listinfo/r-devel
> |     >>
> |
> |     > [[alternative HTML version deleted]]
> |
> |     > ______________________________________________
> |     > R-devel at r-project.org mailing list
> |     > https://stat.ethz.ch/mailman/listinfo/r-devel
> |
> | ______________________________________________
> | R-devel at r-project.org mailing list
> | https://stat.ethz.ch/mailman/listinfo/r-devel
>



[Rd] Request: Increasing MAX_NUM_DLLS in Rdynload.c
Spencer Graves