From iuke-tier@ey m@iii@g oii uiow@@edu  Sat Jun  8 02:52:43 2024
From: iuke-tier@ey m@iii@g oii uiow@@edu (iuke-tier@ey m@iii@g oii uiow@@edu)
Date: Fri, 7 Jun 2024 19:52:43 -0500 (CDT)
Subject: [Rd] [External] Re: clarifying and adjusting the C API for R
In-Reply-To: <CALOkxuzKfiGTQ8rzZcM9uy0f+acrdXKeSdxz2qSX2_04+=YAgw@mail.gmail.com>
References: <5a1ccb59-3e90-31eb-132a-35c3a1839bbf@uiowa.edu>
 <CAHiA-ZnQK238zu6yXAxCNBASP6LX+74S8LjvOfmC-9ujiAcU+g@mail.gmail.com>
 <6ca3dc90-b06d-1678-8faf-4c8d382dd8b3@uiowa.edu>
 <CALOkxuzKfiGTQ8rzZcM9uy0f+acrdXKeSdxz2qSX2_04+=YAgw@mail.gmail.com>
Message-ID: <eb46c07e-ba96-9a09-243c-88ef52b6f787@uiowa.edu>

On Sat, 8 Jun 2024, Reed A. Cartwright wrote:

> [You don't often get email from racartwright at gmail.com. Learn why this is important at https://aka.ms/LearnAboutSenderIdentification ]
>
> Would it be reasonable to move the non-API stuff that cannot be hidden
> into header files inside a "details" directory (or some other specific
> naming scheme)?
>
> That's what I use when I need to separate a public API from an internal API.

As do I, as does everyone else. As I wrote originally: " ... for a
variety of reasons that isn't achievable, at least not in the near
term." Can we leave it at that please?

luke

>
> On Fri, Jun 7, 2024 at 7:30?AM luke-tierney--- via R-devel
> <r-devel at r-project.org> wrote:
>>
>> On Fri, 7 Jun 2024, Steven Dirkse wrote:
>>
>>> You don't often get email from sdirkse at gams.com. Learn why this is important
>>> Thanks for sharing this overview of an interesting and much-needed project.
>>> You mention that R exports about 1500 symbols (on platforms supporting
>>> visibility) but this subject isn't mentioned explicitly again in your note,
>>> so I'm wondering how things tie together.  Un-exported symbols cannot be
>>> part of the API - how would people use them in this case?  In a perfect
>>> world the set of exported symbols could define the API or match it exactly,
>>> but I guess that isn't the case at present.  So I conclude that R exports
>>> extra (i.e. non-API) symbols.  Is part of the goal to remove these extra
>>> exports?
>>
>> No. We'll hide what we can, but base packages for one need access to
>> some entry points that should not be in the API, so those have to stay
>> un-hidden.
>>
>> Best,
>>
>> luke
>>
>>>
>>> -Steve
>>>
>>> On Thu, Jun 6, 2024 at 10:47?AM luke-tierney--- via R-devel
>>> <r-devel at r-project.org> wrote:
>>>       This is an update on some current work on the C API for use in R
>>>       extensions.
>>>
>>>       The internal R implementation makes use of tens of thousands of
>>>       C
>>>       entry points. On Linux and Windows, which support visibility
>>>       restrictions, most of these are visible only within the R
>>>       executble or
>>>       shared library. About 1500 are not hidden and are visible to
>>>       dynamically loaded shared libraries, such as ones in packages,
>>>       and to
>>>       embedding applications.
>>>
>>>       There are two main reasons for limiting access to entry points
>>>       in a
>>>       software framework:
>>>
>>>       - Some entry points are very easy to use in ways that corrupt
>>>       internal
>>>          data, leading to segfaults or, worse, incorrect computations
>>>       without
>>>          segfaults.
>>>
>>>       - Some entry point expose internal structure and other
>>>       implementation
>>>          details, which makes it hard to make improvements without
>>>       breaking
>>>          client code that has come to depend on these details.
>>>
>>>       The API of C entry points that can be used in R extensions, both
>>>       for
>>>       packages and embedding, has evolved organically over many years.
>>>       The
>>>       definition for the current release expressed in the Writing R
>>>       Extensions manual (WRE) is roughly:
>>>
>>>            An entry point can be used if (1) it is declared in a
>>>       header file
>>>            in R.home("include"), and (2) if it is documented for use
>>>       in WRE.
>>>
>>>       Ideally, (1) would be necessary and sufficient, but for a
>>>       variety of
>>>       reasons that isn't achievable, at least not in the near term.
>>>       (2) can
>>>       be challenging to determine; in particular, it is not amenable
>>>       to a
>>>       computational answer.
>>>
>>>       An experimental effort is underway to add annotations to the WRE
>>>       Texinfo source to allow (2) to be answered unambiguously. The
>>>       annotations so far mostly reflect my reading or WRE and may be
>>>       revised
>>>       as they are reviewed by others. The annotated document can be
>>>       used for
>>>       programmatically identifying what is currently considered part
>>>       of the C
>>>       API. The result so far is an experimental function
>>>       tools:::funAPI():
>>>
>>>           > head(tools:::funAPI())
>>>                            name                    loc apitype
>>>            1 Rf_AdobeSymbol2utf8 R_ext/GraphicsDevice.h    eapi
>>>            2        alloc3DArray                    WRE     api
>>>            3          allocArray                    WRE     api
>>>            4           allocLang                    WRE     api
>>>            5           allocList                    WRE     api
>>>            6         allocMatrix                    WRE     api
>>>
>>>       The 'apitype' field has three possible levels
>>>
>>>            | api  | stable (ideally) API |
>>>            | eapi | experimental API     |
>>>            | emb  | embedding API        |
>>>
>>>       Entry points in the embedded API would typically only be used in
>>>       applications embedding R or providing new front ends, but might
>>>       be
>>>       reasonable to use in packages that support embedding.
>>>
>>>       The 'loc' field indicates how the entry point is identified as
>>>       part of
>>>       an API: explicit mention in WRE, or declaration in a header file
>>>       identified as fully part of an API.
>>>
>>>       [tools:::funAPI() may not be completely accurate as it relies on
>>>       regular expressions for examining header files considered part
>>>       of the
>>>       API rather than proper parsing. But it seems to be pretty close
>>>       to
>>>       what can be achieved with proper parsing.  Proper parsing would
>>>       add
>>>       dependencies on additional tools, which I would like to avoid
>>>       for
>>>       now. One dependency already present is that a C compiler has to
>>>       be on
>>>       the search path and cc -E has to run the C pre-processor.]
>>>
>>>       Two additional experimental functions are available for
>>>       analyzing
>>>       package compliance: tools:::checkPkgAPI and
>>>       tools:::checkAllPkgsAPI.
>>>       These examine installed packages.
>>>
>>>       [These may produce some false positives on macOS; they may or
>>>       may not
>>>       work on Windows at this point.]
>>>
>>>       Using these tools initially showed around 200 non-API entry
>>>       points
>>>       used across packages on CRAN and BIOC. Ideally this number
>>>       should be
>>>       reduced to zero. This will require a combination of additions to
>>>       the
>>>       API and changes in packages.
>>>
>>>       Some entry points can safely be added to the API. Around 40 have
>>>       already been added to WRE with API annotations; another 40 or so
>>>       can
>>>       probably be added after review.
>>>
>>>       The remainder mostly fall into two groups:
>>>
>>>       - Entry points that should never be used in packages, such as
>>>          SET_OBJECT or SETLENGTH (or any non-API SETXYZ functions for
>>>       that
>>>          matter) that can create inconsistent or corrupt internal
>>>       state.
>>>
>>>       - Entry points that depend on the existence of internal
>>>       structure that
>>>          might be subject to change, such as the existence of promise
>>>       objects
>>>          or internal structure of environments.
>>>
>>>       Many, if not most, of these seem to be used in idioms that can
>>>       either
>>>       be accomplished with existing higher-level functions already in
>>>       the
>>>       API, or by new higher level functions that can be created and
>>>       added. Working through these will take some time and
>>>       coordination
>>>       between R-core and maintainers of affected packages.
>>>
>>>       Once things have gelled a bit more I hope to turn this into a
>>>       blog
>>>       post that will include some examples of moving non-API entry
>>>       point
>>>       uses into compliance.
>>>
>>>       Best,
>>>
>>>       luke
>>>
>>>       --
>>>       Luke Tierney
>>>       Ralph E. Wareham Professor of Mathematical Sciences
>>>       University of Iowa                  Phone:
>>>        319-335-3386
>>>       Department of Statistics and        Fax:
>>>        319-335-3017
>>>           Actuarial Science
>>>       241 Schaeffer Hall                  email:
>>>        luke-tierney at uiowa.edu
>>>       Iowa City, IA 52242                 WWW:
>>>       https://urldefense.com/v3/__http://www.stat.uiowa.edu__;!!IKRxdwAv5BmarQ!foNGcMBk1Ky20Cgz66006bUDTWTxmZhh2ntk8-PLXUqCy2s6xw68UOo-fy7OsIRpHBwgMtfQyBkcYZUZBvMvo18$
>>>
>>>       ______________________________________________
>>>       R-devel at r-project.org mailing list
>>>       https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-devel__;!!IKRxdwAv5BmarQ!foNGcMBk1Ky20Cgz66006bUDTWTxmZhh2ntk8-PLXUqCy2s6xw68UOo-fy7OsIRpHBwgMtfQyBkcYZUZnVX5taE$
>>>
>>>
>>>
>>>
>>>
>>
>> --
>> Luke Tierney
>> Ralph E. Wareham Professor of Mathematical Sciences
>> University of Iowa                  Phone:             319-335-3386
>> Department of Statistics and        Fax:               319-335-3017
>>     Actuarial Science
>> 241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
>> Iowa City, IA 52242                 WWW:  https://urldefense.com/v3/__http://www.stat.uiowa.edu__;!!IKRxdwAv5BmarQ!foNGcMBk1Ky20Cgz66006bUDTWTxmZhh2ntk8-PLXUqCy2s6xw68UOo-fy7OsIRpHBwgMtfQyBkcYZUZBvMvo18$
>> ______________________________________________
>> R-devel at r-project.org mailing list
>> https://urldefense.com/v3/__https://stat.ethz.ch/mailman/listinfo/r-devel__;!!IKRxdwAv5BmarQ!foNGcMBk1Ky20Cgz66006bUDTWTxmZhh2ntk8-PLXUqCy2s6xw68UOo-fy7OsIRpHBwgMtfQyBkcYZUZnVX5taE$
>

-- 
Luke Tierney
Ralph E. Wareham Professor of Mathematical Sciences
University of Iowa                  Phone:             319-335-3386
Department of Statistics and        Fax:               319-335-3017
    Actuarial Science
241 Schaeffer Hall                  email:   luke-tierney at uiowa.edu
Iowa City, IA 52242                 WWW:  http://www.stat.uiowa.edu/


[Rd] [External] Re: clarifying and adjusting the C API for R
iuke-tier@ey m@iii@g oii uiow@@edu