From jasoncbarnhart at msn.com  Wed May 30 02:13:31 2007
From: jasoncbarnhart at msn.com (Jason Barnhart)
Date: Tue, 29 May 2007 17:13:31 -0700
Subject: [Rd] AIX testers needed
References: <BAY116-DAV9C25C47DAB32FBD9519F0CF320@phx.gbl>
	<dc41e1260705191853r5e9fed51j2f9d07840d4ad7f5@mail.gmail.com>
	<BAY116-DAV4CB005F0377A682BFCD72CF370@phx.gbl>
	<dc41e1260705211932u27cc2bf8jc8507b7b6423361c@mail.gmail.com>
Message-ID: <BAY116-DAV17BB9EF6ADE3DC3A521C4CF2E0@phx.gbl>

Ei-ji,

Thank you very much for your reply; your suggestion resolved the
issue.

Follow up (for others):

    1) In addition to setting the LDR_CNTRL environment variable
    at run-time, R can be compiled by adding '-Wl,-
    bmaxdata:0xn0000000' to LD_FLAGS (where 'n' is the number of
    256MB increments of RAM requested) to engage large program
    support (using GCC). This will support up to 2GB of RAM.

    2) Setting LD_FLAGS set to '-Wl,-bmaxdata:0xD0000000/DSA' I
    was able to compile R and use 3.0+ GB of RAM in a 32-bit
    build.  AIX documentation states this will support 3.5 GB of
    RAM.

    3) Under a 64-bit build, R could use all available RAM on the
    server.     One caveat here: requesting more RAM than the
    system has available forces the OS to kill R without warning.
    This could probably be prevented by using the -bmaxdata
    option, but I have not tested this hypothesis.

    4) AIX system documentation for 'Large Program Support' can
    be found     at
    http://publib.boulder.ibm.com/infocenter/pseries/v5r3/index.jsp
    for AIX 5.3 under 'Programming for AIX', then 'General
    Programming Concepts: Writing and Debugging Programs'


----- Original Message ----- 
From: "Ei-ji Nakama" <nakama at ki.rim.or.jp>
To: "Jason Barnhart" <jasoncbarnhart at msn.com>
Cc: <r-devel at stat.math.ethz.ch>
Sent: Monday, May 21, 2007 7:32 PM
Subject: Re: [Rd] AIX testers needed


> Hi.
>
> I was not careful to 32bit very much.
> You can expand memory for data to 2G by LDR_CNTRL.
> It is as follows.
>
> LDR_CNTRL="MAXDATA=0x10000000" 1   256 MB
> LDR_CNTRL="MAXDATA=0x20000000" 2   512 MB
> LDR_CNTRL="MAXDATA=0x30000000" 3   768 MB
> LDR_CNTRL="MAXDATA=0x40000000" 4 1024 MB
> LDR_CNTRL="MAXDATA=0x50000000" 5 1280 MB
> LDR_CNTRL="MAXDATA=0x60000000" 6 1536 MB
> LDR_CNTRL="MAXDATA=0x70000000" 7 1792 MB
> LDR_CNTRL="MAXDATA=0x80000000"  8       2048 MB
>
> try.
> $ export  LDR_CNTRL="MAXDATA=0x80000000"
> $ R -q
>> xx <- matrix(rep(1e+10,1e7),nrow=1e4,ncol=1e3)
>> tmp.df <- as.data.frame(cbind(xx,xx,xx))
>>
>
>
>
> 2007/5/22, Jason Barnhart <jasoncbarnhart at msn.com>:
>> Thank you for responding.
>>
>> I should have added -a on my ulimit command.  Here are its results;
>> which I believe are not the limiting factor.
>>
>> %/ > ulimit -a
>> core file size        (blocks, -c) 1048575
>> data seg size         (kbytes, -d) unlimited
>> file size             (blocks, -f) unlimited
>> max memory size       (kbytes, -m) 32768
>> open files                    (-n) 2000
>> pipe size          (512 bytes, -p) 64
>> stack size            (kbytes, -s) hard
>> cpu time             (seconds, -t) unlimited
>> max user processes            (-u) 128
>> virtual memory        (kbytes, -v) unlimited
>>
>> Here are my gcc-v results.  They are very similiar.
>>
>> %/ > gcc -v
>> Using built-in specs.
>> Target: powerpc-ibm-aix5.3.0.0
>> Configured with: ../gcc-4.1.2/configure
>>     --prefix=/opt/sas/msd/home/barnharj/usr/local
>>     --program-suffix=-4.1.2
>>     --enable-threads=aix
>>     --enable-languages=c,c++,fortran
>>     --disable-nls
>>     --with-mpfr=/opt/sas/msd/home/barnharj/usr/local
>>     --with-gmp=/opt/sas/msd/home/barnharj/usr/local
>> Thread model: aix
>> gcc version 4.1.2
>>
>> My results for gcc -print-multi-lib are indentical to yours.
>>
>> It should be noted that I did not attempt a 64 bit build.  My 
>> system
>> is not configured properly for that just yet.
>>
>> Via private correspondence someone also suggested the following
>> system("ulimit -d unlimited") during an R session.  That failed as 
>> did
>> issuing the command before launching R.
>>
>> I'll keep investigating.
>>
>> My call to configure is listed below.
>>
>> /configure --prefix=$HOME/usr/local --program-suffix=rc --with-readline=no
>>  --with-x=no --enable-memory-profiling
>>
>>
>> ----- Original Message -----
>> From: "Ei-ji Nakama" <nakama at ki.rim.or.jp>
>> To: "Jason Barnhart" <jasoncbarnhart at msn.com>
>> Cc: <r-devel at stat.math.ethz.ch>
>> Sent: Saturday, May 19, 2007 6:53 PM
>> Subject: Re: [Rd] AIX testers needed
>>
>>
>> > Hi.
>> >
>> > system("ulimit") of AIX gives back file block size.
>> > A limit of memory is `ulimit -m'.
>> >
>> > I made gcc-4.1.2 and made R-2.5.0+patched, but the problem did 
>> > not
>> > happen.
>> >
>> > $ gcc-4.1 -v
>> > Using built-in specs.
>> > Target: powerpc-ibm-aix5.2.0.0
>> > Configured with: ../configure --with-as=/usr/bin/as
>> > --with-ld=/usr/bin/ld --disable-nls --prefix=/usr/local1/gcc-4.1.2
>> > --enable-threads --host=powerpc-ibm-aix5.2.0.0 --program-suffix=-4.1
>> > --with-gmp=/usr/local1 --with-mpfr=/usr/local1
>> > --enable-languages=c,c++,f95
>> > Thread model: aix
>> > gcc version 4.1.2
>> >
>> > $ gcc-4.1 -print-multi-lib
>> > .;
>> > pthread;@pthread
>> > ppc64;@maix64
>> > pthread/ppc64;@pthread at maix64
>> >
>> > $ export OBJECT_MODE=64
>> > $ ./configure CC="gcc-4.1 -maix64" \
>> >                   F77="gfortran-4.1 -maix64" \
>> >                   CXX="g++-4.1 -maix64" \
>> >                   --enable-BLAS-shlib --without-iconv
>> >
>> >
>> > 2007/5/19, Jason Barnhart <jasoncbarnhart at msn.com>:
>> >> Per the request to test the latest tarball referenced below, I 
>> >> have
>> >> built R on AIX 5.3. There is a memory issue, please see 3) 
>> >> below.
>> >>
>> >>     1) Build with --enable-BLAS-shlib option.  Builds and
>> >>     passes "make check".
>> >>
>> >>     2) GNU libiconv was installed; R configured *without*
>> >> the --without-
>> >>     iconv option. Builds and passes "make check."
>> >>
>> >>     3) Memory issue:
>> >>         a) Although the server possesses 8GB of RAM and
>> >>         system("ulimit") returns "unlimited" as its value, R 
>> >> does
>> >> not
>> >>         "have enough memory."
>> >>
>> >>         b) This code works on R-2.4.0 on WinXP with 2GB of RAM, 
>> >> but
>> >>         fails on the AIX build.  Example code:
>> >>
>> >>         > xx <- matrix(rep(1e+10,1e7),nrow=1e4,ncol=1e3)
>> >>         > tmp.df <- as.data.frame(cbind(xx,xx,xx))
>> >>         Error: cannot allocate vector of size 228.9 Mb
>> >>         > gc()
>> >>                    used (Mb) gc trigger  (Mb) max used  (Mb)
>> >>         Ncells   233035  6.3     467875  12.5   350000   9.4
>> >>         Vcells 10104141 77.1   31854441 243.1 30104289 229.7
>> >>
>> >>     4) Used gcc-4.1.2; I have not tried the native compiler.
>> >>
>> >> Questions:
>> >>
>> >>     1) Are there suggestions on how to diagnose and resolve the
>> >> memory
>> >>     issue?  I'm still building my tool chain and I am in the 
>> >> midst
>> >> of
>> >>     installing gdb.
>> >>
>> >>     2) Regarding the build process itself, is there more
>> >> documentation
>> >>     or results that I should forward?
>> >>
>> >> Thanks,
>> >> -jason
>> >>
>> >>
>> >> > version               _
>> >> platform       powerpc-ibm-aix5.3.0.0
>> >> arch           powerpc
>> >> os             aix5.3.0.0
>> >> system         powerpc, aix5.3.0.0
>> >> status         RC
>> >> major          2
>> >> minor          5.0
>> >> year           2007
>> >> month          04
>> >> day            22
>> >> svn rev        41276
>> >> language       R
>> >> version.string R version 2.5.0 RC (2007-04-22 r41276)
>> >>
>> >>
>> >> ----- Original Message -----
>> >> From: "Prof Brian Ripley" <ripley at stats.ox.ac.uk>
>> >> To: <R-devel at r-project.org>
>> >> Cc: "Sheth, Jagat K" <jagat.k.sheth at citigroup.com>; "kamil
>> >> Marcinkowski" <kamil at ualberta.ca>; "Tiong Lim"
>> >> <t.lim at auckland.ac.nz>;
>> >> "Eric Harley" <ericctharley at gmail.com>; "Matthew Beason"
>> >> <mbeason at harrahs.com>; <watplatt at us.ibm.com>
>> >> Sent: Wednesday, March 28, 2007 11:27 PM
>> >> Subject: [Rd] AIX testers needed
>> >>
>> >>
>> >> > With much thanks to Ei-ji Nakama, R 2.5.0 alpha supports 
>> >> > building
>> >> > on
>> >> > AIX
>> >> > (at least AIX 5.2 on one system).
>> >> >
>> >> > Would anyone able to test this please get the latest tarball 
>> >> > from
>> >> >
>> >> > http://cran.r-project.org/src/base-prerelease/R-latest.tar.gz
>> >> >
>> >> > and try installing (after reading the AIX notes in 
>> >> > R-admin.html
>> >> > section
>> >> > C.9).
>> >> >
>> >> > In particular it would be very helpful to know if
>> >> >
>> >> > 1) --enable-BLAS-shlib works (it is the default everywhere 
>> >> > except
>> >> > AIX and
>> >> > Darwin)
>> >> >
>> >> > 2) if people succeed in installing GNU libiconv and building
>> >> > without
>> >> > needing --without-iconv.
>> >> >
>> >> > I am Cc:ing all the people I tracked down who reported 
>> >> > attempted
>> >> > AIX
>> >> > installations in 2006 in the hope that they may still be
>> >> > interested.
>> >> >
>> >> > Please report success and any hints or problems in the R-devel
>> >> > list.
>> >> >
>> >> > --
>> >> > Brian D. Ripley,                  ripley at stats.ox.ac.uk
>> >> > Professor of Applied Statistics,
>> >> > http://www.stats.ox.ac.uk/~ripley/
>> >> > University of Oxford,             Tel:  +44 1865 272861 (self)
>> >> > 1 South Parks Road,                     +44 1865 272866 (PA)
>> >> > Oxford OX1 3TG, UK                Fax:  +44 1865 272595
>> >> >
>> >> > ______________________________________________
>> >> > R-devel at r-project.org mailing list
>> >> > https://stat.ethz.ch/mailman/listinfo/r-devel
>> >> >
>> >>
>> >> ______________________________________________
>> >> R-devel at r-project.org mailing list
>> >> https://stat.ethz.ch/mailman/listinfo/r-devel
>> >>
>> >>
>> >>
>> >
>> >
>> > --
>> > EI-JI Nakama  <nakama at ki.rim.or.jp>
>> > "\u4e2d\u9593\u6804\u6cbb"  <nakama at ki.rim.or.jp>
>> >
>>
>>
>>
>>
>
>
> -- 
> EI-JI Nakama  <nakama at ki.rim.or.jp>
> "\u4e2d\u9593\u6804\u6cbb"  <nakama at ki.rim.or.jp>
>



[Rd] AIX testers needed
Jason Barnhart