From S.N.Maijers at student.ru.nl  Tue Jun  5 15:03:43 2012
From: S.N.Maijers at student.ru.nl (Sander Maijers)
Date: Tue, 5 Jun 2012 15:03:43 +0200
Subject: [Rd] Memory leak during instantiation of a reference class?
Message-ID: <20120605150343.7b428ac8@student.ru.nl>

Hi,

I have a problem with a large data set wrapped in a reference class. I do that to access the data by reference from within various functions in a script.

# The class definition:

	setRefClass("data",
          		   fields = list(h5_df                = "list",
                           		     training_data    = "array",
                           		     true_class         = "vector"));

# The class instantiation:

          data_1 <- new("data",
                             h5_df               = as.list(h5_df),
                             training_data   = training_data,
                             true_class        = true_class);


The objects assigned to the "data" fields during instantiation:

Browse[1]> str(training_data)
 num [1:501, 1:62, 1:579] -10.4 -11.3 -12.1 -12.8 -13.3 ...
 - attr(*, "dimnames")=List of 3
  ..$ : NULL
  ..$ : NULL
  ..$ : NULL


Browse[1]> str(true_class)
 num [1:1002] 1 1 1 1 1 1 1 1 1 1 ...


Browse[1]> str(h5_df)
Dotted pair list of 11
 $ avg.txt.lp    :Garbage collection 135 = 44+18+73 (level 2) ... 
49.4 Mbytes of cons cells used (63%)
2840.0 Mbytes of vectors used (30%)
 num [1:501, 1:62, 1:607] -10.4 -11.3 -12.1 -12.8 -13.3 ...
 $ avg.tls.txt.lp: num [1:501, 1:62, 1:281] -10.4 -11.3 -12.1 -12.8 -13.3 ...
 $ avg.tls.pic.lp: num [1:501, 1:62, 1:305] -2.81 -4.95 -6.98 -8.79 -10.32 ...
 $ avg.tls.lp    : num [1:501, 1:62, 1:884] -10.4 -11.3 -12.1 -12.8 -13.3 ...
 $ avg.tls.aud.lp: num [1:501, 1:62, 1:298] -13.9 -11.36 -8.91 -6.61 -4.55 ...
 $ avg.pic.lp    : num [1:501, 1:62, 1:653] -2.81 -4.95 -6.98 -8.79 -10.32 ...
 $ avg.aud.lp    : num [1:501, 1:62, 1:662] -13.9 -11.36 -8.91 -6.61 -4.55 ...
 $ avg.anm.txt.lp: num [1:501, 1:62, 1:273] -3.37 -2.96 -2.59 -2.27 -2 ...
 $ avg.anm.pic.lp: num [1:501, 1:62, 1:285] 11.7 12.3 12.8 13.2 13.5 ...
 $ avg.anm.lp    : num [1:501, 1:62, 1:855] -3.37 -2.96 -2.59 -2.27 -2 ...
 $ avg.anm.aud.lp:Garbage collection 136 = 44+18+74 (level 2) ... 
49.4 Mbytes of cons cells used (63%)
2980.8 Mbytes of vectors used (32%)
 num [1:501, 1:62, 1:297] -13.57 -11.96 -10.36 -8.77 -7.2 ...


The procedure that gives rise to the problem:

> rm(list=ls())
> gc()
         used (Mb) gc trigger   (Mb)  max used   (Mb)
Ncells 921231 49.2    1476915   78.9   1166886   62.4
Vcells 792872  6.1  565291490 4312.9 690075184 5264.9
> load_data()
Processing object: avg_anm_aud_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332028
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  297 / 501  62  297 
 501  62  297  Tidying name avg_anm_aud_lp .. to avg.anm.aud.lp 
 ...Finished dataset 
Processing object: avg_anm_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332031
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  855 / 501  62  855 
 501  62  855  Tidying name avg_anm_lp .. to avg.anm.lp 
 ...Finished dataset 
Processing object: avg_anm_pic_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332034
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  285 / 501  62  285 
 501  62  285  Tidying name avg_anm_pic_lp .. to avg.anm.pic.lp 
 ...Finished dataset 
Processing object: avg_anm_txt_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332037
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  273 / 501  62  273 
 501  62  273  Tidying name avg_anm_txt_lp .. to avg.anm.txt.lp 
 ...Finished dataset 
Processing object: avg_aud_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332040
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  662 / 501  62  662 
 501  62  662  Tidying name avg_aud_lp .. to avg.aud.lp 
 ...Finished dataset 
Processing object: avg_pic_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332043
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  653 / 501  62  653 
 501  62  653  Tidying name avg_pic_lp .. to avg.pic.lp 
 ...Finished dataset 
Processing object: avg_tls_aud_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332046
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  298 / 501  62  298 
 501  62  298  Tidying name avg_tls_aud_lp .. to avg.tls.aud.lp 
 ...Finished dataset 
Processing object: avg_tls_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332049
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  884 / 501  62  884 
 501  62  884  Tidying name avg_tls_lp .. to avg.tls.lp 
 ...Finished dataset 
Processing object: avg_tls_pic_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332052
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  305 / 501  62  305 
 501  62  305  Tidying name avg_tls_pic_lp .. to avg.tls.pic.lp 
 ...Finished dataset 
Processing object: avg_tls_txt_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332055
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  281 / 501  62  281 
 501  62  281  Tidying name avg_tls_txt_lp .. to avg.tls.txt.lp 
 ...Finished dataset 
Processing object: avg_txt_lp ...... its a dataset...Dataset has ID83886080
Dataset has tid 50332058
Dataset has space id 67108866
Dataset has rank 3
Dataset has dims/maxdims: 501  62  607 / 501  62  607 
 501  62  607  Tidying name avg_txt_lp .. to avg.txt.lp 
 ...Finished dataset 
Garbage collection 118 = 44+17+57 (level 0) ... 
49.4 Mbytes of cons cells used (63%)
2277.9 Mbytes of vectors used (53%)

Called from: load_data()
Browse[1]> str(data_1)
Garbage collection 119 = 44+18+57 (level 1) ... 
49.4 Mbytes of cons cells used (63%)
4119.7 Mbytes of vectors used (96%)
Reference class 'data' [package ".GlobalEnv"] with 3 fields
Garbage collection 120 = 44+18+58 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
4256.9 Mbytes of vectors used (92%)
Garbage collection 121 = 44+18+59 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
4539.6 Mbytes of vectors used (91%)
Garbage collection 122 = 44+18+60 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
4974.5 Mbytes of vectors used (92%)
Garbage collection 123 = 44+18+61 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
5263.6 Mbytes of vectors used (92%)
Garbage collection 124 = 44+18+62 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
5673.9 Mbytes of vectors used (93%)
Garbage collection 125 = 44+18+63 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
5956.6 Mbytes of vectors used (92%)
Garbage collection 126 = 44+18+64 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
6391.4 Mbytes of vectors used (93%)
Garbage collection 127 = 44+18+65 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
6680.6 Mbytes of vectors used (92%)
Garbage collection 128 = 44+18+66 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
5673.9 Mbytes of vectors used (75%)
Garbage collection 129 = 44+18+67 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
7583.0 Mbytes of vectors used (94%)
Garbage collection 130 = 44+18+68 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
8030.0 Mbytes of vectors used (94%)
Garbage collection 131 = 44+18+69 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
8370.5 Mbytes of vectors used (94%)
Garbage collection 132 = 44+18+70 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
7373.5 Mbytes of vectors used (79%)
 $ h5_df        :Garbage collection 133 = 44+18+71 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
9070.6 Mbytes of vectors used (92%)
Error: cannot allocate vector of size 154.8 Mb
Garbage collection 134 = 44+18+72 (level 2) ... 
49.3 Mbytes of cons cells used (62%)
9070.6 Mbytes of vectors used (92%)
Error during wrapup: cannot allocate vector of size 143.8 Mb


The computer system that runs this code is has a 64-bit CPU and 12 GiB RAM.
It seems that there is a memory leak somewhere. Any suggestions on how to solve this problem?

Regards,
Sander Maijers



[Rd] Memory leak during instantiation of a reference class?
Sander Maijers