<jupyter_start><jupyter_text>Setup enviroment Installs packages and gets data so that it is easy to spin up notebooks at places like colab or sagemaker<jupyter_code># tiledb deps
!pip install tiledb<jupyter_output>Collecting tiledb
  Downloading tiledb-0.10.4-cp37-cp37m-manylinux2010_x86_64.whl (17.5 MB)
[K     |████████████████████████████████| 17.5 MB 13 kB/s 
[?25hRequirement already satisfied: numpy>=1.16.5 in /usr/local/lib/python3.7/dist-packages (from tiledb) (1.19.5)
Installing collected packages: tiledb
Successfully installed tiledb-0.10.4<jupyter_text>Set user options<jupyter_code># Setup how much synthetic data is generated
import numpy as np
n_seconds = 3600 # One hour
sampling_rate = 10 # in Hz A.K.A. SPS
sample_tolerence = np.timedelta64(1, 'ms') # allow this much jitter without creating a data section
start_dt = np.datetime64('2000-01-01')

attempted_time_dim_array = "attempted_time_dim_array"
time_attr_array = "time_attr_array"
sparse_array_time_dim_array = "sparse_array_time_dim_array"<jupyter_output><empty_output><jupyter_text>Create synthetic data for testing<jupyter_code>import numpy as np
# Create and write data
def create_tone(freq, time, amplitude=1.0, phase = 0.0):
  return amplitude*np.sin(2*np.pi*freq*time + phase)

t = np.arange(0, n_seconds, 1.0 / sampling_rate)

def create_tone_int16(freq, time, amplitude=1.0, phase = 0.0):
  return np.int16(32768*amplitude*np.sin(2*np.pi*freq*time + phase))

data = {
    't' : start_dt + t * np.timedelta64(1000, 'ms'), # TODO do I need to us nanosecs here?
    'x' : create_tone_int16(freq=1.01, time=t, amplitude=0.1),
}

print(f"For {n_seconds} esconds @ {sampling_rate} SPS created: {data['x'].shape[0]} samples")<jupyter_output>For 3600 esconds @ 10 SPS created: 36000 samples<jupyter_text>Naive approch for dimension<jupyter_code>import tiledb

# Always remove and recreate local arrays
if tiledb.array_exists(attempted_time_dim_array):
  tiledb.remove(attempted_time_dim_array)
  print(f"Removed existing {attempted_time_dim_array}")

# Create the dimensions
d1 = tiledb.Dim(name="time", 
                domain=(start_dt, start_dt + np.timedelta64(n_seconds, 's')),
                tile = n_seconds*sampling_rate,
                dtype=np.datetime64('', 'ms').dtype) # defines resolution I think

# Create a domain using the dimension
dom1 = tiledb.Domain(d1)

# Create an attribute
t = tiledb.Attr(name="t", dtype=np.datetime64('', 'ms').dtype) # Have to specify resolution
x = tiledb.Attr(name="x", dtype=np.int16)



# datetime in tileDB ref: https://tiledb-inc-tiledb.readthedocs-hosted.com/en/1.6.3/tutorials/datetimes.html

# Create the array schema, setting `sparse=False` to indicate a dense array
schema1 = tiledb.ArraySchema(domain=dom1, sparse=False, attrs=[t, x])

tiledb.Array.create(attempted_time_dim_array, schema1)
# Write data to tiles
try:
  write_info_1 = {}
  with tiledb.open(attempted_time_dim_array, 'w') as A:
      A[:] = data
      
      write_info_1 = A.last_write_info
  print(f"attempted_time_dim_array write information: {write_info_1}")
  
except tiledb.TileDBError as e:
  print(f"Got expected error: {e}")

!du -h {attempted_time_dim_array}<jupyter_output>4.0K	attempted_time_dim_array/__meta
8.0K	attempted_time_dim_array/__schema
16K	attempted_time_dim_array<jupyter_text>Use time attribute<jupyter_code>import tiledb

# Always remove and recreate local arrays
if tiledb.array_exists(time_attr_array):
  tiledb.remove(time_attr_array)
  print(f"Removed existing {time_attr_array}")

# Create the dimensions
d1 = tiledb.Dim(name="sample_num", 
                domain=(0, data['t'].shape[0]-1), # minus one probably a zero based thing
                dtype=np.uint64)

# Create a domain using the dimension
dom1 = tiledb.Domain(d1)

# Create an attribute
t = tiledb.Attr(name="t", dtype=np.datetime64('', 'ns').dtype) # Have to specify resolution
x = tiledb.Attr(name="x", dtype=np.int16)


# Create the array schema, setting `sparse=False` to indicate a dense array
schema1 = tiledb.ArraySchema(domain=dom1, sparse=False, attrs=[t, x])

tiledb.Array.create(time_attr_array, schema1)
# Write data to tiles
write_info_1 = {}
with tiledb.open(time_attr_array, 'w') as A:
    A[:] = data
    
    write_info_1 = A.last_write_info
print(f"time_attr_array write information: {write_info_1}")
!du -h {time_attr_array}<jupyter_output>4.0K	time_attr_array/__meta
8.0K	time_attr_array/__schema
364K	time_attr_array/__1636412533243_1636412533243_0b48cd0635594a2489cd7bc9579298ed_10
380K	time_attr_array<jupyter_text>Spare array with time dimension  Chat from tileDB Slack: <NAME>  12:38 PMHello again team.  I am trying to understand how to use datetime dimensions properly for a dense array```  Trying to create an array to hold 1 second of 100Hz datansecs = 1data = np.arange(100)start_dt = np.datetime64('2000-01-01')d1 = tiledb.Dim(name="time",                 domain=(start_dt, start_dt + np.timedelta64(n_seconds, 's')),                dtype=np.datetime64(10, 'ms').dtype)  how do I specify 10ms period?dom1 = tiledb.Domain(d1)x = tiledb.Attr(name="x", dtype=np.int16)schema1 = tiledb.ArraySchema(domain=dom1, sparse=False, attrs=[x])tiledb.Array.create("time_example", schema1)This seems fine but it expecting data for every ms rather than every 10mswith tiledb.open("time_example", 'w') as A:    A[:] = data Raises: TileDBError: [TileDB::Writer] Error: Buffer sizes check failed; Invalid number of cells given for attribute 'x' (100 != 1001)(edited)white_check_markeyesraised_hands```<NAME>  12:46 PMjoined tiledb-embedded.dirk  1:04 PMMatt:  TileDB maps the NumPy datetime resolutions, cf https://numpy.org/doc/stable/reference/arrays.datetime.html which has secs, millisecs, microsecs, ... nanosesc, ...  but not every power of ten in between.   You can however use a sparse array and have datetimes (at ms resolution) and then write a record every 10 ms.<NAME>  7:47 PMHi @<NAME>, quick follow-up on that — Dirk is right, there’s no support for multiples of integral units, but sparse works well. (you can get interval spacing over a datetime range with some fiddling in numpy, as below, but it would be easier with pandas, b/c pandas supports frequencies). Here’s a quick modification of your example to use sparse:```import tiledb, numpy as np\  Trying to create an array to hold 1 second of 100Hz datansecs = 1data = np.arange(100)start_dt = np.datetime64('2000-01-01')end_dt = start_dt + np.timedelta64(nsecs, 's')coords = np.linspace(start_dt.astype('M8[ms]').astype('int64'), end_dt.astype('M8[ms]').astype('int64'), 100).astype('M8[ms]')d1 = tiledb.Dim(name="time",                 domain=(start_dt, end_dt),                dtype=np.datetime64(1, 'ms').dtype)  how do I specify 10ms period?dom1 = tiledb.Domain(d1)x = tiledb.Attr(name="x", dtype=np.int16)schema1 = tiledb.ArraySchema(domain=dom1, sparse=True, attrs=[x])  sparse -> Truetiledb.Array.create("time_example", schema1)with tiledb.open("time_example", 'w') as A:    A[coords.tolist()] = data```  Code<jupyter_code>import tiledb

# Always remove and recreate local arrays
if tiledb.array_exists(sparse_array_time_dim_array):
  tiledb.remove(sparse_array_time_dim_array)
  print(f"Removed existing {sparse_array_time_dim_array}")

# Create the dimensions
d1 = tiledb.Dim(name="time", 
                domain=(start_dt, start_dt + np.timedelta64(n_seconds, 's')),
                tile = n_seconds*sampling_rate,
                dtype=np.datetime64('', 'ms').dtype) # defines resolution I think

# Create a domain using the dimension
dom1 = tiledb.Domain(d1)

# Create an attribute
t = tiledb.Attr(name="t", dtype=np.datetime64('', 'ms').dtype) # Have to specify resolution
x = tiledb.Attr(name="x", dtype=np.int16)

# datetime in tileDB ref: https://tiledb-inc-tiledb.readthedocs-hosted.com/en/1.6.3/tutorials/datetimes.html

schema1 = tiledb.ArraySchema(domain=dom1, sparse=True, attrs=[t, x])

tiledb.Array.create(sparse_array_time_dim_array, schema1)
# Write data to tiles
end_dt = start_dt + np.timedelta64(n_seconds, 's')
coords = np.linspace(start_dt.astype('M8[ms]').astype('int64'), end_dt.astype('M8[ms]').astype('int64'), n_seconds*sampling_rate).astype('M8[ms]')

write_info_1 = {}
with tiledb.open(sparse_array_time_dim_array, 'w') as A:
    A[coords] = data
    
    write_info_1 = A.last_write_info
print(f"time_attr_array write information: {write_info_1}")
!du -h {time_attr_array}<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup