# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# # Cadence Optimization for AGN selection in LSST

# The sampling of observations is crucial to AGN selection by varaiablity. Here, we will be investigating how different proposed cadence for LSST can affect AGN selections, in particular we would like to see whether non-uniform cadence like rolling cadence will hurt AGN science in LSST. 

# The basic procdeuce is described below, with links to notebooks that expand each step in more details:
# 1. Get LSST cadence and minimum seperation between two visits within one night using MAF
# 2. Simulate mock light curves using [Kali](https://github.com/AstroVPK/kali), using CARMA(1,0)/Damped Random Walk(**DRW**) & CARMA(2,1)/Damped Harmonic Oscillator(**DHO**) model sampled at the minimum seperation obtained above. 
# 3. Downsample the mock LC at the LSST cadence obtained from step one.
# 4, Use Kali to find the best-fit CARMA parameters and compare the best-fit parameters with input parameters used to generate the mock LCs to determine the quality of each cadence being tested.
#
#
# - step 1, see [Step 1](#Step-1)  
# - setp 2 - 3, see [Kali notebook](lc.ipynb)  
# - comparision between cadences, see [compare](compare.ipynb)  

# ## Step 1

# The Operation Simulator(OpSim) is a sofeware developed by the LSST DM team to simulate observing schedules in order to better optimize the LSST output for different science goals. Simulated observing schedules are stored in SQLite database files. The offical baseline cadence includes an universal Wide-Fast-Deep(WFD) main survey and several mini-surveys such as the Deep Drilling Fields(DDFs). The WFD survey will scan the sky uniformly and covers entire visible sky within approximately 3 days. A detailed description for the current baseline survey and different variations can be obtained from the [LSST Observing Strategy White Paper](https://arxiv.org/pdf/1708.04058.pdf). The links to the database files storing the observing schedules, with short description of each cadence, is available fron the offical [LSST web page](https://www.lsst.org/scientists/simulations/opsim/lsst-survey-strategy).  
#
# The Metrics Analysis Framework(MAF) is another tools developed by the LSST DM team for easy access and analysis of OpSim output. In the rest of this notebook, I will show how to use MAF to get observing times and minimum seperation between visits within the same night given the RA and DEC for a paritcular location in the sky. The cadence used to demonstrate the workflow is minon_1016, the baseline cadence simulated using OpSim v3.3.5. The code can be used on all other simulated cadences(transion from OpSim v3 to v4 should be taken care of)

import lsst.sims.maf
lsst.sims.maf.__version__

# %matplotlib inline   
import matplotlib.pyplot as plt
import numpy as np

# import our python modules
import lsst.sims.maf.db as db
import lsst.sims.maf.metrics as metrics
import lsst.sims.maf.slicers as slicers
import lsst.sims.maf.stackers as stackers
import lsst.sims.maf.plots as plots
import lsst.sims.maf.metricBundles as metricBundles

# ### minion_1016

from numpy import amin as m

# +
# setup database connection, path should be modified accordingly
outDir ='minior_1016' # this is not important
Baseline = '/home/mount/Opsim DB/minion_1016_sqlite.db'

# specify output directory for metrics result
resultsDb = db.ResultsDb(outDir=outDir)


# -

# retrive min inter_night gap, and observation history with the input of database file name and
# arrays of RA and DEC
def find_min_gap(dbFile, ra, dec):
    # establish connection to sqllite database file
    opsimdb = db.OpsimDatabase(dbFile)
    
    # While we're in transition between opsim v3 and v4, this may be helpful:
    # print("{dbFile} is an opsim version {version} database".format(dbFile=dbFile, version=opsimdb.opsimVersion))
    if opsimdb.opsimVersion == "V3":
        # For v3 databases:
        mjdcol = 'expMJD'
        degrees = False
    else:
        # For v4 and alternate scheduler databases.
        mjdcol = 'observationStartMJD'
        degrees = True
    
    # IntraNightGapsMetric returns the gap (in days) between observations within the same night
    # custom reduceFunc to find min gaps 
    metric = metrics.cadenceMetrics.IntraNightGapsMetric(reduceFunc=np.amin, mjdCol=mjdcol)
    # PassMetric just pass all values
    metric_pass = metrics.simpleMetrics.PassMetric(cols=['filter','fiveSigmaDepth', mjdcol, 'expDate'])
    # slicer for slicing pointing history
    slicer = slicers.UserPointsSlicer(ra, dec, lonCol='fieldRA', latCol='fieldDec', latLonDeg=degrees)
    # sql constrains, here I put none
    sql = '' #'night < 365'
    
    # bundles to combine metric, slicer and sql constrain together
    bundle = metricBundles.MetricBundle(metric,slicer,sql)
    date_bundle = metricBundles.MetricBundle(metric_pass, slicer, sql)
    # In case you are using a dither stacker, we can check what columns are actually being pulled from the database.
    print(bundle.dbCols)
    
    # create metric bundle group and returns
    bg = metricBundles.MetricBundleGroup({'sep': bundle, 'cadence':date_bundle}, opsimdb, outDir=outDir, resultsDb=resultsDb)
    bg.runAll()
    opsimdb.close()
    return bg


# specify ra & dec
ra = 58
dec = -27

# Now baseline 
BL_result = find_min_gap(Baseline, ra, dec)

# +
# assign min intra_night gap in hours to gap
gap = BL_result.bundleDict['sep'].metricValues.data[0]
# assign pointing history to obsHist
cadence = BL_result.bundleDict['cadence'].metricValues.data[0]
# put ra, dec, and gap into array 'pos_gap'
meta = [ra, dec, gap, 'minion_1016']

# Now let's save it to a file, .npz will be automatically added
outfile = '/home/mount/MAF output/58_-27_bl'
np.savez(outfile, meta=meta, cadence=cadence)
# -

# ***

# In the cell above, I saved the MAF output to a npz file for later use. To load the data from the npz to a numpy array, just use `data = numpy.load(filename)` in the destination notebook. To checkout the arrays stored in the file, use `data.files`. To access the arrays stored, you can use `ar1 = data['filename1']`. For more verbose description, please checkout the [numpy documentation](https://docs.scipy.org/doc/numpy/reference/generated/numpy.savez.html).

# In my code, I saved two arrays into the file '58_-27_bl.npz' and give each array a name. The 'meta' array contains the ra, dec, minimum seperation between two visits within one night and the cadence ID. The cadence array stores the simulated observing data at that paricular location. First few record for the cadence array is shown below:

cadence[:10]

StarcoderdataJupyterScriptsDedupFiltered