<jupyter_start><jupyter_text>Sector Neutral  Install packages<jupyter_code>import sys
!{sys.executable} -m pip install -r requirements.txt
import cvxpy as cvx
import numpy as np
import pandas as pd
import time
import os
import quiz_helper
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
plt.rcParams['figure.figsize'] = (14, 8)<jupyter_output><empty_output><jupyter_text>following zipline bundle documentationhttp://www.zipline.io/bundles.htmlingesting-data-from-csv-files  data bundle<jupyter_code>import os
import quiz_helper
from zipline.data import bundles
os.environ['ZIPLINE_ROOT'] = os.path.join(os.getcwd(), '..', '..','data','module_4_quizzes_eod')
ingest_func = bundles.csvdir.csvdir_equities(['daily'], quiz_helper.EOD_BUNDLE_NAME)
bundles.register(quiz_helper.EOD_BUNDLE_NAME, ingest_func)
print('Data Registered')<jupyter_output>Data Registered<jupyter_text>Build pipeline engine<jupyter_code>from zipline.pipeline import Pipeline
from zipline.pipeline.factors import AverageDollarVolume
from zipline.utils.calendars import get_calendar

universe = AverageDollarVolume(window_length=120).top(500) 
trading_calendar = get_calendar('NYSE') 
bundle_data = bundles.load(quiz_helper.EOD_BUNDLE_NAME)
engine = quiz_helper.build_pipeline_engine(bundle_data, trading_calendar)<jupyter_output><empty_output><jupyter_text>View Data¶With the pipeline engine built, let's get the stocks at the end of the period in the universe we're using. We'll use these tickers to generate the returns data for the our risk model.<jupyter_code>universe_end_date = pd.Timestamp('2016-01-05', tz='UTC')

universe_tickers = engine\
    .run_pipeline(
        Pipeline(screen=universe),
        universe_end_date,
        universe_end_date)\
    .index.get_level_values(1)\
    .values.tolist()
    
universe_tickers<jupyter_output><empty_output><jupyter_text>Get Returns data<jupyter_code>from zipline.data.data_portal import DataPortal

data_portal = DataPortal(
    bundle_data.asset_finder,
    trading_calendar=trading_calendar,
    first_trading_day=bundle_data.equity_daily_bar_reader.first_trading_day,
    equity_minute_reader=None,
    equity_daily_reader=bundle_data.equity_daily_bar_reader,
    adjustment_reader=bundle_data.adjustment_reader)<jupyter_output><empty_output><jupyter_text>Get pricing data helper function<jupyter_code>def get_pricing(data_portal, trading_calendar, assets, start_date, end_date, field='close'):
    end_dt = pd.Timestamp(end_date.strftime('%Y-%m-%d'), tz='UTC', offset='C')
    start_dt = pd.Timestamp(start_date.strftime('%Y-%m-%d'), tz='UTC', offset='C')

    end_loc = trading_calendar.closes.index.get_loc(end_dt)
    start_loc = trading_calendar.closes.index.get_loc(start_dt)

    return data_portal.get_history_window(
        assets=assets,
        end_dt=end_dt,
        bar_count=end_loc - start_loc,
        frequency='1d',
        field=field,
        data_frequency='daily')<jupyter_output><empty_output><jupyter_text>get pricing data into a dataframe<jupyter_code>returns_df = \
    get_pricing(
        data_portal,
        trading_calendar,
        universe_tickers,
        universe_end_date - pd.DateOffset(years=5),
        universe_end_date)\
    .pct_change()[1:].fillna(0) #convert prices into returns

returns_df<jupyter_output><empty_output><jupyter_text>Sector data helper functionWe'll create an object for you, which defines a sector for each stock.  The sectors are represented by integers.  We inherit from the Classifier class.  [Documentation for Classifier](https://www.quantopian.com/posts/pipeline-classifiers-are-here), and the [source code for Classifier](https://github.com/quantopian/zipline/blob/master/zipline/pipeline/classifiers/classifier.py)<jupyter_code>from zipline.pipeline.classifiers import Classifier
from zipline.utils.numpy_utils import int64_dtype
class Sector(Classifier):
    dtype = int64_dtype
    window_length = 0
    inputs = ()
    missing_value = -1

    def __init__(self):
        self.data = np.load('../../data/project_4_sector/data.npy')

    def _compute(self, arrays, dates, assets, mask):
        return np.where(
            mask,
            self.data[assets],
            self.missing_value,
        )
sector = Sector()
sector
len(sector.data)
sector.data<jupyter_output><empty_output><jupyter_text>Quiz 1How many unique sectors are in the sector variable?  Answer 1 here<jupyter_code>print(f"set of unique categories: {set(sector.data)}")<jupyter_output>set of unique categories: {0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, -1}<jupyter_text>Create an alpha factor based on momentumWe want to calculate the one-year return.  In other words, get the close price of today, minus the close price of 252 trading days ago, and divide by that price from 252 days ago.$1YearReturn_t = \frac{price_{t} - price_{t-252}}{price_{t-252}}$<jupyter_code>from zipline.pipeline.factors import Returns<jupyter_output><empty_output><jupyter_text>We'll use 2 years of data to calculate the factor **Note:** Going back 2 years falls on a day when the market is closed. Pipeline package doesn't handle start or end dates that don't fall on days when the market is open. To fix this, we went back 2 extra days to fall on the next day when the market is open.<jupyter_code>factor_start_date = universe_end_date - pd.DateOffset(years=2, days=2)
factor_start_date
## 1 year returns can be the basis for an alpha factor
p1 = Pipeline(screen=universe)
rets1 = Returns(window_length=252, mask=universe)
p1.add(rets1,"1YearReturns")
df1 = engine.run_pipeline(p1, factor_start_date, universe_end_date)
#graphviz lets us visualize the pipeline
import graphviz
p1.show_graph(format='png')<jupyter_output><empty_output><jupyter_text>View the data of the factor<jupyter_code>df1.head()<jupyter_output><empty_output><jupyter_text>Explore the demean functionThe Returns class inherits from zipline.pipeline.factors.factor.  [The documentation for demean is located here](https://www.zipline.io/appendix.htmlzipline.pipeline.factors.Factor.demean), and is also pasted below:```demean(mask=sentinel('NotSpecified'), groupby=sentinel('NotSpecified'))[source]Construct a Factor that computes self and subtracts the mean from row of the result.If mask is supplied, ignore values where mask returns False when computing row means, and output NaN anywhere the mask is False.If groupby is supplied, compute by partitioning each row based on the values produced by groupby, de-meaning the partitioned arrays, and stitching the sub-results back together.Parameters:	mask (zipline.pipeline.Filter, optional) – A Filter defining values to ignore when computing means.groupby (zipline.pipeline.Classifier, optional) – A classifier defining partitions over which to compute means.```  Quiz 2By looking at the documentation, and then the source code for `demean`, what are two parameters for this function?  Which one or ones would you call if you wanted to demean by sector and wish to demean for all values in the chosen universe?[The source code](https://www.zipline.io/_modules/zipline/pipeline/factors/factor.htmlFactor.demean) has useful comments to help you answer this question.  Answer 2 here We would use the groupby parameter, and we don't need to use the mask parameter, since we are not going to exclude any of the stocks in the universe from the demean calculation.  Quiz 3Turn 1 year returns into an alpha factorWe can do some processing to convert our signal (1 year return) into an alpha factor. One step is to demean by sector.* demeanFor each stock, we want to take the average return of stocks that are in the same sector, and then remove this from the return of each individual stock.  Answer 3<jupyter_code>#TODO
# create a pipeline called p2
p2 = Pipeline(screen=universe)
# create a factor of one year returns, deman by sector
factor_demean_by_sector = (
    Returns(window_length=252, mask=universe).
    demean(groupby=Sector()) #we use the custom Sector class that we reviewed earlier
)
# add the factor to the p2 pipeline
p2.add(factor_demean_by_sector, 'Momentum_1YR_demean_by_sector')<jupyter_output><empty_output><jupyter_text>visualize the second pipeline<jupyter_code>p2.show_graph(format='png')<jupyter_output><empty_output><jupyter_text>Quiz 4How does this pipeline compare with the first pipeline that we created earlier?  Answer 4 hereThe second pipeline now adds sector information in the GroupedRowTransform('demean') step.  run pipeline and view the factor data<jupyter_code>df2 = engine.run_pipeline(p2, factor_start_date, universe_end_date)
df2.head()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup