# ---
# jupyter:
#   jupytext:
#     text_representation:
#       extension: .py
#       format_name: light
#       format_version: '1.5'
#       jupytext_version: 1.14.4
#   kernelspec:
#     display_name: Python 3
#     language: python
#     name: python3
# ---

# ## Part 3: Working With Remote Servers
#
# This tutorial will introduce how to start a remote Hangar server, and how to work with [remotes](api.rst#hangar.repository.Remotes) from the client side.
#
# Particular attention is paid to the concept of a ***partially fetch* / *partial clone*** operations. This is a key component of the Hangar design which provides the ability to quickly and efficiently work with data contained in remote repositories whose full size would be significatly prohibitive to local use under most circumstances.
#
# *Note:*
#
# > At the time of writing, the API, user-facing functionality, client-server negotiation protocols, and test coverage of the remotes implementation is generally adqequate for this to serve as an "alpha" quality preview. However, please be warned that significantly less time has been spent in this module to optimize speed, refactor for simplicity, and assure stability under heavy loads than the rest of the Hangar core. While we can guarantee that your data is secure on disk, you may experience crashes from time to time when working with remotes. In addition, sending data over the wire should NOT be considered secure in ANY way. No in-transit encryption, user authentication, or secure access limitations are implemented at this moment. We realize the importance of these types of protections, and they are on our radar for the next release cycle. If you are interested in making a contribution to Hangar, this module contains a lot of low hanging fruit which would would provide drastic improvements and act as a good intro the the internal Hangar data model. Please get in touch with us to discuss!

# ### Starting a Hangar Server
#
# To start a Hangar server, navigate to the command line and simply execute:
#
# ```
# $ hangar server
# ```
#
# This will get a local server instance running at `localhost:50051`. The IP and port can be configured by setting the `--ip` and `--port` flags to the desired values in the command line.
#
# A blocking process will begin in that terminal session. Leave it running while you experiment with connecting from a client repo.

# ### Using Remotes with a Local Repository
#
# The [CLI](cli.rst#hangar-cli-documentation) is the easiest way to interact with the remote server from a local repository (though all functioanlity is mirrorred via the [repository API](api.rst#hangar.repository.Remotes) (more on that later).
#
# Before we begin we will set up a repository with some data, a few commits, two branches, and a merge.

# #### Setup a Test Repo
#
# As normal, we shall begin with creating a repository and adding some data. This should be familiar to you from previous tutorials.

# +
from hangar import Repository
import numpy as np
from tqdm import tqdm

testData = np.loadtxt('/Users/rick/projects/tensorwerk/hangar/dev/data/dota2Dataset/dota2Test.csv', delimiter=',', dtype=np.uint8)
trainData = np.loadtxt('/Users/rick/projects/tensorwerk/hangar/dev/data/dota2Dataset/dota2Train.csv', delimiter=',', dtype=np.uint16)

testName = 'test'
testPrototype = testData[0]
trainName = 'train'
trainPrototype = trainData[0]
# -

repo = Repository('/Users/rick/projects/tensorwerk/hangar/dev/intro/')
repo.init(user_name='<NAME>', user_email='<EMAIL>', remove_old=True)
co = repo.checkout(write=True)

# +
co.add_ndarray_column(testName, prototype=testPrototype)
testcol = co.columns[testName]

pbar = tqdm(total=testData.shape[0])
with testcol as tcol:
    for gameIdx, gameData in enumerate(testData):
        if (gameIdx % 500 == 0):
            pbar.update(500)
        tcol.append(gameData)
pbar.close()

co.commit('initial commit on master with test data')

repo.create_branch('add-train')
co.close()
repo.log()

# +
co = repo.checkout(write=True, branch='add-train')

co.add_ndarray_column(trainName, prototype=trainPrototype)
traincol = co.columns[trainName]

pbar = tqdm(total=trainData.shape[0])
with traincol as trcol:
    for gameIdx, gameData in enumerate(trainData):
        if (gameIdx % 500 == 0):
            pbar.update(500)
        trcol.append(gameData)
pbar.close()

co.commit('added training data on another branch')
co.close()
repo.log()
# -

co = repo.checkout(write=True, branch='master')
co.metadata['earaea'] = 'eara'
co.commit('more changes here')
co.close()
repo.log()

# ### Pushing to a Remote

# We will use the [API remote add()](api.rst#hangar.repository.Remotes.add) method to add a remote, however, this can also be done with the [CLI command](cli.rst#hangar-remote-add):
#
#     $ hangar remote add origin localhost:50051

repo.remote.add('origin', 'localhost:50051')

# Pushing is as simple as running the [push()](api.rst#hangar.repository.Remotes.push) method
# from the [API](api.rst#hangar.repository.Remotes.push) or [CLI](cli.rst#hangar-push):
#
#     $ hangar push origin master

# Push the `master` branch:

repo.remote.push('origin', 'master')

# Push the `add-train` branch:

repo.remote.push('origin', 'add-train')

# #### Details of the Negotiation Processs
#
# > The following details are not necessary to use the system, but may be of interest to some readers
#
# When we push data, **we perform a negotation with the server** which basically occurs like this:
#
#
# - Hi, I would like to push this branch, do you have it?
#
#   - If yes, what is the latest commit you record on it?
#
#     - Is that the same commit I'm trying to push? If yes, abort.
#
#     - Is that a commit I don't have? If yes, someone else has updated that branch, abort.
#
# - Here's the commit digests which are parents of my branches head, which commits are you missing?
#
# - Ok great, I'm going to scan through each of those commits to find the data hashes they contain. Tell me which ones you are missing.
#
# - Thanks, now I'll send you all of the data corresponding to those hashes. It might be a lot of data, so we'll handle this in batches so that if my connection cuts out, we can resume this later
#
# - Now that you have the data, I'm going to send the actual commit references for you to store, this isn't that much information, but you'll be sure to verify that I'm not trying to pull any funny buisness and send you incorrect data.
#
# - Now that you've received everything, and have verified it matches what I told you it is, go ahead and make those commits I've pushed `available` as the `HEAD` of the branch I just sent. It's some good work that others will want!
#
#
# When we want to fetch updates to a branch, essentially the exact same thing happens in reverse. Instead of asking the server what it doesn't have, we ask it what it does have, and then request the stuff that we are missing!
#

# ### Partial Fetching and Clones
#
# **Now we will introduce one of the most important and unique features of Hangar remotes: Partial fetch/clone of data!**
#
# *There is a very real problem with keeping the full history of data - **it's huge**!* The size of data can very easily exceeds what can fit on (most) contributors laptops or personal workstations. This section explains how Hangar can handle working with columns which are prohibitively large to download or store on a single machine.
#
# As mentioned in High Performance From Simplicity, under the hood Hangar deals with “Data” and “Bookkeeping” completely separately. We’ve previously covered what exactly we mean by Data in How Hangar Thinks About Data, so we’ll briefly cover the second major component of Hangar here.
# In short “Bookkeeping” describes everything about the repository. By everything, we do mean that the Bookkeeping records describe everything: all commits, parents, branches, columns, samples, data descriptors, schemas, commit message, etc. Though complete, these records are fairly small (tens of MB in size for decently sized repositories with decent history), and are highly compressed for fast transfer between a Hangar client/server.
#
# A brief technical interlude
#
# > There is one very important (and rather complex) property which gives Hangar Bookeeping massive power: existence of some data piece is always known to Hangar and stored immutably once committed. However, the access pattern, backend, and locating information for this data piece may (and over time, will) be unique in every hangar repository instance.
# >
# > Though the details of how this works is well beyond the scope of this document, the following example may provide some insight into the implications of this property:
# >
# > If you clone some Hangar repository, Bookeeping says that “some number of data pieces exist” and they should retrieved from the server. However, the bookeeping records transfered in a fetch / push / clone operation do not include information about where that piece of data existed on the client (or server) computer. Two synced repositories can use completly different backends to store the data, in completly different locations, and it does not matter - Hangar only guarantees that when collaborators ask for a data sample in some checkout, that they will be provided with identical arrays, not that they will come from the same place or be stored in the same way. Only when data is actually retrieved the “locating information” is set for that repository instance.
# Because Hangar makes no assumptions about how/where it should retrieve some piece of data, or even an assumption that it exists on the local machine, and because records are small and completely describe history, once a machine has the Bookkeeping, it can decide what data it actually wants to materialize on its local disk! These partial fetch / partial clone operations can materialize any desired data, whether it be for a few records at the head branch, for all data in a commit, or for the entire historical data. A future release will even include the ability to stream data directly to a Hangar checkout and materialize the data in memory without having to save it to disk at all!
#
# More importantly: since Bookkeeping describes all history, merging can be performed between branches which may contain partial (or even no) actual data. Aka **you don’t need data on disk to merge changes into it.** It’s an odd concept which will be shown in this tutorial

# ### Cloning a Remote Repo

#     $ hangar clone localhost:50051

cloneRepo = Repository('/Users/rick/projects/tensorwerk/hangar/dev/dota-clone/')

# When we perform the initial clone, we will only receive the `master` branch by default.

cloneRepo.clone('<NAME>', '<EMAIL>', 'localhost:50051', remove_old=True)

cloneRepo.log()

cloneRepo.list_branches()

# To get the `add-train` branch, we [fetch](api.rst#hangar.repository.Remotes.fetch) it from the remote:

cloneRepo.remote.fetch('origin', 'add-train')

cloneRepo.list_branches()

cloneRepo.log(branch='origin/add-train')

# We will create a local branch from the `origin/add-train` branch, just like in `Git`

cloneRepo.create_branch('add-train', 'a=957d20e4b921f41975591cc8ee51a4a6912cb919')

cloneRepo.list_branches()

cloneRepo.log(branch='add-train')

# #### Checking out a Parial Clone/Fetch
#
# When we [fetch](api.rst#hangar.repository.Remotes.fetch)/[clone](api.rst#hangar.repository.Repository.clone), the transfers are very quick, because only the commit records/history were retrieved. The data was not sent, because it may be very large to get the entire data across all of history.
#
# When you check out a commit with partial data, you will be shown a warning indicating that some data is not available locally. An error is raised if you try to access that particular sample data. Otherwise, everything will appear as normal.

co = cloneRepo.checkout(branch='master')

co

# we can see from the `repr` that the columns contain partial remote references

co.columns

co.columns['test']

testKey = next(co.columns['test'].keys())

co.columns['test'][testKey]

# #### Fetching Data from a Remote
#
# To retrieve the data, we use the [fetch_data()](api.rst#hangar.repository.Remotes.fetch_data)
# method (accessible via the [API](api.rst#hangar.repository.Remotes.fetch_data) or
# [fetch-data](cli.rst#hangar-fetch-data) via the CLI).
#
# The amount / type of data to retrieve is extremly configurable via the following options:
#
# .. include:: ./noindexapi/apiremotefetchdata.rst
#
# This will retrieve all the data on the `master` branch, but not on the `add-train` branch.

cloneRepo.remote.fetch_data('origin', branch='master')

co = cloneRepo.checkout(branch='master')

co

# Unlike before, we see that there is no partial references from the `repr`

co.columns

co.columns['test']

# ***When we access the data this time, it is available and retrieved as requested!***

co['test', testKey]

co.close()

# #### Working with mixed local / remote checkout Data
#
# If we were to checkout the `add-train` branch now, we would see that there is no `arrayset "train"` data, but there will be data common to the ancestor that `master` and `add-train` share.

cloneRepo.log('add-train')

# In this case, the common ancestor is commit: `9<PASSWORD>`

# To show that there is no data on the `add-train` branch:

co = cloneRepo.checkout(branch='add-train')

co

co.columns

co['test', testKey]

trainKey = next(co.columns['train'].keys())

co.columns['train'][trainKey]

co.close()

# ### Merging Branches with Partial Data
#
# Even though we don't have the actual data references in the `add-train` branch, it is still possible to merge the two branches!
#
# This is possible because Hangar doesn't use the data contents in its internal model of checkouts / commits, but instead thinks of a checkouts as a sequence of columns / metadata / keys & their associated data hashes (which are very small text records; ie. "bookkeeping"). To show this in action, lets merge the two branches `master` (containing all data locally) and `add-train` (containing partial remote references for the `train` arrayset) together and push it to the Remote!

cloneRepo.log('master')

cloneRepo.log('add-train')

# **Perform the Merge**

cloneRepo.merge('merge commit here', 'master', 'add-train')

# **IT WORKED!**

cloneRepo.log()

# We can check the summary of the master commit to check that the contents are what we expect (containing both `test` and `train` columns)

cloneRepo.summary()

# #### Pushing the Merge back to the Remote
#
# To push this merge back to our original copy of the Repository (`repo`), we just push the `master` branch back to the remote via the API or CLI.

cloneRepo.remote.push('origin', 'master')

# Looking at our current state of our other instance of the repo `repo` we see that the merge changes aren't yet propogated to it (since it hasn't fetched from the remote yet).

repo.log()

# To fetch the merged changes, just [fetch()](api.rst#hangar.repository.Remotes.fetch) the branch as normal. Like all fetches, this will be a fast operation, as it will be a `partial fetch` operation, not actually transfering the data.

repo.remote.fetch('origin', 'master')

repo.log('origin/master')

# To bring our `master` branch up to date is a simple fast-forward merge.

repo.merge('ff-merge', 'master', 'origin/master')

repo.log()

# **Everything is as it should be!** Now, try it out for yourself!

repo.summary()



StarcoderdataJupyterScriptsDedupFiltered