<jupyter_start><jupyter_text>Part 2: Checkouts, Branching, & MergingThis section deals with navigating repository history, creating & mergingbranches, and understanding conflicts  The Hangar WorkflowThe hangar workflow is intended to mimic common ``git`` workflows in which smallincremental changes are made and committed on dedicated ``topic`` branches.After the ``topic`` has been adequatly set, ``topic`` branch is ``merged`` intoa separate branch (commonly referred to as ``master``, though it need not be theactual branch named ``"master"``), where well vetted and more permanent changesare kept.    Create Branch -> Checkout Branch -> Make Changes -> Commit Making the Initial CommitLet's initialize a new repository and see how branching works in Hangar<jupyter_code>from hangar import Repository
import numpy as np
repo = Repository(path='foo/pth')
repo_pth = repo.init(user_name='Test User', user_email='<EMAIL>')<jupyter_output><empty_output><jupyter_text>When a repository is first initialized, it has no history, no commits.<jupyter_code>repo.log() # -> returns None<jupyter_output><empty_output><jupyter_text>Though the repository is essentially empty at this point in time, there is onething which is present: A branch with the name: ``"master"``.<jupyter_code>repo.list_branches()<jupyter_output><empty_output><jupyter_text>This ``"master"`` is the branch we make our first commit on; until we do, therepository is in a semi-unstable state; with no history or contents, most of thefunctionality of a repository (to store, retrieve, and work with versions ofdata across time) just isn't possible. A significant potion of otherwisestandard operations will generally flat out refuse to to execute (ie. read-onlycheckouts, log, push, etc.) until the first commit is made.One of the only options available at this point in time is to create awrite-enabled checkout on the ``"master"`` branch and begin to add data so wecan make a commit. let’s do that now:<jupyter_code>co = repo.checkout(write=True)<jupyter_output><empty_output><jupyter_text>As expected, there are no arraysets or metadata samples recorded in the checkout.<jupyter_code>print(f'number of metadata keys: {len(co.metadata)}')
print(f'number of arraysets: {len(co.arraysets)}')<jupyter_output>number of metadata keys: 0
number of arraysets: 0<jupyter_text>Let’s add a dummy array just to put something in the repository history tocommit. We'll then close the checkout so we can explore some useful tools whichdepend on having at least on historical record (commit) in the repo.<jupyter_code>dummy = np.arange(10, dtype=np.uint16)
aset = co.arraysets.init_arrayset(name='dummy_arrayset', prototype=dummy)
aset['0'] = dummy
initialCommitHash = co.commit('first commit with a single sample added to a dummy arrayset')
co.close()<jupyter_output><empty_output><jupyter_text>If we check the history now, we can see our first commit hash, and that it is labeled with the branch name `"master"`<jupyter_code>repo.log()<jupyter_output>* 0fd892b7ce9d9d0150c68bb5483876d58c28cbf1 ([1;31mmaster[m) : first commit with a single sample added to a dummy arrayset<jupyter_text>So now our repository contains:- A commit: a fully independent description of the entire repository state as  it existed at some point in time. A commit is identified by a `commit_hash`- A branch: a label pointing to a particular `commit` / `commit_hash`Once committed, it is not possible to remove, modify, or otherwise tamper withthe contents of a commit in any way. It is a permanent record, which Hangar hasno method to change once written to disk.In addition, as a ``commit_hash`` is not only calculated from the ``commit``\ ’scontents, but from the ``commit_hash`` of its parents (more on this to follow),knowing a single top-level ``commit_hash`` allows us to verify the integrity ofthe entire repository history. This fundamental behavior holds even in cases ofdisk-corruption or malicious use. Working with Checkouts & BranchesAs mentioned in the first tutorial, we work with the data in a repository thougha ``checkout``. There are two types of checkouts (each of which have differentuses and abilities):**Checking out a branch/commit for reading:** is the process of retrievingrecords describing repository state at some point in time, and setting up accessto the referenced data.-  Any number of read checkout processes can operate on a repository (on   any number of commits) at the same time.**Checking out a branch for writing:** is the process of setting up a (mutable)``staging area`` to temporarily gather record references / data before allchanges have been made and staging area contents are ``committed`` in a newpermanent record of history (a ``commit``)-  Only one write-enabled checkout can ever be operating in a repository   at a time-  When initially creating the checkout, the ``staging area`` is not   actually “empty”. Instead, it has the full contents of the last ``commit``   referenced by a branch’s ``HEAD``. These records can be removed/mutated/added   to in any way to form the next ``commit``. The new ``commit`` retains a   permanent reference identifying the previous ``HEAD`` ``commit`` was used as   it’s base ``staging area``-  On commit, the branch which was checked out has it’s ``HEAD`` pointer   value updated to the new ``commit``\ ’s ``commit_hash``. A write-enabled   checkout starting from the same branch will now use that ``commit``\ ’s   record content as the base for it’s ``staging area``. Creating a branchA branch is an individual series of changes/commits which diverge from the mainhistory of the repository at some point in time. All changes made along a branchare completely isolated from those on other branches. After some point in time,changes made in a disparate branches can be unified through an automatic``merge`` process (described in detail later in this tutorial). In general, the``Hangar`` branching model is semantically identical ``Git``; Hangar branchesalso have the same lightweight and performant properties which make working with``Git`` branches so appealing.In hangar, branch must always have a ``name`` and a ``base_commit``. However, Ifno ``base_commit`` is specified, the current writer branch ``HEAD`` ``commit``is used as the ``base_commit`` hash for the branch automatically.<jupyter_code>branch_1 = repo.create_branch(name='testbranch')
branch_1<jupyter_output><empty_output><jupyter_text>viewing the log, we see that a new branch named: `testbranch` is pointing to our initial commit<jupyter_code>print(f'branch names: {repo.list_branches()} \n')
repo.log()<jupyter_output>branch names: ['master', 'testbranch'] 

* 0fd892b7ce9d9d0150c68bb5483876d58c28cbf1 ([1;31mmaster[m) ([1;31mtestbranch[m) : first commit with a single sample added to a dummy arrayset<jupyter_text>If instead, we do actually specify the base commit (with a different branchname) we see we do actually get a third branch. pointing to the same commit as``"master"`` and ``"testbranch"``<jupyter_code>branch_2 = repo.create_branch(name='new', base_commit=initialCommitHash)
branch_2
repo.log()<jupyter_output>* 0fd892b7ce9d9d0150c68bb5483876d58c28cbf1 ([1;31mmaster[m) ([1;31mnew[m) ([1;31mtestbranch[m) : first commit with a single sample added to a dummy arrayset<jupyter_text>Making changes on a branchLet’s make some changes on the ``"new"`` branch to see how things work. We cansee that the data we added previously is still here (``dummy`` arrayset containingone sample labeled ``0``)<jupyter_code>co = repo.checkout(write=True, branch='new')
co.arraysets
co.arraysets['dummy_arrayset']
co.arraysets['dummy_arrayset']['0']<jupyter_output><empty_output><jupyter_text>Let's add another sample to the `dummy_arrayset` called `1`<jupyter_code>arr = np.arange(10, dtype=np.uint16)
# let's increment values so that `0` and `1` aren't set to the same thing
arr += 1  

co.arraysets['dummy_arrayset']['1'] = arr<jupyter_output><empty_output><jupyter_text>We can see that in this checkout, there are indeed, two samples in the `dummy_arrayset`<jupyter_code>len(co.arraysets['dummy_arrayset'])<jupyter_output><empty_output><jupyter_text>That's all, let's commit this and be done with this branch<jupyter_code>co.commit('commit on `new` branch adding a sample to dummy_arrayset')
co.close()<jupyter_output><empty_output><jupyter_text>How do changes appear when made on a branch?If we look at the log, we see that the branch we were on (`new`) is a commit ahead of `master` and `testbranch`<jupyter_code>repo.log()<jupyter_output>* 186f1ccae28ad8f58bcae95dd8c1115a3b0de9dd ([1;31mnew[m) : commit on `new` branch adding a sample to dummy_arrayset
* 0fd892b7ce9d9d0150c68bb5483876d58c28cbf1 ([1;31mmaster[m) ([1;31mtestbranch[m) : first commit with a single sample added to a dummy arrayset<jupyter_text>The meaning is exactly what one would intuit. we made some changes, they werereflected on the ``new`` branch, but the ``master`` and ``testbranch`` brancheswere not impacted at all, nor were any of the commits!  Merging (Part 1) Fast-Forward MergesSay we like the changes we made on the ``new`` branch so much that we want themto be included into our ``master`` branch! How do we make this happen for thisscenario??Well, the history between the ``HEAD`` of the ``"new"`` and the ``HEAD`` of the``"master"`` branch is perfectly linear. In fact, when we began making changeson ``"new"``, our staging area was *identical* to what the ``"master"`` ``HEAD``commit references are right now!If you’ll remember that a branch is just a pointer which assigns some ``name``to a ``commit_hash``, it becomes apparent that a merge in this case reallydoesn’t involve any work at all. With a linear history between ``"master"`` and``"new"``, any ``commits`` exsting along the path between the ``HEAD`` of``"new"`` and ``"master"`` are the only changes which are introduced, and we canbe sure that this is the only view of the data records which can exist!What this means in practice is that for this type of merge, we can just updatethe ``HEAD`` of ``"master"`` to point to the ``"HEAD"`` of ``"new"``, and themerge is complete.This situation is referred to as a **Fast Forward (FF) Merge**. A FF merge issafe to perform any time a linear history lies between the ``"HEAD"`` of some``topic`` and ``base`` branch, regardless of how many commits or changes whichwere introduced.For other situations, a more complicated **Three Way Merge** is required. Thismerge method will be explained a bit more later in this tutorial<jupyter_code>co = repo.checkout(write=True, branch='master')<jupyter_output><empty_output><jupyter_text>Performing the MergeIn practice, you’ll never need to know the details of the merge theory explainedabove (or even remember it exists). Hangar automatically figures out which mergealgorithms should be used and then performed whatever calculations are needed tocompute the results.As a user, merging in Hangar is a one-liner!<jupyter_code>co.merge(message='message for commit (not used for FF merge)', dev_branch='new')<jupyter_output><empty_output><jupyter_text>Let's check the log!<jupyter_code>repo.log()
co.branch_name
co.commit_hash
co.arraysets['dummy_arrayset']<jupyter_output><empty_output><jupyter_text>As you can see, everything is as it should be!<jupyter_code>co.close()<jupyter_output><empty_output><jupyter_text>Making a changes to introduce diverged historiesLet’s now go back to our ``"testbranch"`` branch and make some changes there sowe can see what happens when changes don’t follow a linear history.<jupyter_code>co = repo.checkout(write=True, branch='testbranch')
co.arraysets
co.arraysets['dummy_arrayset']<jupyter_output><empty_output><jupyter_text>We will start by mutating sample `0` in `dummy_arrayset` to a different value<jupyter_code>dummy_aset = co.arraysets['dummy_arrayset']
old_arr = dummy_aset['0']
new_arr = old_arr + 50
new_arr
dummy_aset['0'] = new_arr<jupyter_output><empty_output><jupyter_text>let’s make a commit here, then add some metadata and make a new commit (all onthe ``testbranch`` branch)<jupyter_code>co.commit('mutated sample `0` of `dummy_arrayset` to new value')
repo.log()
co.metadata['hello'] = 'world'
co.commit('added hellow world metadata')
co.close()<jupyter_output><empty_output><jupyter_text>Looking at our history how, we see that none of the original branches referenceour first commit anymore<jupyter_code>repo.log()<jupyter_output>* 836ba8ff1fe552fb65944e2340b2a2ef2b2b62d4 ([1;31mtestbranch[m) : added hellow world metadata
* 2fe5c53a899ba6accbe8c19debd9a489e3baeaed : mutated sample `0` of `dummy_arrayset` to new value
* <KEY> : first commit with a single sample added to a dummy arrayset<jupyter_text>We can check the history of the ``"master"`` branch by specifying it asan argument to the ``log()`` method<jupyter_code>repo.log('master')<jupyter_output>* 1<PASSWORD> ([1;31mmaster[m) ([1;31mnew[m) : commit on `new` branch adding a sample to dummy_arrayset
* <KEY> : first commit with a single sample added to a dummy arrayset<jupyter_text>Merging (Part 2) Three Way MergeIf we now want to merge the changes on `"testbranch"` into `"master"`, we can't just follow a simple linear history; **the branches have diverged**. For this case, Hangar implements a **Three Way Merge** algorithm which does the following:- Find the most recent common ancestor `commit` present in both the `"testbranch"` and `"master"` branches- Compute what changed between the common ancestor and each branch's `HEAD` commit- Check if any of the changes conflict with eachother (more on this in a later tutorial)- If no conflicts are present, compute the results of the merge between the two sets of changes- Create a new `commit` containing the merge results reference both branch `HEAD`s as parents of the new `commit`, and update the `base` branch `HEAD` to that new     `commit`'s `commit_hash`<jupyter_code>co = repo.checkout(write=True, branch='master')<jupyter_output><empty_output><jupyter_text>Once again, as a user, the details are completely irrelevant, and the operationoccurs from the same one-liner call we used before for the FF Merge.<jupyter_code>co.merge(message='merge of testbranch into master', dev_branch='testbranch')<jupyter_output><empty_output><jupyter_text>If we now look at the log, we see that this has a much different look thenbefore. The three way merge results in a history which references changes madein both diverged branches, and unifies them in a single ``commit``<jupyter_code>repo.log()<jupyter_output>*   fd4a07ada0f138870924fc4ffee47839b77f1fbe ([1;31mmaster[m) : merge of testbranch into master
[1;31m|[m[1;32m\[m  
[1;31m|[m * 836ba8ff1fe552fb65944e2340b2a2ef2b2b62d4 ([1;31mtestbranch[m) : added hellow world metadata
[1;31m|[m * 2fe5c53a899ba6accbe8c19debd9a489e3baeaed : mutated sample `0` of `dummy_arrayset` to new value
* [1;32m|[m 186f1ccae28ad8f58bcae95dd8c1115a3b0de9dd ([1;31mnew[m) : commit on `new` branch adding a sample to dummy_arrayset
[1;32m|[m[1;32m/[m  
* 0fd892b7ce9d9d0150c68bb5483876d58c28cbf1 : first commit with a single sample added to a dummy arrayset<jupyter_text>Manually inspecting the merge result to verify it matches our expectations``dummy_arrayset`` should contain two arrays, key ``1`` was set in the previouscommit originally made in ``"new"`` and merged into ``"master"``. Key ``0`` wasmutated in ``"testbranch"`` and unchanged in ``"master"``, so the update from``"testbranch"`` is kept.There should be one metadata sample with they key ``"hello"`` and the value``"world"``<jupyter_code>co.arraysets
co.arraysets['dummy_arrayset']
co.arraysets['dummy_arrayset']['0']
co.arraysets['dummy_arrayset']['1']
co.metadata
co.metadata['hello']<jupyter_output><empty_output><jupyter_text>**The Merge was a success!**<jupyter_code>co.close()<jupyter_output><empty_output><jupyter_text>ConflictsNow that we've seen merging in action, the next step is to talk about conflicts.  How Are Conflicts Detected?Any merge conflicts can be identified and addressed ahead of running a ``merge``command by using the built in ``diff`` tools. When diffing commits, Hangar willprovide a list of conflicts which it identifies. In general these fall into 4categories:1. **Additions** in both branches which created new keys (samples /   arraysets / metadata) with non-compatible values. For samples &   metadata, the hash of the data is compared, for arraysets, the schema   specification is checked for compatibility in a method custom to the   internal workings of Hangar.2. **Removal** in ``Master Commit/Branch`` **& Mutation** in ``Dev Commit /   Branch``. Applies for samples, arraysets, and metadata   identically.3. **Mutation** in ``Dev Commit/Branch`` **& Removal** in ``Master Commit /   Branch``. Applies for samples, arraysets, and metadata   identically.4. **Mutations** on keys both branches to non-compatible values. For   samples & metadata, the hash of the data is compared, for arraysets, the   schema specification is checked for compatibility in a method custom to the   internal workings of Hangar. Let's make a merge conflictTo force a conflict, we are going to checkout the ``"new"`` branch and set themetadata key ``"hello"`` to the value ``"foo conflict... BOO!"``. If we then tryto merge this into the ``"testbranch"`` branch (which set ``"hello"`` to a valueof ``"world"``) we see how hangar will identify the conflict and halt withoutmaking any changes.Automated conflict resolution will be introduced in a future version of Hangar,for now it is up to the user to manually resolve conflicts by making anynecessary changes in each branch before reattempting a merge operation.<jupyter_code>co = repo.checkout(write=True, branch='new')
co.metadata['hello'] = 'foo conflict... BOO!'
co.commit ('commit on new branch to hello metadata key so we can demonstrate a conflict')
repo.log()<jupyter_output>* 0a0c4dbcfe63ce10fd2a87a98b785ce03099b09e ([1;31mnew[m) : commit on new branch to hello metadata key so we can demonstrate a conflict
* 186f1ccae28ad8f58bcae95dd8c1115a3b0de9dd : commit on `new` branch adding a sample to dummy_arrayset
* 0fd892b7ce9d9d0150c68bb5483876d58c28cbf1 : first commit with a single sample added to a dummy arrayset<jupyter_text>**When we attempt the merge, an exception is thrown telling us there is a conflict!**<jupyter_code>co.merge(message='this merge should not happen', dev_branch='testbranch')<jupyter_output>HANGAR VALUE ERROR:: Merge ABORTED with conflict: {'aset': ConflictRecords(t1=(), t21=(), t22=(), t3=(), conflict=False), 'meta': ConflictRecords(t1=(MetadataRecordKey(meta_name='hello'),), t21=(), t22=(), t3=(), conflict=True), 'sample': {'dummy_arrayset': ConflictRecords(t1=(), t21=(), t22=(), t3=(), conflict=False)}, 'conflict_found': True}<jupyter_text>Checking for ConflictsAlternatively, use the diff methods on a checkout to test for conflicts before attempting a merge.<jupyter_code>merge_results, conflicts_found = co.diff.branch('testbranch')
conflicts_found
conflicts_found['meta']<jupyter_output><empty_output><jupyter_text>The type codes for a `ConflictRecords` `namedtuple` such as the one we saw:        ConflictRecords(t1=('hello',), t21=(), t22=(), t3=(), conflict=True)  are as follow:-  ``t1``: Addition of key in master AND dev with different values.-  ``t21``: Removed key in master, mutated value in dev.-  ``t22``: Removed key in dev, mutated value in master.-  ``t3``: Mutated key in both master AND dev to different values.-  ``conflict``: Bool indicating if any type of conflict is present.  To resolve, remove the conflict<jupyter_code>del co.metadata['hello']
co.metadata['resolved'] = 'conflict by removing hello key'
co.commit('commit which removes conflicting metadata key')
co.merge(message='this merge succeeds as it no longer has a conflict', dev_branch='testbranch')<jupyter_output><empty_output><jupyter_text>We can verify that history looks as we would expect via the log!<jupyter_code>repo.log()<jupyter_output>*   b3b097d069f351e5b4688f1ebf30ae1a5aa94f4a ([1;31mnew[m) : this merge succeeds as it no longer has a conflict
[1;31m|[m[1;32m\[m  
* [1;32m|[m 9af80ed5df5d893b5e918f1a060cce4c46d9ddec : commit which removes conflicting metadata key
* [1;32m|[m 0a0c4dbcfe63ce10fd2a87a98b785ce03099b09e : commit on new branch to hello metadata key so we can demonstrate a conflict
[1;32m|[m * 836ba8ff1fe552fb65944e2340b2a2ef2b2b62d4 ([1;31mtestbranch[m) : added hellow world metadata
[1;32m|[m * 2fe5c53a899ba6accbe8c19debd9a489e3baeaed : mutated sample `0` of `dummy_arrayset` to new value
* [1;32m|[m 186f1ccae28ad8f58bcae95dd8c1115a3b0de9dd : commit on `new` branch adding a sample to dummy_arrayset
[1;32m|[m[1;32m/[m  
* 0fd892b7ce9d9d0150c68bb5483876d58c28cbf1 : first commit with a single sample added to a dummy arrayset
StarcoderdataJupyterStructuredCleanDedup