code:
% Open Letter to the Free Software Community
%
% 2 Apr 2017

To the free software community:

Over the past six months, the Libreboot project has been in a state of discord.
After an issue with a transgender employee at the FSF escalated, Libreboot
publicly left GNU with little consultation from the community. Relations with
so many people were strained. Friendships broken, lines of code never written:
the chaos needs to come to an end.

A series of organisational issues with Libreboot enabled this fallout to occur.
We have since corrected them:

Previously, the libreboot repository and the website could only be modified by
the project leader, Leah Rowe. This setup created a single point of failure,
with little leeway for dissenting contributors. Since then, I have joined the
project as the sysadmin. Along with another contributor, Sebastian "Swift Geek"
Grzywna, direct access to the code and servers is shared. Though the project
cannot yet be completely decentralised, this change is a win for transparency.

Previously, most of libreboot.org, including public statements such as those
regarding GNU, were issued by Leah herself. The rest of the team and the
community were not consulted. As Damien Zammit, a former contributor noted, the
word "we" on old Libreboot notices meant "Leah". But alas, there is no room for
the "royal we" in democracy.

Finally, on a personal note, Leah was at the time struggling with gender
dysphoria and substance abuse. Since then, she has been managing these issues.
She agrees that her behaviour was rash and is determined to find a unifying
solution.

With all of this in mind, were the allegations against the Free Software
Foundation true? Perhaps. Perhaps not. At this point, it doesn't matter.
Indeed, it is unlikely that Libreboot will ever rejoin GNU, but feuding in an
already fragmented community helps nobody. The world of free software is
shrinking and under attack. Though the FSF may make mistakes from time to time,
so do we. We do not need another divide.

Similarly, we would like to work closely with librecore, another project
working to deblob coreboot, whose team includes Damien Zammit. Once librecore
matures, libreboot plans to switch to librecore in lieu of our current
deblobbed fork. That is, there will be less duplication of effort; instead of
competing with librecore, libreboot will complement it.

As a technical update, we are currently working on a Libreboot port to
the X220. Leah and Swift are investigating ways to disable the ME on
Sandybridge hardware, which potentially means more modern Intel hardware may be
supported. Additionally, Paul Kocialkowski has been working on supporting
several new Chromebooks with ARM chips; these ports will also be available in
an upcoming release.

No more "royal we". No more notorious surprises. No more late night
"typofixes".

Transparency and collaboration are the way forward.

~Alyssa Rosenzweig

### A message from Leah, to further complement Alyssa's submission:

I acknowledge that what I did was wrong. Although I felt justified at the time,
I ended up hurting a lot of people, most of whom were uninvolved with any of
the relevant events. At the time of taking Libreboot out of GNU, I was going
through intense personal difficulty in my life, and I was highly unstable.
Presented with a similiar situation, I likely won't respond the way I did
before. I've changed a lot, and I promise this type of thing won't happen
again.

What I'd like the most, is to simply focus on Libreboot as I've always done,
and to forget about what happened in the past and move on.

I sincerely hope that the FSF, GNU project and others who I have hurt, can do
the same. My only goal at present is to continue improving Libreboot, and to do
everything in my power to make libre hardware a reality for everyone, as is the
goal of the Libreboot project.

I will remain in my place as a developer in this project, but everything that I
do from now on will be the result of open collaboration with others in the
project. I very nearly single handedly destroyed this project, and caused a lot
of damage to the entire community, damage which I now wish to repair. I love
Libreboot, Free Software and the community, and my only wish is to see it
thrive.

I wish well any and all outreach organisations that do work with the FSF; e.g.
Outreachy, who are working to do what they can to improve conditions for women,
LGBT people and other minorities in the sphere of computing and engineering
fields, related to Free Software.

I hope that any damage I caused to the community is not permanent.

I'm deeply sorry for the damage and upset that I caused. I will not repeat the
same mistakes twice. To a very specific person, who cannot be named in the
interest of their privacy, but for whom context will allow that person
to know this is for them: I'm especially sorry for the damage that I did to
you during those months, and I sincerely hope that you get what you are
currently seeking. I fully support you in your endeavour, and if I could help
you, I would. You still mean a great deal to me.

I'm especially sorry to John Sullivan and Ruben Rodriguez of the FSF, both of
whom I publicly slandered on the Libreboot website. They did not deserve any of
that. I was being highly abusive towards you both.

I would also like to express my sincerest apology to those GNU maintainers
whose emails I published on the Libreboot website, when all of that chaos was
underway.

I'm sorry to all the other Libreboot maintainers at the time, some of whom felt
like they had no choice but to quit the project because of my actions. Doubly
so for Damien Zammit and Timothy Pearson, both of whom I let down in various
ways afterwards aswell.

I would like to sincerely apologize to Timothy Pearson of Raptor Engineering
in particular, for the damage that I caused with Libreboot's exit from GNU,
while you were running your TALOS workstation campaign. Although I do not know
whether the campaign would have succeeded without all that chaos, my own
actions certainly did not help.

Finally, I would like to once again apologize to the community as a whole, for
the damage that I caused. I hope that I can prove to you all that I do wish to
make amends for the damage, and to improve, and to re-build bridges with the
community, and to once again work with the community in bringing Libreboot and
Free Software forward, into the future.

~Leah Rowe


code:
#!/bin/bash
#
# Copyright (C) 2017 Alyssa Rosenzweig <alyssa@rosenzweig.io>
# Copyright (C) 2017 Leah Rowe <info@minifree.org>
#
# This program is free software: you can redistribute it and/or modify
# it under the terms of the GNU General Public License as published by
# the Free Software Foundation, either version 3 of the License, or
# (at your option) any later version.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program.  If not, see <http://www.gnu.org/licenses/>.

[ "x${DEBUG+set}" = 'xset' ] && set -v
set -e

echo $1
FILE=${1%.md}

cat $1 > temp.md

OPTS=

if [ "${FILE}" != "./index" ]; then
        if [[ $FILE == *index ]]
        then
            DEST="../"
        else
            DEST="./"
        fi

        RETURN="Back to previous index"
        OPTS="--css /headerleft.css -T Libreboot"
fi

echo "" >> temp.md
printf "[License](/license.md) --\n" >> temp.md
printf "[Authors](/contrib.md) --\n" >> temp.md
printf "[Conduct Guidelines](/conduct.md) --\n" >> temp.md
printf "[Management Guidelines](/management.md) \n" >> temp.md

# change out .md -> .html
sed temp.md -i -e 's/\.md\(#[a-z\-]*\)*)/.html\1)/g'
sed temp.md -i -e 's/\.md\(#[a-z\-]*\)*]/.html\1]/g'

# work around issue #2872
TOC=$(grep -q "^x-toc-enable: true$" temp.md && echo "--toc --toc-depth=2") || TOC=""

# work around heterogenous pandoc versions
SMART=$(pandoc -v | grep -q '2\.0' || echo "--smart") || SMART=""

# chuck through pandoc
pandoc $TOC $SMART temp.md -s --css /global.css $OPTS \
        --template template.html --metadata return="$RETURN"> $FILE.html

# additionally, produce bare file for RSS
pandoc $1 > $FILE.bare.html

license: mit


code:
---
title: Rsync repositories for the release archives
...

Create a directory in your web directory (e.g. libreboot/) and put one
of these in your crontab:

Libreboot project, UK (main rsync mirror)
-----------------------------------------

**rsync -avxP --delete --stats rsync.libreboot.org::mirrormirror
/path/to/docroot/libreboot/**

University of Kent, UK (backup mirror)
--------------------------------------

**rsync -avxP --delete --stats
rsync://rsync.mirrorservice.org/libreboot.org/release/
/path/to/docroot/libreboot/**

Princeton university, USA (backup mirror)
-----------------------------------------

**rsync -avxP --delete --stats
rsync://mirror.math.princeton.edu/pub/libreboot/
/path/to/docroot/libreboot/**

linux.ro, Romania (backup mirror)
---------------------------------

**rsync -avxP --delete --stats rsync://ftp.linux.ro/libreboot/
/path/to/docroot/libreboot/**

partyvan.eu, Sweden (backup mirror)
-----------------------------------

**rsync -avxP --delete --stats
rsync://mirror.se.partyvan.eu/pub/libreboot/
/path/to/docroot/libreboot/**

mirror.si, Slovenia (backup mirror)
-----------------------------------

**rsync -avxP --delete --stats rsync://libreboot.mirror.si/libreboot
/path/to/docroot/libreboot/**

Are you running a mirror? Contact the libreboot project, and the link will be
added to the [download](download.md) page.

Libreboot includes statically linked executables. If you need the
sources for those statically linked dependencies inside the executables,
then you can contact the libreboot project using the details on the home
page; source code will be provided. You can download this source code
from [here](ccsource.md).


code:
---
title: Tasks for the next stable version release of Libreboot
...

Libreboot now has an issue tracker, at
<https://notabug.org/libreboot/libreboot/issues>. The purpose of this page is
to list current high priority tasks, which are desirable and/or required for
the next release.

Libreboot releases happen whenever Libreboot is in a state that the maintainers
agree is suitable to be considered for making a new stable release. This is to
be determined by overall consensus, per release.

This page is updated periodically, to reflect the issues currently open on the
bug tracker.

Port x86 to the Paper build system
----------------------------------

<https://notabug.org/libreboot/libreboot/issues/108>

Paper is the new build system in Libreboot, feature-complete for ARM and with
many improvements, but the old build system is still used for x86 Intel and AMD
systems in Libreboot.

Paper has more features, is more modular, is better designed and superior in
terms of maintainability.

Dump coreboot as firmware upstream. Adopt librecore instead
-----------------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/44>

It may be necessary to keep coreboot, for the ARM chromebooks (rk3288 chipset).

Libreboot is a coreboot distribution, which integrates payloads, utilities,
documentation, additional patches/fixes and integrates everything together.
You can think of it like a GNU+Linux distribution, but it's a distribution of
the boot firmware, not the operating system. Libreboot incorporates coreboot,
deblobbing it with deblob scripts.

Librecore is a whole new fork of coreboot, and it is its own upstream, with
many coreboot developers working on it. It deblobs coreboot in-tree, without
deblob scripts. The only blobs that Librecore distributes is CPU microcode
updates, and we can simply exclude those in Libreboot, just as the current
build system deletes them after it downloads coreboot.

Librecore generally has a much more focused team, and their priority is libre
hardware. It is a very suitable upstream for Libreboot, much more so than the
coreboot project. Librecore is a very new project (formed in December 2016).

See: <http://librecore.info/>
IRC channel (on freenode): #librecore

Its main developers are damo22, avph, funfunctor and others from the #coreboot
IRC channel. Of interest: damo22 and avph used to work for Libreboot, before
forming the Librecore project.

Current board fixes in Libreboot's git repository, not in the current stable
release of Libreboot:

-   gm45: use correct vbt string (needed for external monitors)
-   i945: add fake vbt (needed for external screens)
-   ga-g41m-es2l: enable use of PCIe x16 port
-   i945: add support for external monitors in grub
-   add vesafb mode to ga-g41m-es2l
-   gm45: add support for external monitors in grub
-   Fix low-res screens on T60
-   add hda verb to t60 (fixes mic)
-   gm45: fix IOMMU

There may be others. The above patches are most likely already merged in
Librecore's main Git repository. This should be investigated. If so, then they
can be deleted from Libreboot since they would already be merged upstream.

Reproducible builds
-------------------

<https://notabug.org/libreboot/libreboot/issues/15>

NOTE: Integrate librecore first.

NOTE: Finish Paper integration first.

ThinkPad X220 support
---------------------

<https://notabug.org/libreboot/libreboot/issues/48>

NOTE: Merge after Librecore is adopted in Libreboot.

ThinkPad T400S support
----------------------

<https://notabug.org/libreboot/libreboot/issues/7>

NOTE: Merge after Librecore is adopted in Libreboot.

Lenovo G505S support
--------------------

<https://notabug.org/libreboot/libreboot/issues/104>

NOTE: Merge after Librecore is adopted in Libreboot.

ThinkPad W500 support
---------------------

<https://notabug.org/libreboot/libreboot/issues/87>

Merge this upstream in coreboot and librecore, and port downstream to
Libreboot

NOTE: Merge after Librecore is adopted in Libreboot.

ThinkPad R500 support
---------------------

<https://notabug.org/libreboot/libreboot/issues/107>

Merge this upstream in coreboot and librecore, and port downstream to
Libreboot.

NOTE: Merge after Librecore is adopted in Libreboot.

Fix broken reboot on Intel D510MO
---------------------------------

<https://notabug.org/libreboot/libreboot/issues/103>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Fedora doesn't boot on Libreboot systems. Fix that
--------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/102>

This is a Fedora issue.

TODO: Consider closing this bug (report it to Fedora instead).
NOTE: This is most likely E\_NOT\_OUR\_BUG.

Add support for various i945 desktops
-------------------------------------

<https://notabug.org/libreboot/libreboot/issues/45>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Adopt Unofficial BASH Strict Mode in the entire build system (Paper)
--------------------------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/84>

NOTE: Merge x86 into Paper build system first.

Investigate GPU hanging issue on GM45 laptops
---------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/62>

Fix this in Librecore, if possible. Otherwise, submit bug report (regression)
on Linux kernel bug tracker.

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Fix or work around for issues with dock drive hotplug detection on Linux 4.10.x
-------------------------------------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/61>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Fix Parabola build issue for cbfstool in coreboot
-------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/59>

This may already be fixed in Librecore, our future upstream.

NOTE: This may affect any GNU+Linux distribution which uses a modern toolchain.

Disable framebuffer backdoor on SandyBridge (for ThinkPad X220)
---------------------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/58>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Add SeaBIOS payload option for all x86 boards in Libreboot
----------------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/55>

Fix boot issue for Debian and other systems (and fix FDE in grub.cfg)
---------------------------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/54>

i945 and GM45: Support fine tuning BLC\_PWM\_CTL register value in CBFS
-----------------------------------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/50>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Add support for low-power cstates on MacBook2,1
-----------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/49>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Implement support for T400 docking station
------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/46>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.

Bug: KGPE-D16 stuck at: calibrating apic timer
----------------------------------------------

<https://notabug.org/libreboot/libreboot/issues/39>

NOTE: Investigate this in Librecore. Libreboot is abandoning coreboot.


code:
<!DOCTYPE html>

  <meta charset="utf-8">
  <meta name="generator" content="pandoc">
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes">
$for(author-meta)$
  <meta name="author" content="$author-meta$">
$endfor$
$if(date-meta)$
  <meta name="dcterms.date" content="$date-meta$">
$endif$
$if(keywords)$
  <meta name="keywords" content="$for(keywords)$$keywords$$sep$, $endfor$">
$endif$
  <title>$if(title-prefix)$$title-prefix$ – $endif$$pagetitle$</title>

$if(quotes)$

$endif$
$if(highlighting-css)$

$endif$
$for(css)$
  <link rel="stylesheet" href="$css$">
$endfor$
$if(math)$
  $math$
$endif$
$for(header-includes)$
  $header-includes$
$endfor$
<link rel="alternate" type="application/rss+xml" title="RSS Feed" href="/feed.xml"/>

$for(include-before)$
$include-before$
$endfor$
$if(title)$

$title$

$if(subtitle)$

$subtitle$

$endif$
$for(author)$

$author$

$endfor$
$if(date)$

$date$

$endif$

$endif$
$if(return)$
$return$
$endif$
$if(toc)$

$toc$

$endif$
$body$
$for(include-after)$
$include-after$
$endfor$


code:
# -*- mode: org; fill-column: 80; -*-
#+TITLE: Lambda Calculus
#+AUTHOR: Zelphir Kaltstahl
#+STARTUP: content indent align inlineimages hideblocks entitiesplain nologdone nologreschedule nologredeadline nologrefile
#+KEYWORDS: lambda calculus notes
#+TODO: TODO INPROGRESS | DONE CANCELLED
#+DATE: [2022-10-14 Fr]
#+LANGUAGE: en
#+PRIORITIES: A E E
#+OPTIONS: ^:{}
#+OPTIONS: H:10
#+OPTIONS: toc:2
#+OPTIONS: tags:nil
#+OPTIONS: tasks:nil
#+OPTIONS: H:6
#+OPTIONS: p:nil
#+OPTIONS: pri:nil
#+OPTIONS: prop:nil
#+OPTIONS: todo:nil
#+OPTIONS: stat:nil
#+OPTIONS: |:t
#+OPTIONS: inline:nil

* Lambda
:PROPERTIES:
:CUSTOM_ID: lambda
:END:

** Encoding booleans in lambda calculus
:PROPERTIES:
:CUSTOM_ID: encoding-booleans
:END:

Booleans can be seen as making a choice between 2 alternatives. Imagine an ~if~
expression in any programming language. If checks a condition, which results in
a boolean and depending on that boolean, it makes a choice between the things to
do, if the boolean is ~true~ and the alternative things to do, if the boolean is
~false~. So in principle all the boolean needs to do is allowing to make a
choice between 2 alternatives.

If booleans have this property, then it will be possible to design other boolean
functions like ~not~ on top of them. As long as they return ~true~ or ~false~
again, they create a perfect abstraction layer on top of the representation of
booleans. As long as booleans are the result, no other part of the program needs
to know anything more specific about them. How they are represented is merely an
implementation detail.

One way to encode booleans is the following:

*** true
:PROPERTIES:
:CUSTOM_ID: encoding-booleans-true
:END:

~true~ is encoded by choosing the first of 2 arguments:

#+begin_src lambda
λx.λy.x
#+end_src

*** false
:PROPERTIES:
:CUSTOM_ID: encoding-booleans-false
:END:

~false~ is encoded by choosing the second of 2 arguments:

#+begin_src lambda
λx.λy.y
#+end_src

*** Implementing another boolean function: ~not~
:PROPERTIES:
:CUSTOM_ID: encoding-booleans-other
:END:

~not~ applies the choice of ~b~ to ~false~ and ~true~:

#+begin_src lambda
  λb.b false true
= λb.b (λx.λy.y) (λx.λy.x)
= NOT

  NOT true
= (λb.b (λx.λy.y) (λx.λy.x)) true
= (λb.b (λx.λy.y) (λx.λy.x)) (λx.λy.x)
= (λx.λy.x) (λx.λy.y) (λx.λy.x)
= λx is (λx.λy.y) . λy is (λx.λy.x) . x
= (λx.λy.y)
= λx.λy.y
= false
#+end_src

In the case of ~b~ being ~false~, this will result in the second argument being
chose, which is ~true~. In the case of ~b~ being ~true~, the first argument will
be chosen, which is ~false~. ~not~ returns our representation of ~true~ and
~false~!

** Function application ("β-reduction")
:PROPERTIES:
:CUSTOM_ID: function-application
:END:

Function application is written by putting a function and its argument next to
each other (juxtapositioning). For example:

#+begin_src lambda
λa.a b
#+end_src

In the expression above the function ~λa.a~, the identity function, is applied
to its argument ~b~.

** Function composition
:PROPERTIES:
:CUSTOM_ID: function-composition
:END:

Function composition means taking 2 or more functions and building a new
function, which applies them in an appropriate order. The newly built function
is the composed function.

Function composition works by taking functions as arguments and then
juxtapositioning them in the function body:

#+begin_src lambda
compose := λf.λg.λx.f (g x)
#+end_src

repo_name: ZelphirKaltstahl/lambda

license: unknown


code:
# -*- mode: org; fill-column: 80; -*-
#+TITLE: Lambda Calculus Notes
#+AUTHOR: Zelphir Kaltstahl
#+STARTUP: content indent align inlineimages hideblocks entitiesplain nologdone nologreschedule nologredeadline nologrefile
#+KEYWORDS: lambda calculus notes
#+TODO: TODO INPROGRESS | DONE CANCELLED
#+DATE: [2022-10-14 Fr]
#+LANGUAGE: en
#+PRIORITIES: A E E
#+OPTIONS: ^:{}
#+OPTIONS: H:10
#+OPTIONS: toc:2
#+OPTIONS: tags:nil
#+OPTIONS: tasks:nil
#+OPTIONS: H:6
#+OPTIONS: p:nil
#+OPTIONS: pri:nil
#+OPTIONS: prop:nil
#+OPTIONS: todo:nil
#+OPTIONS: stat:nil
#+OPTIONS: |:t
#+OPTIONS: inline:nil

* About
:PROPERTIES:
:CUSTOM_ID: about
:END:

This repository contains notes about lambda calculus.


code:
 The MIT License (MIT)

Copyright © 2022 Stephen Stengel <stephen.stengel@cwu.edu>

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the “Software”), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

repo_name: stephenstengel/my

path: LICENCE


code:
This project is my implementation of u-net in Keras.

My goal is to use it for DIBCO image binarization research and also as a stress tester.

I will start by making it run on my hardware under a conda environment and using PlaidML. Later I will make a docker image so that I can try it on other computers more easily.

Stephen Stengel <stephen.stengel@cwu.edu> 2022

path: README


code:
TODO

Helper functions.

use this to map the image cutting directly into the model: https://stackoverflow.com/questions/64326029/load-tensorflow-images-and-create-patches

ROC/auc curve stuff

k folds loop wrapped around the whole program basically.

Clean up the code to merge to main

Fix background / foreground accuracy thing. The model currently marks background as true and checks accuracy from that instead of looking at the text.

memory size errors -- Use the skimage library to convert images

Split program into three parts-- gather images, train, and evaluate. Make each able to run independently or through use of a superscript.

use jaccard / dice for training accuracy?

use library to make running on windows easier. \ /

Read more papers.

Test batch size. I notice a difference is small tests, but wtf is it actually. I wouldn't expect this to have any effect at all.

make a baseline network and code to run it. (I've got the network in the baseline form that I want; I just don't have the code to analyze it yet.

confusion matrix

Data augmentation  --  random rotations, overlap the test squares, etc.

Data Augmentation -- my generation idea - check generated data against original and delete if too similar to the original image

roc curve

color pixels -- mask correct / incorrect pixel output in green / red

Try smaller squares  -- Do a square size test after I have baseline model to work with

Adapt square size to text size? Is it possible?

test depth of the u-net

comparisons to other networks of other people.

Add all years

confusion matrix

k folds

Print runtime to a file.

Add ability to start from saved model so I don't take all day to test evry tiny change.

save image slice indexes when cutting up images for stitching back together purposes.

!!! Use scikit-images built in functions to convert image types !!!
https://scikit-image.org/docs/dev/user_guide/data_types.html
"You should never use astype on an image, because it violates these assumptions about the dtype range:"
Converting all my float images to uint8 should give me 4 times more memory to work with on GPU

Use jaccard and dice to estimate veracity.

Add jaccard and dice as functions to use during training.

Add nicer output of the four images for easier viewing.

Remove global variables; replace with function passing? how will that work with loading a model? if loading model, skip the import steps; add command line flag to load model.

Add function to start program using a saved model and saved images.

Invert all images after the final outputs are saved to disk. This way you don't mess up any of the intermediate code.

-- nice later stuff
Make a GUI to run tests easier. There could be radio buttons for each of the valid flags and a switch for starting fom a saved model. The switch could open a separate tab or something. There could be a built in console readout. I've done a bit of python gtk so it shouldn't be too difficult to hack something together.

Use os independent code for the file management. like os.path.join() etc

Investigate training on rectangles?

path: TODO


code:
A small conda cheat sheet for Anaconda and miniconda.

Make an environment:
conda create --name myenv2

Make an environment with some packages installed
conda create --name myenv package1 package2

Make an environment with a particular version of a package:
conda create --name myenv package1 python=3.6

Export an environment to a file:
conda env export > my-environment.yml

Import an environment from a file:
conda env create -f my-environment.yml

Delete an environment:
conda env remove -n myenv

Activate an environment:
conda activate myenv

Deactivate conda:
conda deactivate

List your environments:
conda env list

Command that I used to make my environment:
conda create --name myunet tensorflow keras numpy Pillow tqdm scikit-learn scikit-image matplotlib

Install a different channel:
conda config --add channels newChannel
conda config --add channels conda-forge

Update/Install from a different channel (aka repository)
conda update --channel other-repository mypackage

Stephen Stengel <stephen.stengel@cwu.edu> 2022 MIT LICENSE

path: cheat-sheet-conda


code:
This is the helpfile. It will show helpful stuff when user enders help on the command line.

Stephen Stengel <stephen.stengel@cwu.edu> 2022

path: helpfile


code:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-
#
#  my-unet.py
#
#  Copyright 2022 Stephen Stengel <stephen@cwu.edu> MIT License
#

#OK to start out I'll use 2017 as test data, and 2016 as evaluation data.
#Also I will resize each image to a square of size 128x128; I can add more complex manipulation later.

#using these as reference for some parts:
#https://github.com/dexterfichuk/U-Net-Keras-Implementation

# There is currently a bug in the image manipulation part causing
# occasional index overflow. I might just re-write this whole thing
# in pytorch anyway...

# The graphs of training accuracy show that the accuracy goes down A LOT

# Also, sometimes it just gets stuck marking the whole image as
# background.

print("Running imports...")
import os
import shutil

import numpy as np
import random
from PIL import Image

from tqdm import tqdm

from skimage.io import imread, imshow, imsave
from skimage.transform import resize
from matplotlib import pyplot as plt
from skimage.util import img_as_bool, img_as_float, img_as_uint, img_as_ubyte
from skimage.util import invert
from skimage.color import rgb2gray, gray2rgb, rgb2hsv, hsv2rgb
from sklearn.metrics import auc

import time

import tensorflow as tf
from keras.layers import Conv2D, MaxPool2D, UpSampling2D, Input, Dropout, Lambda, MaxPooling2D, Conv2DTranspose, Concatenate, Softmax
from tensorflow.keras.optimizers import Adam
from keras import Model, callbacks
from keras import backend

# ~ from rezaunet import BCDU_net_D3

## autoinit stuff
# ~ from autoinit import AutoInit

np.random.seed(55555)
random.seed(55555)

NUM_SQUARES = 100 #Reduced number of square inputs for training. 100 seems to be min for ok results.

HACK_SIZE = 64 #64 is reasonably good for prototyping.
GLOBAL_HACK_height, GLOBAL_HACK_width = HACK_SIZE, HACK_SIZE

IMAGE_CHANNELS = 3 #This might change later for different datasets. idk.

GLOBAL_EPOCHS = 15

GLOBAL_BATCH_SIZE = 4 #just needs to be big enough to fill memory
#64hack, 5 epoch, 16batch nearly fills 8gb on laptop. Half of 16 on other laptop.
#Making batch too high seems to cause problems. 32 caused a NaN error when trying to write the output images on laptop1.

GLOBAL_INITIAL_FILTERS = 16

GLOBAL_SMOOTH_JACCARD = 1
GLOBAL_SMOOTH_DICE = 1

IS_GLOBAL_PRINTING_ON = False
# ~ IS_GLOBAL_PRINTING_ON = True

GLOBAL_SQUARE_TEST_SAVE = True
# ~ GLOBAL_SQUARE_TEST_SAVE = False

GLOBAL_MAX_TEST_SQUARE_TO_SAVE = 66

HELPFILE_PATH = os.path.normpath("helpfile")
OUT_TEXT_PATH = os.path.normpath("accuracies-if-error-happens-lol")

print("Done!")

def main(args):
	startTime = time.time()
	print("Hi!")

	checkArgs(args)

	print("Creating folders to store results...")
	tmpFolder, trainingFolder, checkpointFolder, savedModelFolder, \
			predictionsFolder, wholePredictionsFolder, outTextPath \
			= createFolders()
	print("Done!")

	print("Creating copy of source code and conda environment...")
	copySourceEnv(tmpFolder)
	print("Done!")

	trainImagePath = os.path.normpath("../DIBCO/2017/Dataset/")
	trainTruthPath = os.path.normpath("../DIBCO/2017/GT/")
	testImagePath = os.path.normpath("../DIBCO/2016/DIPCO2016_dataset/")
	testTruthPath = os.path.normpath("../DIBCO/2016/DIPCO2016_Dataset_GT/")
	# ~ testImagePath = os.path.normpath("../DIBCO/2017/Dataset/")
	# ~ testTruthPath = os.path.normpath("../DIBCO/2017/GT/")
	# ~ trainImagePath = os.path.normpath("../DIBCO/2016/DIPCO2016_dataset/")
	# ~ trainTruthPath = os.path.normpath("../DIBCO/2016/DIPCO2016_Dataset_GT/")

	print("Creating train and test sets...")
	trainImages, trainTruth, testImages, \
	testTruths, wholeOriginals, wholeTruths = \
			createTrainAndTestSets(trainImagePath, trainTruthPath, testImagePath, testTruthPath)
	print("Done!")

	#Images not currently called from disk. Commenting for speed testing.
	# ~ saveExperimentImages(trainImages, trainTruth, testImages, testTruths, trainingFolder)

	if IS_GLOBAL_PRINTING_ON:
		mainTestPrintOne(wholeOriginals, wholeTruths, trainImages, trainTruth, testImages, testTruths)

	trainImages, trainTruth, testImages, testTruths \
			= reduceInputForTesting(trainImages, trainTruth, testImages, testTruths, NUM_SQUARES)

	theModel, theHistory = trainUnet(trainImages, trainTruth, checkpointFolder)

	print("Saving model...")
	theModel.save(os.path.join(savedModelFolder, "saved-model.h5"))
	print("Done!")

	performEvaluation(theHistory, tmpFolder, testImages, testTruths, theModel)

	print("shape of testImages right before predict: " + str(np.shape(testImages)))
	modelOut = theModel.predict(testImages)
	binarizedOut = ((modelOut > 0.5).astype(np.uint8) * 255).astype(np.uint8) #######test this thing more

	if GLOBAL_SQUARE_TEST_SAVE:
		saveTestSquares(
				GLOBAL_MAX_TEST_SQUARE_TO_SAVE, modelOut, \
				binarizedOut, testImages, testTruths, predictionsFolder)
	else:
		print("Not saving test square pictures this time.")

	# ~ print("Calculating jaccard and dice for the test squares...")
	# ~ calculateJaccardDiceTestSquares(testTruths, outTextPath, binarizedOut)

	print("Predicting output of whole images...")
	#currently also does the image processing and saving.
	predictionsList = predictAllWholeImages(wholeOriginals, wholeTruths, theModel, HACK_SIZE)

	print("Creating confusion masks...")
	confusionImages, tpList, fpList, tnList, fnList \
			= createConfusionImageList(predictionsList, wholeOriginals, wholeTruths)
	print("Saving whole image predictions and confusion images...")
	saveAllWholeAndConfusion(predictionsList, wholeOriginals, wholeTruths, confusionImages, wholePredictionsFolder)

	print("Creating ROC graph of the whole images test...")
	createROC(tpList, fpList, tnList, fnList, tmpFolder)

	# ~ print("Evaluating jaccard and dice scores...")
	# ~ evaluatePredictionJaccardDice(predictionsList, wholeTruths, outTextPath)

	print("Done!")

	saveRuntimeToFile(startTime, tmpFolder)

	return 0

def saveRuntimeToFile(startTime, tmpFolder):
	elapsedTime = time.time() - startTime
	with open(os.path.join(tmpFolder, "runtime.txt"), "w") as timeFile:
		timeFile.write(str(round(elapsedTime, 4)) + " seconds")

def createFolders():
	sq = str(NUM_SQUARES)
	hk = str(HACK_SIZE)
	ep = str(GLOBAL_EPOCHS)
	ba = str(GLOBAL_BATCH_SIZE)
	tmpFolder = os.path.normpath("./tmp" + sq + "-" + hk + "-" + ep + "-" + ba + "/")
	trainingFolder = os.path.join(tmpFolder, "trainingstuff")
	checkpointFolder = os.path.join(tmpFolder, "checkpoint")
	savedModelFolder = os.path.join(tmpFolder, "saved-model")
	predictionsFolder = os.path.join(tmpFolder, "predictions")
	wholePredictionsFolder = os.path.join(tmpFolder, "whole-predictions")
	foldersToCreate = [ \
			tmpFolder, trainingFolder, \
			checkpointFolder, savedModelFolder, \
			predictionsFolder, wholePredictionsFolder]
	for folder in foldersToCreate:
		if not os.path.isdir(folder):
			os.makedirs(folder)

	#Lol spaghetti
	global OUT_TEXT_PATH
	OUT_TEXT_PATH = os.path.join(tmpFolder, "accuracy-jaccard-dice.txt")

	return tmpFolder, trainingFolder, checkpointFolder, savedModelFolder, predictionsFolder, wholePredictionsFolder, OUT_TEXT_PATH

def copySourceEnv(tmpFolder):
	try:
		shutil.copy("my-unet.py", tmpFolder)
		shutil.copy("working-conda-config.yml", tmpFolder)
	except:
		print("copy error! Source code and conda environment file not copied!")

	#This would be preferred in the final product.
	# ~ print("Creating current copy of environment...")
	# ~ os.system("conda env export >  " + tmpFolder + "working-conda-config-current.yml")
	# ~ print("Done!")

def reduceInputForTesting(trainImages, trainTruth, testImages, testTruths, sizeOfSet ):
	#This block reduces the input for testing.
	highIndex = len(trainImages)
	if sizeOfSet > highIndex + 1: #Just in case user enters more squares than exist.
		sizeOfSet = highIndex + 1
		print("! Limiting size of squares for training to actual number of squares !")
	print("Number of squares to be used for training: " + str(sizeOfSet))
	updateGlobalNumSquares(sizeOfSet)

	rng = np.random.default_rng(12345)

	pickIndexes = rng.integers(low = 0, high = highIndex, size = sizeOfSet)
	trainImages = trainImages[pickIndexes]
	trainTruth = trainTruth[pickIndexes]

	sizeOfTestSet = sizeOfSet
	if sizeOfTestSet > len(testImages):
		sizeOfTestSet = len(testImages)
	rng = np.random.default_rng(23456)
	print("sizeOfTestSet: " + str(sizeOfTestSet))
	pickIndexes = rng.integers(low = 0, high = len(testImages), size = sizeOfTestSet)
	testImages = testImages[pickIndexes]
	testTruths = testTruths[pickIndexes]

	print("There are " + str(len(trainImages)) + " training images.")
	print("There are " + str(len(testImages)) + " testing images.")

	return trainImages, trainTruth, testImages, testTruths

def mainTestPrintOne(wholeOriginals, wholeTruths, trainImages, trainTruth, testImages, testTruths):
	print("shape of wholeOriginals: " + str(np.shape(wholeOriginals)))
	print("shape of wholeTruths: " + str(np.shape(wholeTruths)))
	print("shape of trainImages: " + str(np.shape(trainImages)))
	print("shape of trainTruth: " + str(np.shape(trainTruth)))
	print("shape of testImages: " + str(np.shape(testImages)))
	print("shape of testTruths: " + str(np.shape(testTruths)))
	print("Showing Training stuff...")
	randomBoy = random.randint(0, len(trainImages) - 1)
	print("image " + str(randomBoy) + "...")
	imshow(trainImages[randomBoy] / 255)
	# ~ plt.show()
	print("truth " + str(randomBoy) + "...")
	imshow(np.squeeze(trainTruth[randomBoy]))
	# ~ plt.show()

	print("Showing Testing stuff...")
	randomBoy = random.randint(0, len(testImages) - 1)
	print("image " + str(randomBoy) + "...")
	imshow(testImages[randomBoy] / 255)
	# ~ plt.show()
	print("truth " + str(randomBoy) + "...")
	imshow(np.squeeze(testTruths[randomBoy]))
	# ~ plt.show()

#save copies of some of the squares used in learning.
def saveTestSquares(numToSave, modelOut, binarizedOut, testImages, testTruths, predictionsFolder):
	print("Saving random sample of figures...")
	rng2 = np.random.default_rng(54322)

	if len(modelOut) < numToSave:
		numToSave = len(modelOut)
	saveIndexes = rng2.integers(low = 0, high = len(modelOut), size = numToSave)

	for i in tqdm(saveIndexes):
		imsave(os.path.join(predictionsFolder, "fig[" + str(i) + "]premask.png"), img_as_ubyte(modelOut[i]))
		imsave(os.path.join(predictionsFolder, "fig[" + str(i) + "]predict.png"), img_as_ubyte(binarizedOut[i]))
		imsave(os.path.join(predictionsFolder, "fig[" + str(i) + "]testimg.png"), img_as_ubyte(testImages[i]))
		imsave(os.path.join(predictionsFolder, "fig[" + str(i) + "]truthim.png"), img_as_ubyte(testTruths[i]))
	print("Done!")

def calculateJaccardDiceTestSquares(testTruths, outTextPath, binarizedOut):
	testTruthsUInt = testTruths.astype(np.uint8)

	#Testing the jaccard and dice functions
	with open(os.path.join(outTextPath), "w") as outFile:
		for i in tqdm(range(len(binarizedOut))):
			jac = jaccardIndex(testTruthsUInt[i], binarizedOut[i])
			dice = diceIndex(testTruthsUInt[i], binarizedOut[i])
			thisString = str(i) + "\tjaccard: " + str(jac) + "\tdice: " + str(dice) + "\n"
			outFile.write(thisString)
	print("Done!")

#currently also does the image processing and saving.
def predictAllWholeImages(wholeOriginals, wholeTruths, theModel, squareSize):
	if IS_GLOBAL_PRINTING_ON:
		print("shape of wholeOriginals: " + str(np.shape(wholeOriginals)))
		for i in range(len(wholeOriginals)):
			print(str(np.shape(wholeOriginals[i])))
		print("##########################################################")
		print("shape of wholeTruths: " + str(np.shape(wholeTruths)))
	predictionsList = []
	for i in tqdm(range(len(wholeOriginals))):
		# ~ wholeOriginals, wholeTruths
		predictedImage = predictWholeImage(wholeOriginals[i], theModel, squareSize)
		if IS_GLOBAL_PRINTING_ON:
			print("Shape of predicted image " + str(i) + ": " + str(np.shape(predictedImage)))
		# ~ predictedImage = ((predictedImage > 0.5).astype(np.uint8) * 255).astype(np.uint8) ## jank thing again
		# ~ print("Shape of predicted image " + str(i) + " after mask: " + str(np.shape(predictedImage)))

		predictionsList.append(predictedImage)

	return predictionsList

#This could be split up a bit as well.
#also outputs the binary masks in lists !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! change later
def createConfusionImageList(predictionsList, wholeOriginals, wholeTruths):
	confusionList = []
	tpList = []
	fpList = []
	tnList = []
	fnList = []
	for i in tqdm(range(len(predictionsList))):
		predictedImage = predictionsList[i]
		truePosMask = createMaskTruePositive(wholeTruths[i], predictedImage)
		tpList.append(truePosMask)
		trueNegMask = createMaskTrueNegative(wholeTruths[i], predictedImage)
		tnList.append(trueNegMask)
		falsePosMask = createMaskFalsePositive(wholeTruths[i], predictedImage)
		fpList.append(falsePosMask)
		falseNegMask = createMaskFalseNegative(wholeTruths[i], predictedImage)
		fnList.append(falseNegMask)

		redColor = [1, 0, 0]
		greenColor = [0, 1, 0]
		blueColor = [0, 0, 1]
		yellowColor = [1, 1, 0]
		truePosColor = colorPredictionWithPredictionMask(truePosMask, predictedImage, greenColor)
		trueNegColor = colorPredictionWithPredictionMask(trueNegMask, predictedImage, yellowColor)
		falsePosColor = colorPredictionWithPredictionMask(falsePosMask, predictedImage, blueColor)
		falseNegColor = colorPredictionWithPredictionMask(falseNegMask, predictedImage, redColor )

		confusion = combinePredictionPicture(truePosMask, trueNegMask, falsePosMask, falseNegMask)
		confusionList.append(confusion)

	return confusionList, tpList, fpList, tnList, fnList

def saveAllWholeAndConfusion(predictionsList, wholeOriginals, wholeTruths, confusions, wholePredictionsFolder):
	for i in tqdm(range(len(predictionsList))):
		imsave(os.path.join(wholePredictionsFolder, "img[" + str(i) + "]predicted.png"), img_as_ubyte(predictionsList[i]))
		imsave(os.path.join(wholePredictionsFolder, "img[" + str(i) + "]truth.png"), img_as_ubyte(wholeTruths[i]))
		imsave(os.path.join(wholePredictionsFolder, "img[" + str(i) + "]original.png"), img_as_ubyte(wholeOriginals[i]))
		imsave(os.path.join(wholePredictionsFolder, "img[" + str(i) + "]confusion.png"), img_as_ubyte(confusions[i]))

def createROC(tpList, fpList, tnList, fnList, tmpFolder):
	#convert each list of masks into a list of percentages.
	tpScores = getPercentTrueFromMaskList(tpList)
	fpScores = getPercentTrueFromMaskList(fpList)

	#not currently needed
	# ~ tnScores = getPercentTrueFromMaskList(tnList[i])
	# ~ fnScores = getPercentTrueFromMaskList(fnList[i])

	plotROCandSave(fpScores, tpScores, tmpFolder)

def plotROCandSave(fpList, tpList, tmpFolder):

	# ~ [x for (y,x) in sorted(zip(Y,X), key=lambda pair: pair[0])]

	fpArray = np.asarray(fpList)
	tpArray = np.asarray(tpList)

	indexesSorted = fpArray.argsort()

	fpArray = fpArray[indexesSorted]
	tpArray = tpArray[indexesSorted]

	print("fpArray: " + str(fpArray))
	print("tpArray: " + str(tpArray))

	# ~ print("fpArray: " + str(fpArray))
	# ~ print("tpArray: " + str(tpArray))

	roc_auc = auc(fpArray, tpArray)
	print("contents of roc_auc: " + str(roc_auc))

	# ~ for thing in roc_auc:
		# ~ print(thing)
	# ~ print("##")

	linewidth = 2

	plt.figure()
	plt.plot(
		fpArray,
		tpArray,
		color="darkorange",
		linewidth = linewidth,
		label="ROC curve (area = %0.2f%%)" % roc_auc,
		# ~ label = "ROC curve"
	)
	plt.plot([0, 1], [0, 1], color="navy", linewidth=linewidth, linestyle="--")
	plt.xlim([0.0, 1.0])
	# ~ plt.ylim([0.0, 1.0])
	plt.ylim([0.0, 1.05])
	plt.xlabel("False Positive Rate")
	plt.ylabel("True Positive Rate")
	plt.title("ROC curve !")
	plt.legend(loc="lower right")
	plt.savefig(os.path.join(tmpFolder, "roc-curve.png"))
	# ~ plt.show()

def getPercentTrueFromMaskList(inList):
	scoreList = []
	for i in tqdm(range(len(inList))):
		scoreList.append( getPercentTrueFromMask(inList[i]))

	return scoreList

######################################################### I actually don't need percent. I neet TPRa and FPR (tp, fp, tn, fn) ##########################
def getPercentTrueFromMask(inMask):
	# ~ mFlat = backend.flatten( img_as_uint(inMask) )
	# ~ numTrue = backend.sum(mFlat)
	mFlat = np.asarray(inMask).flatten()
	numTrue = np.sum(mFlat)

	# ~ mFlat = np.asarray(mFlat)
	totalNum = mFlat.size

	return numTrue / totalNum

def evaluatePredictionJaccardDice(predictionsList, wholeTruths, outTextPath):
	print("Calculating jaccard and dice...")
	with open(outTextPath, "w") as outFile:
		for i in tqdm(range(len(predictionsList))):
			thisTruth = np.asarray(wholeTruths[i])
			thisTruth = thisTruth.astype(np.uint8)
			jac = jaccardIndex(thisTruth, predictionsList[i])
			dice = diceIndex(thisTruth, predictionsList[i])
			thisString = str(i) + "\tjaccard: " + str(jac) + "\tdice: " + str(dice) + "\n"
			outFile.write(thisString)
	print("Done!")

def checkArgs(args):
	if len(args) >= 1:
		for a in args:
			if str(a) == "help" \
					or str(a).lower() == "-help" \
					or str(a).lower() == "--help" \
					or str(a).lower() == "--h":
				with open(HELPFILE_PATH, "r") as helpfile:
					for line in helpfile:
						print(line, end = "")
				sys.exit(0)

	if len(args) < 5:
			print("bad input");
			sys.exit(-1)
	else:
		global NUM_SQUARES
		NUM_SQUARES = int(sys.argv[1])
		global HACK_SIZE
		HACK_SIZE = int(sys.argv[2])
		global GLOBAL_HACK_height
		global GLOBAL_HACK_width
		GLOBAL_HACK_height, GLOBAL_HACK_width = HACK_SIZE, HACK_SIZE
		global GLOBAL_EPOCHS
		GLOBAL_EPOCHS = int(sys.argv[3])
		global GLOBAL_BATCH_SIZE
		GLOBAL_BATCH_SIZE = int(sys.argv[4])
		if len(args) >= 6:
			if str(sys.argv[5]) == "print":
				global IS_GLOBAL_PRINTING_ON
				IS_GLOBAL_PRINTING_ON = True
				print("Printing of debugging messages is enabled.")

	if NUM_SQUARES < 100:
		print("100 squares is really the bare minimum to get any meaningful result.")
		sys.exit(-1)
	if HACK_SIZE not in [64, 128, 256, 512]:
		print("Square size must be 64, 128, 256, or 512." \
				+ " 128 is recommended for training. 64 for testing")
		sys.exit(-2)
	if GLOBAL_EPOCHS < 1:
		print("Yeah no.")
		print("You need at least one epoch, silly!")
		sys.exit(-3)
	if GLOBAL_BATCH_SIZE < 1 or GLOBAL_BATCH_SIZE > NUM_SQUARES:
		print("Global batch size should be between 1 and the number" \
				+ " of training squares. Pick a better number.")
		sys.exit(-5)

def updateGlobalNumSquares(newNumSquares):
		global NUM_SQUARES
		NUM_SQUARES = newNumSquares

def performEvaluation(history, tmpFolder, testImages, testTruths, theModel):
	print("Performing evaluation...###############################################################")

	print("Calculating scores...")
	print("len testImages: " + str(len(testImages)))
	scores = theModel.evaluate(testImages, testTruths)
	print("Done!")
	print("Scores object: " + str(scores))

	print(str(history.history))
	print("%s: %.2f%%" % (theModel.metrics_names[1], scores[1]*100))

	print("history...")
	print(history)
	print("history.history...")
	print(history.history)

	accuracy = history.history["acc"]
	jaccInd = history.history["jaccardIndex"]
	diceInd = history.history["diceIndex"]
	val_accuracy = history.history["val_acc"]
	val_jaccInd = history.history["val_jaccardIndex"]
	val_diceInd = history.history["val_diceIndex"]

	loss = history.history["loss"]
	val_loss = history.history["val_loss"]

	epochs = range(1, len(accuracy) + 1)

	#make a loop for this bit?
	saveGraphNumbers(accuracy, epochs, "acc", tmpFolder)
	saveGraphNumbers(jaccInd, epochs, "jaccardIndex", tmpFolder)
	saveGraphNumbers(diceInd, epochs, "diceIndex", tmpFolder)
	saveGraphNumbers(val_accuracy, epochs, "val_acc", tmpFolder)
	saveGraphNumbers(val_jaccInd, epochs, "val_jaccardIndex", tmpFolder)
	saveGraphNumbers(val_diceInd, epochs, "val_diceIndex", tmpFolder)
	saveGraphNumbers(loss, epochs, "loss", tmpFolder)
	saveGraphNumbers(val_loss, epochs, "val_loss", tmpFolder)

	plt.plot(epochs, accuracy, "^", label="Training accuracy")
	plt.plot(epochs, val_accuracy, "2", label="Validation accuracy")
	plt.plot(epochs, jaccInd, "*", label="Jaccard Index")
	plt.plot(epochs, val_jaccInd, "p", label="Validation Jaccard Index")
	plt.plot(epochs, diceInd, "s", label="Dice Index")
	plt.plot(epochs, val_diceInd, "D", label="Validation Dice Index")
	plt.title("Training and validation accuracy")
	plt.legend()
	plt.savefig(os.path.join(tmpFolder, "trainvalacc.png"))
	plt.clf()

	plt.plot(epochs, loss, "^", label="Training loss")
	plt.plot(epochs, val_loss, "2", label="Validation loss")
	plt.title("Training and validation loss")
	plt.legend()
	plt.savefig(os.path.join(tmpFolder, "trainvalloss.png"))
	plt.clf()

def saveGraphNumbers(array, epochs, nameString, tmpFolder):
	outFolder = os.path.join(tmpFolder, "graph-numbers")
	if not os.path.isdir(outFolder):
		os.makedirs(outFolder)

	with open(os.path.join(outFolder, nameString + ".txt"), "w") as statFile:
		statFile.write(nameString + " over " + str(epochs) + "epochs" + "\n")
		for thing in array:
			statFile.write(str(thing) + "\n")

def trainUnet(trainImages, trainTruth, checkpointFolder):

	#print("shape of trainImages: " + str(trainImages.shape))
	standardUnetLol = createStandardUnet()
	# ~ standardUnetLol = BCDU_net_D3( (GLOBAL_HACK_height, GLOBAL_HACK_width, IMAGE_CHANNELS) )
	standardUnetLol.summary()

	checkpointer = callbacks.ModelCheckpoint(
			filepath = checkpointFolder,
			monitor = "val_jaccardIndex",
			save_best_only = True,
			mode = "max")
	earlyStopper = callbacks.EarlyStopping( \
			monitor="val_jaccardIndex", \
			patience = 5, \
			mode = "max", \
			min_delta = 0.01, \
			#restore best weights restores the weights at the epoch
			#with the best results at the end of training.
			restore_best_weights = True)

	callbacks_list = [earlyStopper, checkpointer]

	myHistory = standardUnetLol.fit(
			x = trainImages,
			y = trainTruth,
			epochs = GLOBAL_EPOCHS,
			batch_size = GLOBAL_BATCH_SIZE,
			callbacks = callbacks_list,
			validation_split = 0.33333)

	return standardUnetLol, myHistory

def createStandardUnet():
	input_size=(GLOBAL_HACK_height, GLOBAL_HACK_width, IMAGE_CHANNELS)
	inputs = Input(input_size)
	conv5, conv4, conv3, conv2, conv1 = encode(inputs)
	output = decode(conv5, conv4, conv3, conv2, conv1)
	model = Model(inputs, output)

	# ~ autoinit test. Uncomment to add the autoinit thingy
	# ~ model = AutoInit().initialize_model(model)

	# ~ model.compile(optimizer = Adam(learning_rate=1e-4), loss='categorical_crossentropy',  metrics=["acc"])
	model.compile(
			# ~ optimizer = "adam",
			optimizer = Adam(),

			loss = "binary_crossentropy",
			# ~ loss = jaccardLoss,
			metrics = ["acc", jaccardIndex, diceIndex])

	return model

#dropout increase in middle to reduce runtime in addition to dropping out stuff.
def encode(inputs):
	sfilter = GLOBAL_INITIAL_FILTERS
	conv1 = Conv2D(sfilter, (3, 3), activation = 'relu', padding = "same")(inputs)
	conv1 = Dropout(0.1)(conv1)
	conv1 = Conv2D(sfilter, (3, 3), activation = 'relu', padding = "same")(conv1)
	pool1 = MaxPooling2D((2, 2))(conv1)

	conv2 = Conv2D(sfilter * 2, (3, 3), activation = 'relu', padding = "same")(pool1)
	conv2 = Dropout(0.1)(conv2)
	conv2 = Conv2D(sfilter * 2, (3, 3), activation = 'relu', padding = "same")(conv2)
	pool2 = MaxPooling2D((2, 2))(conv2)

	conv3 = Conv2D(sfilter * 4, (3, 3), activation = 'relu', padding = "same")(pool2)
	conv3 = Dropout(0.2)(conv3)
	conv3 = Conv2D(sfilter * 4, (3, 3), activation = 'relu', padding = "same")(conv3)
	pool3 = MaxPooling2D((2, 2))(conv3)

	conv4 = Conv2D(sfilter * 8, (3, 3), activation = 'relu', padding = "same")(pool3)
	conv4 = Dropout(0.2)(conv4)
	conv4 = Conv2D(sfilter * 8, (3, 3), activation = 'relu', padding = "same")(conv4)
	pool4 = MaxPooling2D((2, 2))(conv4)

	conv5 = Conv2D(sfilter * 16, (3, 3), activation = 'relu', padding = "same")(pool4)
	conv5 = Dropout(0.3)(conv5)
	conv5 = Conv2D(sfilter * 16, (3, 3), activation = 'relu', padding = "same")(conv5)

	return conv5, conv4, conv3, conv2, conv1

def decode(conv5, conv4, conv3, conv2, conv1):
	sfilter = GLOBAL_INITIAL_FILTERS
	up6 = Conv2DTranspose(sfilter * 8, (2, 2), strides = (2, 2), padding = "same")(conv5)
	concat6 = Concatenate()([conv4,up6])
	conv6 = Conv2D(sfilter * 8, (3, 3), activation = 'relu', padding = "same")(concat6)
	conv6 = Dropout(0.2)(conv6)
	conv6 = Conv2D(sfilter * 8, (3, 3), activation = 'relu', padding = "same")(conv6)

	up7 = Conv2DTranspose(sfilter * 4, (2, 2), strides = (2, 2), padding = "same")(conv6)
	concat7 = Concatenate()([conv3,up7])
	conv7 = Conv2D(sfilter * 4, (3, 3), activation = 'relu', padding = "same")(concat7)
	conv7 = Dropout(0.2)(conv7)
	conv7 = Conv2D(sfilter * 4, (3, 3), activation = 'relu', padding = "same")(conv7)

	up8 = Conv2DTranspose(sfilter * 2, (2, 2), strides = (2, 2), padding = "same")(conv7)
	concat8 = Concatenate()([conv2,up8])
	conv8 = Conv2D(sfilter * 2, (3, 3), activation = 'relu', padding = "same")(concat8)
	conv8 = Dropout(0.1)(conv8)
	conv8 = Conv2D(sfilter * 2, (3, 3), activation = 'relu', padding = "same")(conv8)

	up9 = Conv2DTranspose(sfilter, (2, 2), strides = (2, 2), padding = "same")(conv8)
	concat9 = Concatenate()([conv1,up9])
	conv9 = Conv2D(sfilter, (3, 3), activation = 'relu', padding = "same")(concat9)
	conv9 = Dropout(0.1)(conv9)
	conv9 = Conv2D(sfilter, (3, 3), activation = 'relu', padding = "same")(conv9)

	conv10 = Conv2D(1, (1, 1), padding = "same", activation = "sigmoid")(conv9)

	return conv10

def saveExperimentImages(trainImages, trainTruth, testImages, testTruths, tmpFolder):
	if not os.path.exists(tmpFolder):
		print("Making a tmp folder...")
		os.system("mkdir tmp")
		print("Done!")
	np.save(tmpFolder + "train-images-object", trainImages)
	np.save(tmpFolder + "train-truth-object", trainTruth)
	np.save(tmpFolder + "test-images-object", testImages)
	np.save(tmpFolder + "test-truth-object", testTruths)

#gets images from file, manipulates, returns
#currently hardcoded to use 2017 as training data, and 2016 as testing
#data because their names are regular!
def createTrainAndTestSets(trainImagePath, trainTruthPath, testImagePath, testTruthPath):
	trainImageFileNames, trainTruthFileNames, \
		testImageFileNames, testTruthFileNames = getFileNames(trainImagePath, trainTruthPath, testImagePath, testTruthPath)

	trainImages, trainTruth, _, _ = getImageAndTruth(trainImageFileNames, trainTruthFileNames)
	trainTruth = convertImagesToGrayscale(trainTruth)

	testImage, testTruth, wholeOriginals, wholeTruths = getImageAndTruth(testImageFileNames, testTruthFileNames)
	testTruth = convertImagesToGrayscale(testTruth)
	wholeTruths = convertImagesToGrayscaleList(wholeTruths)

	#invert the imported images. Tensorflow counts white as truth
	#and black as false. I had been doing the inverse previously.
	trainImages = invertImagesInArray(trainImages)
	trainTruth = invertImagesInArray(trainTruth)
	testImage = invertImagesInArray(testImage)
	testTruth = invertImagesInArray(testTruth)
	wholeOriginals = invertImagesInArray(wholeOriginals)
	wholeTruths = invertImagesInArray(wholeTruths)

	return trainImages, trainTruth, testImage, testTruth, wholeOriginals, wholeTruths

#Inverts all the images in an array. returns an array.
def invertImagesInArray(imgArray):
	for i in range(len(imgArray)):
		imgArray[i] = invert(imgArray[i])

	return imgArray

#This function gets the source image, cuts it into smaller squares, then
#adds each square to an array for output. The original image squares
#will correspond to the base truth squares.
#Try using a method from here to avoid using lists on the arrays:
#https://stackoverflow.com/questions/50226821/how-to-extend-numpy-arrray
#Also returns a copy of the original uncut images as lists.
def getImageAndTruth(originalFilenames, truthFilenames):
	outOriginals, outTruths = [], []

	wholeOriginals = []
	wholeTruths = []
	print("Importing " + originalFilenames[0] + " and friends...")
	for i in tqdm(range(len(originalFilenames))):
		# ~ print("\rImporting " + originalFilenames[i] + "...", end = "")
		myOriginal = imread(originalFilenames[i])[:, :, :3] #this is pretty arcane. research later
		myTruth = imread(truthFilenames[i])[:, :, :3] #this is pretty arcane. research later

		#save original images as list for returning to main
		thisOriginal = myOriginal ##Test before removing these temp vals.
		thisTruth = myTruth
		wholeOriginals.append(np.asarray(thisOriginal))
		wholeTruths.append(np.asarray(thisTruth))

		#Now make the cuts and save the results to a list. Then later convert list to array.
		originalCuts = cutImageIntoSmallSquares(myOriginal)
		truthCuts = cutImageIntoSmallSquares(myTruth)

		#for loop to add cuts to out lists, or I think I remember a one liner to do it?
		#yes-- list has the .extend() function. it adds the elements of a list to another list.
		outOriginals.extend(originalCuts)
		outTruths.extend(truthCuts)

	#can move to return line later maybe.
	outOriginals, outTruths = np.asarray(outOriginals), np.asarray(outTruths)

	return outOriginals, outTruths, wholeOriginals, wholeTruths

#Cut an image into smaller squares, returns them as a list.
#inspiration from:
#https://stackoverflow.com/questions/5953373/how-to-split-image-into-multiple-pieces-in-python#7051075
#Change to using numpy methods later for much speed-up?:
#https://towardsdatascience.com/efficiently-splitting-an-image-into-tiles-in-python-using-numpy-d1bf0dd7b6f7?gi=2faa21fa5964
#The input is in scikit-image format. It is converted to pillow to crop
#more easily and for saving???. Then converted back for the output list.
#Whitespace is appended to the right and bottom of the image so that the crop will include everything.
def cutImageIntoSmallSquares(skImage):
	skOutList = []
	myImage = Image.fromarray(skImage)
	imageWidth, imageHeight = myImage.size
	tmpW = ((imageWidth // HACK_SIZE) + 1) * HACK_SIZE
	tmpH = ((imageHeight // HACK_SIZE) + 1) * HACK_SIZE
	#Make this next line (0,0,0) once you switch the words to white and background to black.........##############################################################################
	tmpImg = Image.new(myImage.mode, (tmpW, tmpH), (255, 255, 255))
	wHehe, hHehe = myImage.size
	heheHack = (0, 0, wHehe, hHehe)
	# ~ tmpImg.paste(myImage, myImage.getbbox())
	if IS_GLOBAL_PRINTING_ON:
		print("tmpImg.mode: " + str(tmpImg.mode))
		print("tmpImg.getbbox(): " + str(tmpImg.getbbox()))
		print("tmpImg.size: " + str(tmpImg.size))
		print("myImage.mode: " + str(myImage.mode))
		print("myImage.getbbox(): " + str(myImage.getbbox()))
		print("myImage width, height: " + "(" + str(imageWidth) + "," + str(imageHeight) + ")")
		print("myImage.size: " + str(myImage.size))
		print("heheHack: " + str(heheHack))
	tmpImg.paste(myImage, heheHack)
	myImage = tmpImg

	# ~ tmp2 = np.asarray(myImage)
	# ~ imshow(tmp2)
	# ~ plt.show()

	for upper in range(0, imageHeight, HACK_SIZE):
		lower = upper + HACK_SIZE
		for left in range(0, imageWidth, HACK_SIZE):
			right = left + HACK_SIZE
			cropBounds = (left, upper, right, lower)
			cropped = myImage.crop(cropBounds)
			cropped = np.asarray(cropped)
			skOutList.append(cropped)

			# ~ imshow(cropped / 255)
			# ~ plt.show()

	return skOutList

#This function cuts a large input image into little squares, uses the
#trained model to predict the binarization of each, then stitches each
#image back into a whole for output.
def predictWholeImage(inputImage, theModel, squareSize):
	if IS_GLOBAL_PRINTING_ON:
		print("squareSize: " + str(squareSize))
	##get dimensions of the image
	height, width, _ = inputImage.shape
	##get the number of squares per row of the image
	squaresWide = (width // squareSize) + 1
	widthPlusRightBuffer = squaresWide * squareSize
	squaresHigh = (height // squareSize) + 1
	heightPlusBottomBumper = squaresHigh * squareSize

	#Dice the image into bits
	if IS_GLOBAL_PRINTING_ON:
		print("shape of input Image right before dicing: " + str(np.shape(inputImage)))
	# ~ print("input Image right before dicing as string: " + str(inputImage))
	dicedImage = cutImageIntoSmallSquares(inputImage)
	# ~ print("shape of dicedImage right before hacking: " + str(np.shape(dicedImage)))
	# ~ #put output into list with extend then np.asaray the whole list to match elswhere.
	# ~ tmpList = []
	# ~ for i in range(len(dicedImage)):
		# ~ tmpList.extend(dicedImage[i])
	# ~ dicedImage = np.asarray(tmpList)

	##Predict the outputs of each square
	dicedImage = np.asarray(dicedImage)
	# ~ print("shape of dicedImage right before predict: " + str(np.shape(dicedImage)))
	# ~ print("dicedImage right before predict as string: " + str(dicedImage))
	modelOut = theModel.predict(dicedImage)

	##This is the code from main. I know it's bad now, but I'll keep it
	##consistent until I create a helper function for it. ######################################################################################################
	binarizedOuts = ((modelOut > 0.5).astype(np.uint8) * 255).astype(np.uint8)

	#Stitch image using dimensions from above
	#combine each image row into numpy array

	theRowsList = []

	# ~ print("squaresHigh: " + str(squaresHigh))
	# ~ print("squaresWide: " + str(squaresWide))
	# ~ print("squareSize: " + str(squareSize))

	bigOut = np.zeros(shape = (squareSize * squaresHigh, squareSize * squaresWide, 1), dtype = np.uint8) #swap h and w?
	for i in range(squaresHigh):
		for j in range(squaresWide):
			# ~ print("i: " + str(i) + "\tj: " + str(j))
			# ~ print("sqHi: " + str(squaresHigh) + "\tsqWi: " + str(squaresWide))
			thisSquare = binarizedOuts[(i * squaresWide) + j] #w? I got an index out of range here.
			iStart = i * squareSize
			iEnd = (i * squareSize) + squareSize
			jStart = j * squareSize
			jEnd = (j * squareSize) + squareSize
			bigOut[iStart : iEnd , jStart : jEnd ] = thisSquare

	# ~ combined = np.asarray(theRowsList)
	# ~ combined = combined.reshape((64,64,1))

	#Remove the extra padding from the edge of the image.
	outImage = bigOut[ :height, :width]
	# ~ outImage = bigOut

	return outImage

def convertImagesToGrayscale(inputImages):
	outImage = []
	for image in inputImages:
		outImage.append( rgb2gray(image) )

	return np.asarray(outImage)

#Returns a list instead of an np array
def convertImagesToGrayscaleList(inputImages):
	outImage = []
	for image in inputImages:
		outImage.append( np.asarray(rgb2gray(image)) )

	return outImage

#returns the filenames of the images for (trainImage, trainTruth),(testimage, testTruth)
#hardcoded!
#Test is currently hardcodded to 2016
def getFileNames(trainImagePath, trainTruthPath, testImagePath, testTruthPath):
	trainTruthNamePairs = []

	# ~ trainImagePath = os.path.normpath("../DIBCO/2017/Dataset/")
	# ~ trainTruthPath = os.path.normpath("../DIBCO/2017/GT/")
	trainTruthNamePairs.append( (trainImagePath, trainTruthPath) )
	# ~ trainImageFileNames, trainTruthFileNames = \
			# ~ createTrainImageAndTrainTruthFileNames(trainImagePath, trainTruthPath)

	#need to handle non-bmp
	# ~ trainPath = "../DIBCO/2009/DIBC02009_Test_images-handwritten/"
	# ~ gtPath = "../DIBCO/2009/DIBCO2009-GT-Test-images_handwritten/"
	# ~ trainTruthNamePairs.append( (trainPath, gtPath) )
	# ~ trainPath = "../DIBCO/2009/DIBCO2009_Test_images-printed/"
	# ~ gtPath = "../DIBCO/2009/DIBCO2009-GT-Test-images_printed/"
	# ~ trainTruthNamePairs.append( (trainPath, gtPath) )

	#non-bmps
	# ~ trainPath = "../DIBCO/2010/DIBC02010_Test_images/"
	# ~ gtPath = "../DIBCO/2010/DIBC02010_Test_GT/"
	# ~ trainTruthNamePairs.append( (trainPath, gtPath) )

	#2011 needs a special function to split the GTs with a wildcard or something.
	#2012 same

	# ~ trainPath = "../DIBCO/2013/OriginalImages/"
	# ~ gtPath = "../DIBCO/2013/GTimages/"
	# ~ trainTruthNamePairs.append( (trainPath, gtPath) )

	# ~ trainPath = "../DIBCO/2014/original_images/"
	# ~ gtPath = "../DIBCO/2014/gt/"
	# ~ trainTruthNamePairs.append( (trainPath, gtPath) )

	trainImageFileNames = []
	trainTruthFileNames = []

	for pair in trainTruthNamePairs:
		tImPath, gtImPath = pair
		trainNames, gtNames = \
			createTrainImageAndTrainTruthFileNames(tImPath, gtImPath)
		trainImageFileNames.extend(trainNames)
		trainTruthFileNames.extend(gtNames)

	#test image section
	# ~ testImagePath = os.path.normpath("../DIBCO/2016/DIPCO2016_dataset/")
	# ~ testTruthPath = os.path.normpath("../DIBCO/2016/DIPCO2016_Dataset_GT/")
	testImageFileNames, testTruthFileNames = \
			createTrainImageAndTrainTruthFileNames(testImagePath, testTruthPath)

	return trainImageFileNames, trainTruthFileNames, \
			testImageFileNames, testTruthFileNames

def createTrainImageAndTrainTruthFileNames(trainImagePath, trainTruthPath):
	trainImageFileNames = createTrainImageFileNamesList(trainImagePath)
	trainTruthFileNames = createTrainTruthFileNamesList(trainImageFileNames)

	trainImageFileNames = appendBMP(trainImageFileNames)
	# ~ print(trainImageFileNames)
	trainTruthFileNames = appendBMP(trainTruthFileNames)
	# ~ print(trainTruthFileNames)

	trainImageFileNames = prependPath(trainImagePath, trainImageFileNames)
	trainTruthFileNames = prependPath(trainTruthPath, trainTruthFileNames)

	return trainImageFileNames, trainTruthFileNames

def createTrainImageFileNamesList(trainImagePath):
	# ~ trainFileNames = next(os.walk(trainImagePath))[2] #this is a clever hack
	# ~ trainFileNames = [name.replace(".bmp", "") for name in trainFileNames]

	trainFileNames = os.listdir(trainImagePath)

	print(trainFileNames)

	# ~ print("pausing...")
	# ~ a = input()

	return [name.replace(".bmp", "") for name in trainFileNames]

#This makes a list with the same order of the names but with _gt apended.
def createTrainTruthFileNamesList(originalNames):
	return [name + "_gt" for name in originalNames]

def appendBMP(inputList):
	return [name + ".bmp" for name in inputList]

def prependPath(myPath, nameList):
	return [os.path.join(myPath, name) for name in nameList]

#I'm copying the code for jaccard similarity and dice from this MIT licenced source.
#https://github.com/masyagin1998/robin
#jaccard is size intersection of the sets / size union of the sets
#Also, I'm going to try the smoothing values suggested in robin and here:
#https://gist.github.com/wassname/f1452b748efcbeb4cb9b1d059dce6f96
#They also suggest abs()
def jaccardIndex(truth, prediction):
	#they are tensors?! not images?!?!?!?!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
	# ~ truth = img_as_bool(np.asarray(truth)) #fail
	# ~ prediction = img_as_bool(np.asarray(prediction)) #fail

	smooth = GLOBAL_SMOOTH_JACCARD
	predictionFlat = backend.flatten(prediction)
	truthFlat = backend.flatten(truth)

	# ~ for i in range(len(predictionFlat)):
		# ~ if predictionFlat[i] >= 2:
			# ~ print("This was the bug. images have non- binary values...#############################################################################################################################################################")
			# ~ print("predictionFlat[" + str(i) + "]: " + str(predictionFlat[i]))

	# ~ intersectionImg = predictionFlat * truthFlat
	numberPixelsSame = backend.sum(truthFlat * predictionFlat)
	#I've found the function tensorflow.reduce_sum() which performs a sum by reduction
	#Is it better than backend.sum?? ##################################################################
	#the docs say it is equivalent except that numpy will change everything to int64

	return float((numberPixelsSame + smooth) / \
			( \
			(backend.sum(predictionFlat) + backend.sum(truthFlat) - numberPixelsSame + smooth) \
			))

#loss function for use in training.
def jaccardLoss(truth, prediction):
	return  1.0 - jaccardIndex(truth, prediction)

#input must be binarized images consisting of values for pixels of either 1 or 0.
def diceIndex(truth, prediction):
	smooth = GLOBAL_SMOOTH_DICE
	predictionFlat = backend.flatten(prediction)
	truthFlat = backend.flatten(truth)
	numberSamePixels = backend.sum(predictionFlat * truthFlat)

	return float((2 * numberSamePixels + smooth) \
			/ (backend.sum(predictionFlat) + backend.sum(truthFlat) + smooth))

#Loss function for use in training
def diceLoss(truth, prediction):
	smooth = GLOBAL_SMOOTH_DICE
	return smooth - diceIndex(truth, prediction)

#This creates a true positive mask from an inverted image (white pixels are text)
def createMaskTruePositive(truth, prediction):
	pFlat = backend.flatten(prediction)
	pFlat = img_as_bool(pFlat)
	tFlat = backend.flatten(truth)
	tFlat = img_as_bool(tFlat)
	mask = pFlat * tFlat

	return np.reshape(mask, np.shape(prediction))

#creaes true negative mask
def createMaskTrueNegative(truth, prediction):
	pFlat = backend.flatten(prediction)
	pFlat = img_as_bool(pFlat)
	tFlat = backend.flatten(truth)
	tFlat = img_as_bool(tFlat)

	##invert to make the 0 into ones
	pFlat = ~pFlat
	tFlat = ~tFlat

	##then multiply
	mask = pFlat * tFlat

	return np.reshape(mask, np.shape(prediction))

#Creates a mask for the False Positives
def createMaskFalsePositive(truth, prediction):
	pFlat = backend.flatten(prediction)
	pFlat = img_as_bool(pFlat)
	tFlat = backend.flatten(truth)
	tFlat = img_as_bool(tFlat)

	#will I need these?
	# ~ falseArray = np.zeros(len(pFlat), dtype = bool)
	# ~ trueArray = np.ones(len(pFlat), dtype = bool)

	##where is the prediction true 1, where the truth is 0 false?
	mask = np.where(pFlat > tFlat, True, False)
	# ~ mask = np.where(pFlat > tFlat, pFlat, ~pFlat)

	return np.reshape(mask, np.shape(prediction))

#returns a mask of all the pixels that are not supposed to be false.
def createMaskFalseNegative(truth, prediction):
	# ~ return createMaskFalsePositive(prediction, truth) #Just swapped the input!?? yes but bug happens. 3-1
	pFlat = backend.flatten(prediction)
	pFlat = img_as_bool(pFlat)
	tFlat = backend.flatten(truth)
	tFlat = img_as_bool(tFlat)

	mask = np.where(pFlat < tFlat, True, False)

	return np.reshape(mask, np.shape(prediction))

#Color the prediction image with the pixels that are correct in red.
def colorPredictionWithPredictionMask(predictionMask, originalPrediction, colorArray):
	prediction = img_as_bool(originalPrediction)
	prediction = np.where( predictionMask >= prediction, True, False ) #This makes the area to paint to white.
	prediction = img_as_float(prediction)

	predictionMask = np.squeeze(predictionMask, axis = 2)
	if IS_GLOBAL_PRINTING_ON:
		print("predictionMask shape: " + str(predictionMask.shape))
	rows, cols = predictionMask.shape

	colorMask = np.zeros((rows, cols, 3))
	colorMask[ predictionMask ] = colorArray

	predictionInColor = np.dstack((prediction, prediction, prediction))

	predictionColor_hsv = rgb2hsv(predictionInColor)
	colorMask_hsv = rgb2hsv(colorMask)

	alpha = 1.0
	predictionColor_hsv[..., 0] = colorMask_hsv[..., 0]
	predictionColor_hsv[..., 1] = colorMask_hsv[..., 1] * alpha

	outImg = hsv2rgb(predictionColor_hsv)

	return img_as_ubyte(outImg)

def combinePredictionPicture(truePosMask, trueNegMask, falsePosMask, falseNegMask):
	redColor = [1, 0, 0]
	greenColor = [0, 1, 0]
	blueColor = [0, 0, 1]
	yellowColor = [1, 1, 0]

	truePosMask = np.squeeze(truePosMask, axis = 2)
	trueNegMask = np.squeeze(trueNegMask, axis = 2)
	falsePosMask = np.squeeze(falsePosMask, axis = 2)
	falseNegMask = np.squeeze(falseNegMask, axis = 2)

	##make a numpy array of ONES, reshape to image size, then convert with imgtofloat
	# ~ print("truePosMask.shape: " + str(truePosMask.shape))
	rows, cols = truePosMask.shape
	# ~ prediction = np.ones( (rows, cols), dtype=bool )
	prediction = np.zeros( (rows, cols), dtype=bool )
	# ~ prediction = img_as_float(prediction)
	predictionRGB = np.dstack((prediction, prediction, prediction))
	# ~ predictionColor_hsv = rgb2hsv(predictionRGB)

	## make the four color masks
	redColorMask = img_as_bool(falseNegMask)
	blueColorMask = img_as_bool(falsePosMask)
	greenColorMask = img_as_bool(truePosMask)
	yellowColorMask = img_as_bool(trueNegMask)

	predictionRGB[redColorMask, 0 ] = 1
	predictionRGB[greenColorMask, 1 ] = 1
	predictionRGB[blueColorMask, 2 ] = 1
	predictionRGB[yellowColorMask, 0 ] = 1
	predictionRGB[yellowColorMask, 1 ] = 1

	# ~ green_image = rgb_image.copy() # Make a copy
	# ~ green_image[:,:,0] = 0
	# ~ green_image[:,:,2] = 0

	# ~ alpha = 1.0
	# ~ predictionColor_hsv[..., 0] = redColorMask[..., 0]
	# ~ predictionColor_hsv[..., 1] = redColorMask[..., 1] * alpha
	# ~ predictionColor_hsv[..., 0] = blueColorMask[..., 0]
	# ~ predictionColor_hsv[..., 1] = blueColorMask[..., 1] * alpha
	# ~ predictionColor_hsv[..., 0] = greenColorMask[..., 0]
	# ~ predictionColor_hsv[..., 1] = greenColorMask[..., 1] * alpha
	# ~ predictionColor_hsv[..., 0] = yellowColorMask[..., 0]
	# ~ predictionColor_hsv[..., 1] = yellowColorMask[..., 1] * alpha
	# ~ outImg = hsv2rgb(predictionColor_hsv)

	# ~ return img_as_ubyte(outImg)
	return img_as_ubyte(predictionRGB)

def makeThisColorMaskHsv(predictionMask, colorArray, rows, cols):
	return rgb2hsv(makeThisColorMaskRGB(predictionMask, colorArray, rows, cols))

def makeThisColorMaskRGB(predictionMask, colorArray, rows, cols):
	thisColorMask = np.zeros((rows, cols, 3))
	thisColorMask[ predictionMask ] = colorArray

	return thisColorMask

if __name__ == '__main__':
	import sys
	sys.exit(main(sys.argv))


code:
Save slice indexes at start when cutting up the image, so you can just
use those later when stitching them together.

Use a batch size of 1 on cpu.

Increase number of epochs when using larger batch size on GPU.

I did a few tests and found that increased batch size reduces accuracy on cpu. So I should increase the number of epochs to compensate on GPU. It's fine for GPU because of parallelization.

imread_collection("*.png") seems to be a simple way to read all in a folder?

https://scikit-image.org/docs/dev/user_guide/getting_started.html  simpler

>>> mask = camera < 87
>>> # Set to "white" (255) the pixels where mask is True
>>> camera[mask] = 255

img.mean() returns the mean value. could be a better binarization metric?

to fill the whole bitfield.
>>> from skimage import exposure
>>> image = exposure.rescale_intensity(img10bit, in_range=(0, 2**10 - 1))
>>> image = exposure.rescale_intensity(img10bit, in_range='uint10')

https://scikit-image.org/docs/dev/user_guide/data_types.html
"You should never use astype on an image, because it violates these assumptions about the dtype range:"
>>> image = exposure.rescale_intensity(img_int32, out_range=(0, 2**31 - 1))
>>> img_uint8 = img_as_ubyte(image)

Using an image from OpenCV with skimage¶

If cv_image is an array of unsigned bytes, skimage will understand it by default. If you prefer working with floating point images, img_as_float() can be used to convert the image:
>>>

>>> from skimage.util import img_as_float
>>> image = img_as_float(any_opencv_image)

Using an image from skimage with OpenCV

The reverse can be achieved with img_as_ubyte():

These can replace my cutImage functions.
np.pad  !!! NOTE: skimage.util.pad is a wrapper for np.pad that appears in ancient code. !!!
skimage.util.view_as_windows(arr_in, window_shape, step=1)

numpy.vstack to take a list of arrays and stack them into one big array!

path: notes


code:
###This code taken from reza azad and lightly changed to fit my project

from keras.models import Model
from keras.layers import Input, concatenate, Conv2D, MaxPooling2D, UpSampling2D, Reshape, core, Dropout

###from keras.optimizers import Adam ### Old way
from tensorflow.keras.optimizers import Adam ## my update to new way

from keras.callbacks import ModelCheckpoint, LearningRateScheduler
from keras import backend as K
from keras.utils.vis_utils import plot_model as plot

##from keras.optimizers import SGD ## old
from tensorflow.keras.optimizers import SGD ## new

from keras.optimizers import *
from keras.layers import *

import numpy as np ## my addition

#input_size is a tupple (h,w,channel)
def BCDU_net_D3(input_size):
	N = input_size[0]
	inputs1 = Input(input_size) ## changed inputs to inputs1 here and everywhere else too
	sfilter = N / 4
	conv1 = Conv2D(sfilter, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(inputs1) ## inputs1
	conv1 = Conv2D(sfilter, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv1)

	pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)
	conv2 = Conv2D(sfilter * 2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool1)
	conv2 = Conv2D(sfilter * 2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv2)
	pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)
	conv3 = Conv2D(sfilter * 4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool2)
	conv3 = Conv2D(sfilter * 4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv3)
	drop3 = Dropout(0.5)(conv3)
	pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)
	# D1
	conv4 = Conv2D(sfilter * 8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(pool3)
	conv4_1 = Conv2D(sfilter * 8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4)
	drop4_1 = Dropout(0.5)(conv4_1)
	# D2
	conv4_2 = Conv2D(sfilter * 8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(drop4_1)
	conv4_2 = Conv2D(sfilter * 8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4_2)
	conv4_2 = Dropout(0.5)(conv4_2)
	# D3
	merge_dense = concatenate([conv4_2,drop4_1], axis = 3)
	conv4_3 = Conv2D(sfilter * 8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge_dense)
	conv4_3 = Conv2D(sfilter * 8, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv4_3)
	drop4_3 = Dropout(0.5)(conv4_3)
	up6 = Conv2DTranspose(sfilter * 4, kernel_size=2, strides=2, padding='same',kernel_initializer = 'he_normal')(drop4_3)
	up6 = BatchNormalization(axis=3)(up6)
	up6 = Activation('relu')(up6)

	x1 = Reshape(target_shape=(1, np.int32(N/4), np.int32(N/4), sfilter * 4))(drop3)
	x2 = Reshape(target_shape=(1, np.int32(N/4), np.int32(N/4), sfilter * 4))(up6)
	merge6  = concatenate([x1,x2], axis = 1)
	merge6 = ConvLSTM2D(filters = sfilter * 2, kernel_size=(3, 3), padding='same', return_sequences = False, go_backwards = True,kernel_initializer = 'he_normal' )(merge6)

	conv6 = Conv2D(sfilter * 4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge6)
	conv6 = Conv2D(sfilter * 4, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv6)

	up7 = Conv2DTranspose(sfilter * 2, kernel_size=2, strides=2, padding='same',kernel_initializer = 'he_normal')(conv6)
	up7 = BatchNormalization(axis=3)(up7)
	up7 = Activation('relu')(up7)

	x1 = Reshape(target_shape=(1, np.int32(N/2), np.int32(N/2), sfilter * 2))(conv2)
	x2 = Reshape(target_shape=(1, np.int32(N/2), np.int32(N/2), sfilter * 2))(up7)
	merge7  = concatenate([x1,x2], axis = 1)
	merge7 = ConvLSTM2D(filters = sfilter, kernel_size=(3, 3), padding='same', return_sequences = False, go_backwards = True,kernel_initializer = 'he_normal' )(merge7)

	conv7 = Conv2D(sfilter * 2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge7)
	conv7 = Conv2D(sfilter * 2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv7)

	up8 = Conv2DTranspose(sfilter, kernel_size=2, strides=2, padding='same',kernel_initializer = 'he_normal')(conv7)
	up8 = BatchNormalization(axis=3)(up8)
	up8 = Activation('relu')(up8)

	x1 = Reshape(target_shape=(1, N, N, sfilter))(conv1)
	x2 = Reshape(target_shape=(1, N, N, sfilter))(up8)
	merge8  = concatenate([x1,x2], axis = 1)
	merge8 = ConvLSTM2D(filters = sfilter / 2, kernel_size=(3, 3), padding='same', return_sequences = False, go_backwards = True,kernel_initializer = 'he_normal' )(merge8)

	conv8 = Conv2D(sfilter, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(merge8)
	conv8 = Conv2D(sfilter, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)
	conv8 = Conv2D(2, 3, activation = 'relu', padding = 'same', kernel_initializer = 'he_normal')(conv8)
	conv9 = Conv2D(1, 1, activation = 'sigmoid')(conv8)

	model = Model(inputs = inputs1, outputs = conv9) ## changed from input ouput to inputs1 outputs
	# ~ model.compile(optimizer = Adam(lr = 1e-4), loss = 'binary_crossentropy', metrics = ["acc", jaccardIndex, diceIndex])
	model.compile(optimizer = Adam(), loss = 'binary_crossentropy', metrics = ["acc", jaccardIndex, diceIndex])

	return model


code:
#!/usr/bin/env python3
# -*- coding: utf-8 -*-

#Just a little script to run all the possible combinations.

import os

#run each of my specifications
sizes = [64, 128, 256, 512]
for thisSize in sizes:
  for batchSize in range(1, 17):
    commandStr = "python3 my-unet.py 50000 " + str(thisSize) + " 100 " + str(batchSize)
    print("Running: " + commandStr)
    os.system(commandStr)


code:
# ---> Python
# Byte-compiled / optimized / DLL files
__pycache__/
*.py[cod]
*$py.class

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
.eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
#  Usually these files are written by a python script from a template
#  before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.coverage.*
.cache
nosetests.xml
coverage.xml
*,cover

# Translations
*.mo
*.pot

# Django stuff:
*.log

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Other
.ztr-directory

path: .gitignore


code:
# urllib imported for connections to URLs.
# json imported for parsing data from ipwhois.app
import urllib
from urllib.request import urlopen
import json

# c2r stands for countries2regions. 2-part array used for matching with user locale.
# Scalable for server distribution based on locale.
c2r = [
    ("US", "us-any-1"), ("PL", "eu-central-1"), ("DE", "eu-central-2"),
    ("BR", "sa-east-1"), ("IL", "us-midwest-1")
]

# User has the option to manually set IP address or use automatic detection.
ip = input("Enter IP Address. Set to 0 for automatic detection: ")

# try-except statement used to let the user know of a possible error
# (API availability or internet connectivity issues).
try:
    # Sending an API request to ipwhois.app
    response = urlopen("http://ipwhois.app/json/"+ip)
    ipgeolocation = json.load(response)
except:
    print('API is unavailable at this time (too many requests?) or failure to establish connection with ipwhois.app.')

# Country code and region output, field "country_code" and "region"
print (ipgeolocation["country_code"] , ipgeolocation["region"])

# x is a temporary variable used to connect the user's current location
# (using country codes) to the nearest server.
# filter function used to eliminate redundant data.
x = list(filter(lambda x: x[0] == ipgeolocation["country_code"], c2r))
print(x)

# If ends up empty, print error message.
# If the country is outside the list after filteration, output a notification.
if not x:
    print("Country out of bounds and unsupported.")


code:
# LocGrab-Connect, py

## Map user to the nearest server based on locale

### Setup
Windows
1. Install python3.5 or newer.
2. Run the file by double-clicking.

### Linux
1. Place the file in a separate folder.
2. Open terminal, change directory to the file directory ( cd ... )
3. Enter python3 `LocMatch.py`

## Development
A sample python script was used from ipwhois.app to begin the development process. To
have user location match with the nearest "server", an array is initialized and the script sends
a request to the API. The user can append an IP address of their liking or use the automatic
mode, where necessary data is retrieved. If the user lives in a country that matches the list
(e.g. the array), the script will print the data normally. Otherwise, a failure notification will be
outputted.\

## Drawbacks to current implementation
1. The array list, while scalable, could become unsustainably large due to the number of
countries within the same region, or just the fact that there are a lot of countries in the
world.
2. The index combo from the array (e.g. ``[('PL', 'eu-central-1')]`` ) is printed even
though the country code, parentheses and apostrophes may appear as redundant for
users.
3. ``try...except..`` could be expanded to show users of possible factors for failure to send
a request to the API.

## Milestones

### Error handling

- ~~Script response to API unavailability~~
- ~~Out-of-bounds response~~

# Sources used
1. AWS (Amazon Web Services) Documentation - Regions and Zones,
https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions
2. IPWHOIS.APP Documentation, https://ipwhois.io/documentation


code:
/build/
/install/
/release/
/sources/
/libreboot.conf
.version
.revision
.tarfiles
.epoch
push
*.vim
/TODO/
/docs/version
/bucts/
/coreboot/
/crossgcc/
/flashrom/
/grub/
/memtest86plus/
/seabios/
/bin/
/resources/utilities/ich9deblob/*.rom
/resources/utilities/ich9deblob/*.bin
/resources/utilities/ich9deblob/ich9deblob
/resources/utilities/ich9deblob/ich9gen
/resources/utilities/ich9deblob/demefactory
/resources/utilities/ich9deblob/obj/*.o
/resources/utilities/ich9deblob/obj/*/*.o
/resources/utilities/ich9deblob/mkdescriptor*.[ch]
/resources/utilities/ich9deblob/mkgbe*.[ch]
/resources/utilities/grub-assemble/grub_*.elf
/mkgbe.[ch]
/*.bin
/resources/utilities/i945-pwm/i945-pwm
/push
/ChangeLog
/NEWS
/version
/versiondate
/gm45macaddress
tocheck*
*.html
!www/template.html
/www/temp.md

repo_name: libreboot/obsolete


code:
Paper Build System
==================

Entry Point
-----------

The entry point for using the paper build system is the main script in the root
directory, named after the build system (paper, libreboot, etc).
Running the main script with no argument will show its general usage.

Configuration
-------------

The paper build system can be configured with a script named after the build
system (paper, libreboot, etc) with a ".conf" suffix. It is automatically loaded
by the build system when running the main script.

It contains environment variables assigned to the desired values. The list of
environment variables is shown in the main script's general usage.

Project and Tool Targets
------------------------

The build system works around the concepts of projects and tools, that define
specific components that can be used to produce either:
* sources
* systems
* images
* tools

Various actions are available for each project and tool targets, many of which
are shown in the main script's general usage. Each project and tool may provide
a usage action that allows to see the specific usage for it. Actions are
executed recursively when no targets are specified.

Each project and tool target has its own directory (either in projects or tools)
that contain a script named after the target and possible a helper script, named
after the script with a "-helper" suffix. The helper script is automatically
included by the build system. Functions in helper scripts are usually prefixed
with the name of the target, where "-" symbols are replaced with "_" symbols.

Each project and tool target's actions are functions defined in the target's
specific script, with names matching the target's actions.

Meta-Targets
------------

Meta targets are project and tool targets that apply the requested action to
individual targets, allowing to execute an action to many targets at once.
For instance, a meta-target named after the build system with a "-all" suffix
would call other meta-targets, prefixed with "-images", "-tools", etc that would
then perform the requested action to all underlying targets.

Projects Actions
----------------

Various generic actions allow preparing projects through a series of steps:
* downloading, extracting or updating the project's sources
* building the project to a build directory
* installing the project to an install directory
* releasing the project to a release directory
* cleaning the build, install and release directories

Actions can be checked by a matching project-specific function, named after the
function to check with a "_check" suffix, to determine whether it is necessary
to run them again to follow the steps. An environment variable can force actions
to be executed, by specifying a space-separated list of projects:
PROJECTS_FORCE.

Projects Configuration and Patches
----------------------------------

Configuration for each project is stored in each project's own directory.
Targets for each project are defined with a "targets" file in each directory
of the "configs" directory. Targets are read recursively, following
sub-directory names for project targets.

Each project's configuration directory can be used to hold target-specific
information and configuration.

An "install" file in each sub-directory indicated which files to grab from the
build files and where to install them in the install directory.

Projects Sources
----------------

Each project can either download specific sources or use another project's
sources, possibly providing its own revision and patches.

Sources are downloaded with the "download" action or can be extracted from
released sources (after placing the sources in the sources directory) with the
"extract" action.

Projects may also keep their sources in a "sources" directory in their
project-specific directory.

Projects Build
--------------

Each project is built to a build directory, named after the project, with the
list of targets separated by a "-" symbol.

An environment variable allows controlling the number of tasks to run at the
same time: TASKS.

Projects Install
----------------

Projects are installed from "install" files in the project directory, that
copies the selected built files to the install directory, named after the
project, with the list of targets separated by a "-" symbol. These install files
are read recursively, following sub-directory names for project targets.

Additional files to install can be specified in the "install" directory and
described in an "install" file. These install files are read recursively,
following sub-directory names for project targets.

Projects Release
----------------

Projects are released to a release directory, named after the project, with the
list of targets separated by a "-" symbol. Each project's install files are
packed into a tarball to the corresponding release directory. A checksum and a
GPG detached signature (if the RELEASE_KEY environment variable is set) are also
generated.

Tarballs are generated reproducibly, from a list of files stored in the
".tarfiles" file. They may also contain the ".epoch", git ".revision" and
".version" files.

Tools Actions
-------------

Tools are used for maintaining the build system, for performing routine tasks
that can be automated. They may have specific actions and not implement any of
the generic actions.

Actions can be checked by a matching tool-specific function, named after the
function to check with a "_check" suffix, to determine whether it is necessary
to run them again to follow the steps. An environment variable can force actions
to be executed, by specifying a space-separated list of tasks: TASKS_FORCE.

Tools Sources
-------------

Tools may keep their sources in a "sources" directory in their tool-specific
directory. These sources may be updated with the "update" action.


code:
Libreboot project

Libreboot documentation is under docs/
The website is https://libreboot.org/
License can be found in COPYING and through parts of the source tree.

Libreboot is a free BIOS or UEFI replacement (free as in freedom); libre boot
firmware that initializes the hardware and starts a bootloader for your
operating system. It's also an open source BIOS, but open source fails to
promote freedom; please call libreboot free software.

Why use libreboot?

Many people use non-free boot firmware, even if they use GNU+Linux. Non-free
BIOS/UEFI firmware often contains backdoors, can be slow and have severe bugs,
where you are left helpless at the mercy of the developers; you have no freedom
over your computing. By contrast, libreboot is building a world where
everyone can use, study, adapt and share software, with true control and
ownership over their technology. In other words, you should use Libreboot for
your freedom's sake!

Libreboot is faster, more secure and more reliable than most non-free firmware,
and can provide many advanced features (such as encrypted /boot/, GPG signature
checking before booting your kernel, ability to load an OS from the flash chip,
and more).

Libreboot's main upstream providers are coreboot (which we deblob, for hardware
initialization), depthcharge (bootloader, and default libreboot payload on ARM),
and GRUB (bootloader, and default libreboot payload on x86). We also
integrate flashrom (for installing libreboot), and several of our own utilities,
scripts and configuration files. All of this is integrated into a single,
coherent package that is easy to use. We add our own patches to the various
upstreams used, and where feasible try to merge upstream as much as possible.

Libreboot provides a fully automated build system and installation process, with
documentation written for non-technical users, in an attempt to make the
software as easy to use as possible. ROM images are provided, along with
utilities, all built from the publicly distributed source code.

This README is licensed under Creative Commons Zero 1.0:
https://creativecommons.org/publicdomain/zero/1.0/


code:
Libreboot 20160907

For existing boards, there are no new board specific changes.

This release adds one new mainboard to libreboot:
* Intel D945GCLF desktop motherboard (thanks to Arthur Heymans)

Other bugfixes:
* Various improvements to the documentation
* re-added "unset superusers" to the grub.cfg, which was needed for some users
  depending on the distros that they used

Libreboot 20160902

This is a bugfix release, based on 20160818. It contains no new board changes.
The previous 20160818 release had build errors in the _src archive, and the
_util archive was only source code.

Changes compared to 20160818:
* Fixed bug where ./build module coreboot always returned non-zero status
* Fixed missing symlink of crossgcc when building from _src (thanks Arthur
Heymans)
* Fixed building with the depthcharge payload (ASUS C201)
* Proper ChangeLog now, instead of pasted git log
* Util archive is now binaries again (source code is in the _src archive)
* Documentation is now in HTML format

Libreboot 20160818

NEW BOARDS ADDED:
* ASUS Chromebook C201 (ARM laptop) (thanks to Paul Kocialkowski)
* Gigabyte GA-G41M-ES2L motherboard (desktop) (thanks to Damien Zammit)
* Intel D510MO motherboard (desktop) (thanks to Damien Zammit)
* ASUS KCMA-D8 motherboard (desktop) (thanks to Timothy Pearson)
* ASUS KFSN4-DRE motherboard (server) (thanks to Timothy Pearson)
* ASUS KGPE-D16 motherboard (server) (thanks to Timothy Pearson)

For boards previously supported, many fixes from upstream have been merged.

Other changes (compared to libreboot 20150518):
(this is a summary. For more detailed change list, refer to the git log)

256MiB VRAM allocated on GM45 (X200, T400, T500, R400) instead of 32MiB.
This is an improvement over both Lenovo BIOS and Libreboot 20150518, allowing
video decoding at 1080p to be smoother. (thanks Arthur Heymans)
To clarify, GM45 video performance in libreboot 20160818 is better than on
the original BIOS and the previous libreboot release.

64MiB VRAM on i945 (X60, T60, MacBook2,1) now supported in coreboot-libre, and
used by default (in the previous release, it was 8MiB allocated). Thanks to
Arthur Heymans.

Higher battery life on GM45 (X200, T400, T500, R400) due to higher cstates now
being supported (thanks Arthur Heymans). C4 power states also supported.

Text mode on GM45 (X200, T400, T500, R400) now works, making it possible to
use MemTest86+ comfortably. (thanks to Nick High from coreboot)

Dual channel LVDS displays on GM45 (T400, T500) are now automatically
detected in coreboot-libre. (thanks Vladimir Serbinenko from coreboot)

Partial fix in coreboot-libre for GRUB display on GM45, for dual channel
LVDS higher resolution LCD panels (T400, T500). (thanks Arthur Heymans)

Massively improved GRUB configuration, making it easier to boot more
encrypted systems automatically, and generally a more useful menu for booting
the system (thanks go to Klemens Nanni of the autoboot project).
Libreboot now uses the grub.cfg provided by the installed GNU+Linux distribution
automatically, if present, switching to that configuration. This is done across
many partitions, where libreboot actively searches for a configuration file
(also on LVM volumes and encrypted volumes). This should make libreboot more
easy to use for non-technical users, without having to modify the GRUB
configuration used in libreboot.

Utilities archives is now source only. You will need to compile the packages
in there (build scripts included, and a script for installing build
dependencies. (binary utility archives are planned again in the next
release, when the new build system is merged)

SeaGRUB is now the default payload on all x86 boards. (SeaBIOS configured to
load a compressed GRUB payload from CBFS immediately, without providing an
interface in SeaBIOS. This way, GRUB is still used but now BIOS services are
available, so you get the best of both worlds). Thanks go to Timothy Pearson
of coreboot for this idea.

crossgcc is now downloaded and built as a separate module to coreboot-libre,
with a universal revision used to build all boards.

Individual boards now have their own coreboot revision and patches, independently
of each other board. This makes maintenance easier.

Updated all utilities, and modules (coreboot, GRUB, etc) to newer versions,
with various bugfixes and improvements upstream.

RTC century byte issue now fixed on GM45 in coreboot-libre, so the date
should now be correctly displayed when running the latest linux kernel, instead
of seeing 1970-01-01 when you boot (thanks to Alexander Couzens from coreboot)

Build system now uses multiple CPU cores when building, speeding up building
for some people. Manually specifying how many cores are needed is also possible,
for those using the build system in a chroot environment. (thanks go to
Timothy Pearson from coreboot)

In the build system (git repository), https:// is now used when cloning coreboot.
http:// is used as a fallback for GRUB, if git:// fails.

New payload, the depthcharge bootloader (free bootloader maintained by Google)
for use on the ASUS Chromebook C201. (thanks go to Paul Kocialkowski)

Various fixes to the ich9gen utility (e.g. flash component density is now
set correctly in the descriptor, gbe-less descriptors now supported)

path: RELEASE