<jupyter_start><jupyter_text>Earth Engine Object DetectionIn this notebook, we'll develop a model to detect cars in 15cm aerial imagery. Part 1: Creating a ModelLets start by importing TensorFlow and the Colab auth library for communication with Google Cloud Storage.<jupyter_code>import tensorflow as tf
from google.colab import auth
auth.authenticate_user()<jupyter_output><empty_output><jupyter_text>Now we'll need to generate training / evaluation data. We'll start by hand annotating the outlines of cars in a roughly 1km^2 region of Mountain View, CA. [We can do this using the geometry editor](https://code.earthengine.google.com/1b573c8d1b3b4bcb9e972eb8994abc4f) in the Earth Engine Code Editor. We can use this annotated data to create a vector mask of cars/non-cars.With the car mask, [we'll generate training and evaluation FeatureCollections and export them to cloud.](https://code.earthengine.google.com/c84a1d9e610ec91044c82766e53fe48a) Lets create a dataset reader in TensorFlow for training/eval data.<jupyter_code># Our input function will return 4 features, each a 'side' x 'side' tensor
# representing the area centered on a pixel with the label 'class'
def input_fn(fileNames, numEpochs=None, shuffle=True, batchSize=100, side=61):
  ds = tf.data.TFRecordDataset(fileNames, compression_type='GZIP')

  feature_columns = {
    'R': tf.FixedLenFeature([side, side], dtype=tf.float32),  
    'G': tf.FixedLenFeature([side, side], dtype=tf.float32),  
    'B': tf.FixedLenFeature([side, side], dtype=tf.float32),    
    'L': tf.FixedLenFeature([side, side], dtype=tf.float32),
    'class': tf.FixedLenFeature([1, 1], dtype=tf.float32)
  }

  def parse(example_proto):
    parsed_features = tf.parse_single_example(example_proto, feature_columns)
    # Separate the class labels from the training features
    labels = parsed_features.pop('class')
    # For faster training / stability, we'll bring our [0, 255] RGBL values into
    # the range [0, 1]
    parsed_features = {
        k:tf.divide(v, 255.0) for (k,v) in parsed_features.items()}
    return parsed_features, labels

  ds = ds.map(parse, num_parallel_calls=5)

  if shuffle:
    # We choose 30 since, with a batch size of 100, we'll keep 3000 (the size
    # of the training data) examples in memory for the shuffle
    ds = ds.shuffle(buffer_size=batchSize * 30)
    
  ds = ds.batch(batchSize).repeat(numEpochs)
  
  iterator = ds.make_one_shot_iterator()
  features, labels = iterator.get_next()
  return features, labels<jupyter_output><empty_output><jupyter_text>Its time to create a model. We'll build a [Fully Convolutional NN](https://people.eecs.berkeley.edu/~jonlong/long_shelhamer_fcn.pdf) so that we can train our model on 61x61 patches, and later apply it to much larger areas for prediction. Note, using a FCNN allows us to make predictions on image data of any dimensions.<jupyter_code># A helper function for defining a convolutional layer. We use batch
# normalization to speed up training given our limited training data, therefore
# we can't use vanilla conv2d(activation='relu', ...)
def conv_layer(inputs, filters, kernel_size, training):
  # Note that the default padding scheme is VALID.
  conv = tf.layers.conv2d(
      inputs=inputs,
      filters=filters,
      kernel_size=kernel_size,
      data_format='channels_last')
  norm = tf.layers.batch_normalization(inputs=conv, training=training)
  return tf.nn.relu(norm)

# Our model will combine convolutions of the full patch on the luminance
# channel with convolutions of the RGB channels on a smaller region of the
# patch. The model will finally scale the predicted 2D region to match the size
# of the input features minus the kernel contributions to the edges.
def fcnn(feat, mode):
  training = mode == tf.estimator.ModeKeys.TRAIN
  # interleave the red, green, and blue channels so that a batch is along axis=0
  rgb = tf.stack([feat['R'], feat['G'], feat['B']], axis=1)
  # Strip a 15 pixel border from the rgb channels. We'll only use the larger
  # area to provide context to the foveated rgb region.
  rgb = rgb[:, :, 15:-15, 15:-15]
  # Convert from NCHW to NHWC
  rgb = tf.transpose(rgb, [0, 2, 3, 1])
  
  # Add a dimension for 'channel' to make this tensor 4D
  l = tf.expand_dims(feat['L'], 3)
  
  # We'll get the size of the original source pixels from l minus the "kernel"
  # surrounding each pixel. We choose to build the meat of our CNN around this
  # reduced region to reduce the model size, training time, etc...
  original_dims=tf.add(tf.shape(l)[1:3], -60)
  
  # Perform 5 convolutions in a row, reducing the information in the luminance 
  # channel to a 25x25 region per-pixel.
  for i in range(6):
    l = conv_layer(inputs=l, filters=3 + i, kernel_size=7, training=training)
       
  rgb = conv_layer(inputs=rgb, filters=8, kernel_size=7, training=training)
  
  # Combine rgb and l to form a 4D tensor with 16 filters
  rgbl = tf.concat([rgb, l], 3)
  
  comb1 = tf.layers.max_pooling2d(
      inputs=rgbl, 
      pool_size=3, 
      strides=2, 
      data_format='channels_last')
  
  comb2 = conv_layer(inputs=comb1, filters=32, kernel_size=5, training=training)
  comb2 = tf.layers.max_pooling2d(
      inputs=comb2, 
      pool_size=3, 
      strides=2, 
      data_format='channels_last')
  
  comb3 = conv_layer(inputs=comb2, filters=64, kernel_size=3, training=training)
  
  # We stay convolutional by substituting a conv op for a dense layer, and
  # keeping the kernel size 1x1.
  dense = conv_layer(
      inputs=comb3,
      filters=64,
      kernel_size=1,
      training=training)
  dropout = tf.layers.dropout(
      inputs=dense,
      rate=0.4, 
      training=training)
  
  # The final layer is just linear activiation; we use the same trick we did
  # with the previous conv layer to produce a single classification.
  dense_final = tf.layers.conv2d(
      inputs=dropout,
      filters=1,
      kernel_size=1,
      data_format='channels_last')
   
  # Squash all predictions into the range (0, 1)
  probs = tf.multiply(tf.add(tf.tanh(dense_final), 1.0), 0.5)
    
  # We won't bother adding the resize op to the graph unless we're running
  # predictions.
  #
  # In a more mature model, we might use a "deconvolution" here by 4x followed
  # by a slight resize to recover a finer amount of detail. Training this way
  # would require larger (in area) training samples so we could give the
  # transposed convolution op something to learn from.
  if mode == tf.estimator.ModeKeys.PREDICT:
    probs = tf.image.resize_images(
        images=probs, 
        size=original_dims)
    
  # Remove the un-needed channel dimension of 1
  probs = tf.squeeze(probs)
  
  # When training/evaluating, 1D tensor of shape [N]. When predicting, 3D tensor
  # of shape [N, H, W]
  return probs<jupyter_output><empty_output><jupyter_text>To facillitate easier training/evaluation/prediction, we'll use TensorFlow's estimator API. We're required todefine a function that the estimator can configure with a mode that will return [estimator specs](https://www.tensorflow.org/api_docs/python/tf/estimator/EstimatorSpec) describing how our modelshould behave depending on the mode.<jupyter_code>def model_fn(features, labels, mode):
  # Whatever mode we're in, we'll always want to generate predictions from the
  # incoming features.
  probs = fcnn(features, mode)
  
  predicted_class = tf.cast(tf.greater(probs, 0.5), tf.float32)

  if mode == tf.estimator.ModeKeys.PREDICT:
    # We reshape the predictions into 1D arrays to make writing prediction data
    # into TFRecord files easier
    #
    # We'll need these prediction labels later when we build TFRecord files
    return tf.estimator.EstimatorSpec(mode=mode, predictions = {
        'class_id': tf.reshape(predicted_class, [-1]),
        'probability': tf.reshape(probs, [-1])
    })

  labels = tf.squeeze(labels)
  # Since we're performing a binary classification, we can use a simple loss
  # function.
  loss = tf.losses.mean_squared_error(labels, probs)

  if mode == tf.estimator.ModeKeys.TRAIN:
    # Adaptive moment estimation has been shown to converge faster than plain
    # old gradient descent in CNNs.
    optimizer = tf.train.AdamOptimizer(learning_rate=0.0001)
    # We need the weight updates to perform the minimization step as batch
    # normalization depends on it
    with tf.control_dependencies(tf.get_collection(tf.GraphKeys.UPDATE_OPS)):
      train_op = optimizer.minimize(
          loss=loss,
          global_step=tf.train.get_global_step())
    
    logging_hook = tf.train.LoggingTensorHook(
        {"batch_predictions" : predicted_class,
        "batch_labels": labels}, 
        every_n_iter=1000)
    return tf.estimator.EstimatorSpec(
        mode=mode, 
        loss=loss, 
        train_op=train_op, 
        training_hooks=[logging_hook])
  
  
  eval_metric_ops = {"accuracy": tf.metrics.accuracy(
      labels=labels, 
      predictions=predicted_class)
  }

  return tf.estimator.EstimatorSpec(
      mode=mode,
      loss=loss,
      eval_metric_ops=eval_metric_ops)<jupyter_output><empty_output><jupyter_text>Now lets create the model object. Don't forget to replace the paths below with the paths to your own GCS bucket / training / evaluation inputs!<jupyter_code>tf.logging.set_verbosity(tf.logging.INFO)
auto_classifier = tf.estimator.Estimator(
    model_fn=model_fn,
    model_dir="gs://cfb-batch-export/eeus18/autoclassifier")<jupyter_output><empty_output><jupyter_text>**And train it!**<jupyter_code># If we want to clear the checkpointed model, we can delete the mode directory 
# to start fresh
# !rm -rf "/autoclassifier"

train_file = 'gs://cfb-batch-export/cars_training.tfrecord.gz'

auto_classifier.train(
    input_fn=lambda: input_fn(fileNames=[train_file]),
    steps=50000)<jupyter_output><empty_output><jupyter_text>And evaluate it! Estimator is awesome!<jupyter_code>eval_file = 'gs://cfb-batch-export/cars_training.tfrecord.gz'

acc = auto_classifier.evaluate(input_fn=lambda: input_fn(
    fileNames=[eval_file], 
    numEpochs=1, 
    batchSize=100,
    shuffle=False))['accuracy']<jupyter_output><empty_output><jupyter_text>Part 2: Creating / Visualizing PredictionsWe'll now need to [export an area on which to perform inference](https://code.earthengine.google.com/3ece5d0b4b2e0f0d4371ba3f5eb5940d).  Note we get a "-mixer.json" with our export which we'll leave alone for now. Be sure to export this image at 15cm/px. We'll define a similar dataset input function as our training / evaluation input function, except we don't carryany class labels in, we'll instead predict these.<jupyter_code># The default value of side is now 316, as our intent is to create predictions
# for 256x256 image patches with a 30 pixel wide border.
def infer_input_fn(fileNames, side=316, batchSize=100):
  ds = tf.data.TFRecordDataset(fileNames, compression_type='GZIP')

  feature_columns = {
    'R': tf.FixedLenFeature([side,side], dtype=tf.float32),  
    'G': tf.FixedLenFeature([side,side], dtype=tf.float32),  
    'B': tf.FixedLenFeature([side,side], dtype=tf.float32),    
    'L': tf.FixedLenFeature([side,side], dtype=tf.float32),
  }

  def parse(example_proto):
    parsed_features = tf.parse_single_example(example_proto, feature_columns)
    parsed_features = {
        k:tf.divide(v, 255.0) for (k,v) in parsed_features.items()}
    return parsed_features
  
  ds = ds.map(parse, num_parallel_calls=5).batch(batchSize)
  
  iterator = ds.make_one_shot_iterator()
  features = iterator.get_next()
  return features<jupyter_output><empty_output><jupyter_text>Lets define a function to take a dictionary of a single patch's predictions and write them to an example. Bywriting examples this way, we'll wind up with an image with 2 bands: 'class_id' and 'probability'<jupyter_code>def make_example(pred_dict):
  class_id = pred_dict['class_id']
  probability = pred_dict['probability']
  return tf.train.Example(
    features=tf.train.Features(
      feature={
        'class_id': tf.train.Feature(
            float_list=tf.train.FloatList(
                value=class_id)),
        'probability': tf.train.Feature(
            float_list=tf.train.FloatList(
                value=probability))
      }
    )
  )<jupyter_output><empty_output><jupyter_text>Don't forget to replace the paths below with the paths to your prediction inputs!<jupyter_code>predict_files = ['gs://cfb-batch-export/cars_inference2-00000.tfrecord.gz',
                 'gs://cfb-batch-export/cars_inference2-00001.tfrecord.gz',
                 'gs://cfb-batch-export/cars_inference2-00002.tfrecord.gz',
                 'gs://cfb-batch-export/cars_inference2-00003.tfrecord.gz',
                 'gs://cfb-batch-export/cars_inference2-00004.tfrecord.gz']<jupyter_output><empty_output><jupyter_text>We're ready to make our predictions. We'll move our predictions into TFRecord files while following a few constraintsso that we can re-ingest these files into Earth Engine. Firstly, we must provide as many predictions as therewere examples in each patch. As each incoming patch has (256+60) x (256+60) examples (pixels), we'llneed our model to produce 256 x 256 labels. Note we ignore the 30 pixel border for ingesting our predictions as this is only context for classifications of the pixels *(we specified 256, 256 as our patch dimensions in Earth Engine, and a kernel of 61, 61)*.To avoid too many large files, we'll keep each file to a minimum of 50 patches of inference labels.<jupyter_code>predictions = auto_classifier.predict(input_fn=lambda: infer_input_fn(
    fileNames=predict_files, 
    batchSize=1,
    side=316),
    yield_single_examples=False)

MAX_RECORDS_PER_FILE = 50
output_path = 'gs://cfb-batch-export/labels/cars_labels-{:05}.tfrecord'

# Create the records we'll ingest into EE
file_number = 0
still_writing = True
total_patches = 0
while still_writing:
  file_path = output_path.format(file_number)
  writer = tf.python_io.TFRecordWriter(file_path)
  print "Writing file: {}".format(file_path)
  try:
    written_records = 0
    while True:
      pred_dict = predictions.next()
      
      writer.write(make_example(pred_dict).SerializeToString())
      
      written_records += 1 
      total_patches += 1
      
      if written_records % 5 == 0:
        print "  Writing patch: {}".format(written_records)
      
      if written_records == MAX_RECORDS_PER_FILE:
        break
  except: 
    # Stop writing for any exception. Note that reaching the end of the prediction
    # dataset throws an exception.
    still_writing=False
  finally:
    file_number += 1
    writer.close()
  
print('Wrote: {} patches.').format(total_patches)<jupyter_output><empty_output><jupyter_text>With our TFRecords in hand, we're ready to ingest them into Earth Engine. Lets get authorized!<jupyter_code>!pip install earthengine-api
!earthengine authenticate --quiet<jupyter_output><empty_output><jupyter_text>Be sure to replace *YOUR AUTH HERE* with your auth code!<jupyter_code>!earthengine authenticate --frontend=http://localhost:8080 --authorization-code=4/UADoP7aQAIrx8pShKZIlhIQXlHpUsBPpTPRJDbX-YZyf9lpJ18ky8yA<jupyter_output><empty_output><jupyter_text>We'll now start the ingestion. If you intend on running this yourself, you'll have to replace `cfb-batch-export` with your cloud bucket and provide your own asset id. We'll also need to pass the mixer file we ignored earlier so Earth Engine knows where our labeled patches came from.<jupyter_code>!earthengine upload image --asset_id=users/cfb/badge gs://cfb-batch-export/test_help/tile3_23-00000.tfrecord gs://cfb-batch-export/test_help/tile3_23-00001.tfrecord gs://cfb-batch-export/test_help/tile3_23-00002.tfrecord gs://cfb-batch-export/test_help/tile3_23-00003.tfrecord  gs://cfb-batch-export/test_help/tile3_23-mixer.json<jupyter_output><empty_output>