<jupyter_start><jupyter_text>Fast-Free Time-Resolved Electrostatic Force Microscopy (FF-trEFM) <NAME>, Ph.D.  <EMAIL>Department of Chemistry  University of Washington  Box 351700  Seattle, Washington, USA 98195  This notebook walks through the basics of processing FFtrEFM data as acquired from the Gage instrument and processing into a Pycroscopy-compatible H5 file. Note that you can, of course, skip Pycroscopy entirely should you wish, but the Pycroscopy package is more broadly compatible with future updates. It also is convenient in saving your data in a single file; the most time-consuming element of processing FFtrFEM data (or microscopy data in general) is often just loading the data. ------------ Imports. You may need to install pycroscopy and pyUSID. See the instructions here:  https://pycroscopy.github.io/pycroscopy/about.html  https://pycroscopy.github.io/USID/about.html  The import will likely bring up a FutureWarning. This effect is from some outdated functions (VirtualDataGroup) in ffta. As of 7/18/2019 I have caught most of these and updated, but not all of them.<jupyter_code>import ffta
import pycroscopy as px
import pyUSID as usid
import numpy as np
from matplotlib import pyplot as plt<jupyter_output><empty_output><jupyter_text>------------ Source files to processThe .IBW file is your image file. Typically you have a multi-layered IBW with the topography+associated channels saved. The code converts those channels into Pycroscopy. As of now it's hard-coded to work on data from the Ginger lab, but that is easy enough to fix going forward. The FF folder is the folder path containing the raw deflection data. For those unfamiliar, the r' is to accomodate Windows paths.<jupyter_code># input file paths manually here:

# Here we will use some example data to better demonstrate the process
ibw_file = r'E:/Data/20190107 - BAPI trEFM/FF2_128_7V.ibw'
ff_folder = r'E:\Data\20190107 - BAPI trEFM\FF02_128x64_455nm_7V_400mA'<jupyter_output><empty_output><jupyter_text>------------ Start processing!This command will then convert your data into an HDF5 (.h5) file on your local disk. It will be autosaved in the folder with the IBW. This process takes awhile, probably about 5-10 minutes depending upon your computer. The actual process is:* convert the IBW file into a H5 file with separate folders for each image layer.* convert each invididual deflection trace into a Pycroscopy-compatible data file. Save in folder FF_Raw within the structure.* average the deflection traces based on number of signals acquired per pixel (typically 30-60), saved FF_Averaged. In future versions, the averaging will be done in the XOP file itself, so that step will be eliminated. Working off the averaged data is typically faster anyway. The net result is this H5 file has two sets of the data within it, in case some particular deflection traces are brokenIn principle, once this process has completed you no longer need the IBW files as the data are within the .H5 file. You should keep a backup on the associated storage of your choosing, but locally that is safe unless you need to reprocess the data.Some parameters to note:> "verbose = False" to avoid the forty million printed lines.  > "average = True" averages the data based on number of samples per pixel, then stores that data. This process shrinks the H5 considerably, and it speeds up processing time. In a typically oscilloscope, this process is done prior to displaying data.  > "mirror = True" (default). For conventional FF-trEFM, the data are acquired on the retrace, meaning as saved they are backwards from the topography. To make them sensibly line up, this mirror parameter is the default case.<jupyter_code>h5_path, parm_dict, h5_avg = ffta.hdf_utils.load_hdf.loadHDF5_ibw(ibw_file_path=ibw_file, ff_file_path=ff_folder, 
                                                                  verbose=True, average=True)<jupyter_output><empty_output><jupyter_text>------------ Using PCA to clean up the dataHere's an example of why Pycroscopy is useful. Here we use SVD to visualize the principal components. We can then filter all the noisy components when reconstructing the data. The vast majority of data in FFtrEFM tends to be within the first ~10 components, give or take. This SVD is processing h5_avg, which is the "averaged" data set where each pixel is the average of the acquired signals. You rarely need to work off the raw data.Note that the visualization doesn't quite work in Jupyter notebooks, but this works in Spyder or equivalent. The second block below fixes that.<jupyter_code># SVD
h5_svd = ffta.analysis.svd.FF_SVD(h5_avg)
# This block is only necessary in Jupyter notebooks.

# Plot abundance maps correctly
h5_U = h5_svd['U']
h5_V = h5_svd['V']
h5_S = h5_svd['S']

num_rows = parm_dict['num_rows']
num_cols = parm_dict['num_cols']

abun_maps = np.reshape(h5_U[:,:25], (num_rows, num_cols,-1))
eigen_vecs = h5_V[:16, :]

num_abun_rows = 4
num_abun_cols = 4
fig, a = plt.subplots(nrows=num_abun_rows, ncols=num_abun_cols, figsize=(10,10))

tt = np.transpose(abun_maps, (2, 0, 1))

# The pycroscopy plot wrapper doesn't work in Jupyter
for r in range(num_abun_rows):
    for c in range(num_abun_cols):
        
        a[r][c].imshow(tt[:][:][r*num_abun_rows+c], cmap='inferno')
# Rebuild
clean_components = [0,1,2,3,4] # change based on SVD

h5_rb = ffta.analysis.svd.FF_SVD_filter(h5_avg, clean_components)<jupyter_output>Reconstructing in batches of 1134 positions.
Batchs should be 346.0909652709961 Mb each.
Completed reconstruction of data from SVD results.  Writing to file.
Done writing reconstructed data to file.<jupyter_text>------------ Actual FF processing!At this point we have:      an H5 file with the raw and averaged data.      a processed set of the deflection data from PCA      So, we now use the ffta package to reconstruct the image. The live imaging below will not work in Jupyter. But, nonetheless this function is operating. It returns 4 new variables as images (in (N, M) shape, rather than Pycroscopy form of (N x M, 2):        tfp : the time-to-first-peak data, in time. This is usually the image of interest      shift : the frequency shift data, which is mostly equivalent to standard EFM      inst_freq : the actual instantaneous frequency data at each pixel.     h5_if : the Pycroscopy H5 reference to the processed image file.<jupyter_code># The actual FF-trEFM processing. 
# Note that "live" plotting does not currently work in Jupyter...
tfp_rb, shift_rb, inst_freq_rb, h5_if = ffta.hdf_utils.analyze_h5.process(h5_svd.file, ref=h5_rb.name)
# Save to CSV files
ffta.hdf_utils.analyze_h5.save_CSV_from_file(h5_if.file, h5_if.parent.name)
# You should make sure to close the file explicitly to preserve the data
h5_svd.file.close()<jupyter_output><empty_output>
StarcoderdataJupyterStructuredCleanDedup