<jupyter_start><jupyter_text>IPython Notebook In DepthToday we are going to dive into some of the interesting features of IPython and the IPython notebook, which are useful for a number of daily tasks in data-intensive science.We will work through these features "live"; feel free to type along with me as we go!  Outline- IPython: command-line vs. notebook- Input/Output History- Tab Completion- Getting help and accessing documentation- Useful Keyboard Shortcuts- Magic Commands- Shell commands- Interactivity with ``ipywidgets``  IPython Command Line and Notebook  Launching the IPython ShellIf you have installed IPython correctly, you should be able to type ``ipython`` in your command prompt and see something like this:```IPython 4.0.1 -- An enhanced Interactive Python.?         -> Introduction and overview of IPython's features.%quickref -> Quick reference.help      -> Python's own help system.object?   -> Details about 'object', use 'object??' for extra details.In [1]:```With that, you're ready to follow along.  Launching the IPython NotebookThe IPython notebook is a browser-based graphical interface to the IPython shell, and builds on it a rich set of dynamic display capabilities.As well as executing Python/IPython statements, the notebook allows the user to include formatted text, static and dynamic visualizations, mathematical equations, javascript widgets, and much more.Furthermore, these documents can be saved in a way that lets other people open them and execute the code on their own systems.Though the IPython notebook is viewed and edited through your web browser window, it must connect to a running Python process in order to execute code.This process (known as a "kernel") can be started by running the following command in your system shell:```$ ipython notebook```This command will launch a local web server which will be visible to your browser.It immediately spits out a log showing what it is doing; that log will look something like this:```$ ipython notebook[NotebookApp] Using existing profile dir: '/home/jake/.ipython/profile_default'[NotebookApp] Serving notebooks from local directory: /home/jake/notebooks/[NotebookApp] The IPython Notebook is running at: http://localhost:8888/[NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation)```At the command, your default browser should automatically open and navigate to the listed local URL;the exact address will depend on your system.If the browser does not open automatically, you can open a window and copy this address (here ``http://localhost:8888/``) manually.  Input/Output HistoryA useful feature of IPython is the storage of input and output history.  Terminal OnlyThere are a few useful shortcuts that can be used only in the IPython terminal.We will demonstrate the following in the terminal:- up arrow for history- partial completion with up arrow- reverse search with ctrl-r  Terminal and Notebook- Previous results can be obtained using underscores: number of underscores is the number of the previous command:<jupyter_code>1 + 1
_ * 100
_ + __
_ + __ ** ___<jupyter_output><empty_output><jupyter_text>Beyond three underscores, you can use underscore followed by a number to indicate the result of a particular line number:<jupyter_code>_3<jupyter_output><empty_output><jupyter_text>More useful often is the ``Out`` array, which stores all previous results:<jupyter_code>Out[3]
Out<jupyter_output><empty_output><jupyter_text>Similarly, you can access the ``In`` array to see the code history:<jupyter_code>In[2]
In<jupyter_output><empty_output><jupyter_text>To see all history at once, use the ``%history`` magic command (more on magic commands below):<jupyter_code>%history<jupyter_output>1 + 1
_ * 100
_ + __
_ + __ ** ___
_4
Out[3]
Out
_3
Out[3]
Out
In[2]
In
%history<jupyter_text>Tab CompletionThe feature of IPython that I use the most frequently is perhaps the tab completion functionality.Tab completion works for finishing built-in commands:```pythonIn [20]: import matp```will be completed to ```pythonIn [20]: import matplotlib``` It works for variables that you have defined:<jupyter_code>my_variable = 4<jupyter_output><empty_output><jupyter_text>```pythonIn [22]: my```will be completed to ```pythonIn [22]: my_variable``` This also works for strings which represent filenames, or pandas columns, etc. It works for importing packages: ```pythonIn [25]: import num```will be completed to ```pythonIn [25]: import numpy``` It works for finding attributes of packages and other objects: ```pythonIn [25]: numpy.ran```will be completed to ```pythonIn [25]: numpy.random```  Accessing Help and Documentation After tab completion, I think the next most useful feature of the notebook is the help functionality.One question mark after any valid object gives you access to its documentation string:<jupyter_code>numpy.random?
def myfunc(x):
    return x ** 2
myfunc?<jupyter_output><empty_output><jupyter_text>Two question marks gives you access to its source code (if the object is implemented in Python):<jupyter_code>myfunc??<jupyter_output><empty_output><jupyter_text>In addition, you can use a single question mark with asterisks to do a wildcard match:<jupyter_code>numpy.*exp*?<jupyter_output><empty_output><jupyter_text>If you are curious about the call signature for a funciton, you can type ``shift tab`` within the open-closed parentheses to see its argument list: Hitting ``shift tab`` multiple times will give you progressively more information about the function: Using a combination of these, you can quickly remind yourself of how to use various funcitons without ever leaving the terminal/notebook.  Useful Keyboard ShortcutsOne of the keys to working effectively with IPython is learning your way around the keyboard.Note: some of the shortcuts below will only work on Linux and Mac; many will work on windows as well  Terminal shortcutsIf you are familiar with emacs, vim, and similar tools, many of the terminal-based keyboard shortcuts will feel very familiar to you:  Navigation| Keystroke                     | Action                                     ||-------------------------------|--------------------------------------------|| ``Ctrl-a``                    | Move cursor to the beginning of the line   || ``Ctrl-e``                    | Move cursor to the end of the line         || ``Ctrl-b`` or ``left-arrow``  | Move cursor back one character             || ``Ctrl-f`` or ``right-arrow`` | Move cursor forward one character          |  Text Entry| Keystroke                     | Action                                          ||-------------------------------|-------------------------------------------------|| ``backspace``                 | Delete previous character in line               || ``Ctrl-d``                    | Delete next character in line                   || ``Ctrl-k``                    | Cut text from cursor to end of line             || ``Ctrl-u``                    | Cut all text in line                            || ``Ctrl-y``                    | Yank (i.e. Paste) text which was previously cut || ``Ctrl-t``                    | Transpose (i.e. switch) previous two characters |  Command History| Keystroke                     | Action                                     ||-------------------------------|--------------------------------------------|| ``Ctrl-p`` or ``up-arrow``    | Access previous command in history         || ``Ctrl-n`` or ``down-arrow``  | Access next command in history             || ``Ctrl-r``                    | Reverse-search through command history     |  Miscellaneous| Keystroke                     | Action                                     ||-------------------------------|--------------------------------------------|| ``Ctrl-l``                    | Clear terminal screen                      || ``Ctrl-c``                    | Interrupt current Python command           || ``Ctrl-d``                    | Exit IPython session                       |  Notebook ShortcutsDepending on your operating system and browser, many of the navigation and text-entry shortcuts will work in the notebook as well. In addition, the notebook has many of its own shortcuts.First, though, we must mention that the notebook has two "modes" of operation: command mode and edit mode.- In **command mode**, you are doing operations that affect entire cells. You can enable command mode by pressing the escape key (or pressing ``ctrl m``). For example, in command mode, the up and down arrows will navigate from cell to cell.- In **edit mode**, you can do operations that affect the contents of a single cell. You can enable edit mode by pressing enter from the command mode. For example, in edit mode, the up and down arrows will navigate lines within the cellTo get a listing of all available shortcuts, enter command mode and press "h"  Magic CommandsIPython extends the functionality of Python with so-called "magic" commands: these are marked with a ``%`` sign.We saw one of these above; the ``%history`` command.Magic commands come in two flavors: *line magics* start with one percent sign, and *cell magics* start with two percent signs.We'll go through a few examples of magic commands here, but first, using what you've seen above, how do you think you might get a list of all available magic commands? How do you think you might get help on any particular command?<jupyter_code>%timeit?<jupyter_output><empty_output><jupyter_text>Profiling with ``timeit``For example, here's the ``%timeit``/``%%timeit`` magic, which can be very useful for quick profiling of your code:<jupyter_code>import numpy as np
x = np.random.rand(1000000)

%timeit x.sum()
L = list(x)

%timeit sum(L)
%%timeit

y = x + 1
z = y ** 2
q = z.sum()<jupyter_output>100 loops, best of 3: 6.87 ms per loop<jupyter_text>Interpreter: ``paste`` and ``cpaste``Try pasting this Python code into your IPython interpreter:```python>>> def donothing(x):        return x```You'll likely get an error.Now try typing ``%paste`` in your interpreter: what happens?Next try typing ``%cpaste`` and then use cmd-v to paste your test: what happens?  Creating a file with ``%%file``Sometimes it's useful to create a file programatically from within the notebook<jupyter_code>%%file myscript.py

def foo(x):
    return x ** 2

z = foo(12)

print(foo(14))<jupyter_output>Overwriting myscript.py<jupyter_text>Running a script with ``%run``<jupyter_code>%run myscript.py
z
foo(2)<jupyter_output><empty_output><jupyter_text>Controlling figures: ``%matplotlib``You can use the ``%matplotlib`` function to specify the matplotlib *backend* you would like to use.For example:- ``%matplotlib`` by itself uses the default system backend- ``%matplotlib inline`` creates inline, static figures (great for publication and/or sharing)- ``%matplotlib notebook`` creates inline, interactive figures (though in my experience it can be a bit unstable)<jupyter_code>%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.plot(np.random.rand(100));<jupyter_output><empty_output><jupyter_text>Help functions and more info- The ``%magic`` function will tell you all about magic commands- The ``%lsmagic`` function will list all available magic commands- Remember that the ``?`` can be used to get documentation!- Though we won't cover it here, it is possible to [create and activate your own magic commands](https://ipython.org/ipython-doc/stable/config/custommagics.html)<jupyter_code>%lsmagic
%magic
%debug?<jupyter_output><empty_output><jupyter_text>Shell CommandsIPython is meant to be an all-purpose scientific computing environment, and access to the shell is critical.Any command that starts with an exclamation point will be passed to the shell.Note that because windows has a different kind of shell than Linux/OSX, shell commands will be different from operating system to operating system.All the commands you have learned previously will work here:<jupyter_code>!ls
!pwd<jupyter_output><empty_output><jupyter_text>You can even seamlessly pass values to and from the Python interpreter.For example, we can store the result of a directory listing:<jupyter_code>contents = !ls
contents<jupyter_output><empty_output><jupyter_text>We can inject Python variables into a shell command with ``{}``:<jupyter_code>!cat {contents[4]}
for filename in contents:
    if filename.endswith('.py'):
        print(filename)
        !head -10 {filename}<jupyter_output>myscript.py

def foo(x):
    return x ** 2

z = foo(12)

print(foo(14))
StarcoderdataJupyterStructuredCleanDedup