Jupyter Notebooks

Throughout this course we will have to create and edit python code. We will primarily use this webpage for convenience, but for day-to-day work in the laboratory, it’s beneficial to utilize a code editor or a notebook environment like JupyterLab. JupyterLab is a robust platform that enables you to develop and modify notebooks within a web browser, while also offering comprehensive capabilities for analyzing and visualizing data.

What is a Jupyter Notebook?

A Jupyter Notebook is a web browser based interactive computing environment that enables users to create documents that include code to be executed, results from the executed code such as plots and images,and finally also an additional documentation in form of markdown text including equations in LaTeX.

These documents provide a complete and self-contained record of a computation that can be converted to various formats and shared with others using email, version control systems (like git/GitHub) or nbviewer.jupyter.org.

Key Components of a Notebook

The Jupyter Notebook ecosystem consists of three main components:

  1. Notebook Editor
  2. Kernels
  3. Notebook Documents

Let’s explore each of these components in detail:

Notebook Editor

The Notebook editor is an interactive web-based application for creating and editing notebook documents. It enables users to write and run code, add rich text, and multimedia content. When running Jupyter on a server, users typically use either the classic Jupyter Notebook interface or JupyterLab, an advanced version with more features.

Key features of the Notebook editor include:

  • Code Editing: Write and edit code in individual cells.
  • Code Execution: Run code cells in any order and display computation results in various formats (HTML, LaTeX, PNG, SVG, PDF).
  • Interactive Widgets: Create and use JavaScript widgets that connect user interface controls to kernel-side computations.
  • Rich Text: Add documentation using Markdown markup language, including LaTeX equations.

The Notebook editor in Jupyter offers several advanced features:

  • Cell Metadata: Each cell has associated metadata that can be used to control its behavior. This includes tags for slideshows, hiding code cells, and controlling cell execution.

  • Magic Commands: Special commands prefixed with % (line magics) or %% (cell magics) that provide additional functionality, such as timing code execution or displaying plots inline.

  • Auto-completion: The editor provides context-aware auto-completion for Python code, helping users write code more efficiently.

  • Code Folding: Users can collapse long code blocks for better readability.

  • Multiple Cursors: Advanced editing with multiple cursors for simultaneous editing at different locations.

  • Split View: The ability to split the notebook view, allowing users to work on different parts of the notebook simultaneously.

  • Variable Inspector: A tool to inspect and manage variables in the kernel’s memory.

  • Integrated Debugger: Some Jupyter environments offer an integrated debugger for step-by-step code execution and inspection.

Kernels

Kernels are the computational engines that execute the code contained in a notebook. They are separate processes that run independently of the notebook editor.

Key responsibilities of kernels include: * Executing user code * Returning computation results to the notebook editor * Handling computations for interactive widgets * Providing features like tab completion and introspection

Jupyter notebooks are language-agnostic. Different kernels can be installed to support various programming languages such as Python, R, Julia, and many others. The default kernel runs Python code, but users can select different kernels for each notebook via the Kernel menu.

Kernels communicate with the notebook editor using a JSON-based protocol over ZeroMQ/WebSockets. For more technical details, see the messaging specification.

Each kernel runs in its own environment, which can be customized to include specific libraries and dependencies. This allows users to create isolated environments for different projects, ensuring that dependencies do not conflict.

Kernels also support interactive features such as:

  • Tab Completion: Provides suggestions for variable names, functions, and methods as you type, improving coding efficiency.
  • Introspection: Allows users to inspect objects, view documentation, and understand the structure of code elements.
  • Rich Output: Supports various output formats, including text, images, videos, and interactive widgets, enhancing the interactivity of notebooks.

Advanced users can create custom kernels to support additional languages or specialized computing environments. This involves writing a kernel specification and implementing the necessary communication protocols.

For managing kernels, Jupyter provides several commands and options:

  • Starting a Kernel: Automatically starts when a notebook is opened.
  • Interrupting a Kernel: Stops the execution of the current code cell, useful for halting long-running computations.
  • Restarting a Kernel: Clears the kernel’s memory and restarts it, useful for resetting the environment or recovering from errors.
  • Shutting Down a Kernel: Stops the kernel and frees up system resources.

Users can also monitor kernel activity and resource usage through the Jupyter interface, ensuring efficient and effective use of computational resources.

JupyterLab Example

The following is an example of a JupyterLab interface with a notebook editor, code cells, markdown cells, and a kernel selector:

Notebook Documents

Notebook documents are self-contained files that encapsulate all content created in the notebook editor. They include code inputs/outputs, Markdown text, equations, images, and other media. Each document is associated with a specific kernel and serves as both a human-readable record of analysis and an executable script to reproduce the work.

Characteristics of notebook documents:

  • File Extension: Notebooks are stored as files with a .ipynb extension.
  • Structure: Notebooks consist of a linear sequence of cells, which can be one of three types:
    • Code cells: Contain executable code and its output.
    • Markdown cells: Contain formatted text, including LaTeX equations.
    • Raw cells: Contain unformatted text, preserved when converting notebooks to other formats.

  • Version Control: Notebook documents can be version controlled using systems like Git. This allows users to track changes, collaborate with others, and revert to previous versions if needed. Tools like nbdime provide diff and merge capabilities specifically designed for Jupyter Notebooks.

  • Cell Tags: Cells in a notebook can be tagged with metadata to control their behavior during execution, export, or presentation. For example, tags can be used to hide input or output, skip execution, or designate cells as slides in a presentation.

  • Interactive Widgets: Notebook documents can include interactive widgets that allow users to manipulate parameters and visualize changes in real-time. This is particularly useful for data exploration and interactive simulations.

  • Extensions: The Jupyter ecosystem supports a wide range of extensions that enhance the functionality of notebook documents. These extensions can add features like spell checking, code formatting, and integration with external tools and services.

  • Security: Notebook documents can include code that executes on the user’s machine, which poses security risks. Jupyter provides mechanisms to sanitize notebooks and prevent the execution of untrusted code. Users should be cautious when opening notebooks from unknown sources.

  • Collaboration: Jupyter Notebooks can be shared and collaboratively edited in real-time using platforms like Google Colab, Microsoft Azure Notebooks, and JupyterHub. These platforms provide cloud-based environments where multiple users can work on the same notebook simultaneously.

  • Customization: Users can customize the appearance and behavior of notebook documents using CSS and JavaScript. This allows for the creation of tailored interfaces and enhanced user experiences.

  • Export Options: In addition to static formats, notebooks can be exported to interactive formats like dashboards and web applications. Tools like Voila convert notebooks into standalone web applications that can be shared and deployed.

  • Provenance: Notebooks can include provenance information that tracks the origin and history of data and computations. This is important for reproducibility and transparency in scientific research.

  • Documentation: Notebook documents can serve as comprehensive documentation for projects, combining code, results, and narrative text. This makes them valuable for teaching, tutorials, and sharing research findings.

  • Performance: Large notebooks with many cells and outputs can become slow and unwieldy. Techniques like cell output clearing, using lightweight data formats, and splitting notebooks into smaller parts can help maintain performance.

  • Integration: Jupyter Notebooks can integrate with a wide range of data sources, libraries, and tools. This includes databases, cloud storage, machine learning frameworks, and visualization libraries, making them a versatile tool for data science and research.

  • Internal Format: Notebook files are JSON text files with binary data encoded in base64, making them easy to manipulate programmatically.

  • Exportability: Notebooks can be exported to various static formats (HTML, reStructuredText, LaTeX, PDF, slide shows) using Jupyter’s nbconvert utility.

  • Sharing: Notebooks can be shared via nbviewer, which renders notebooks from public URLs or GitHub as static web pages, allowing others to view the content without installing Jupyter.

This integrated system of editor, kernels, and documents makes Jupyter Notebooks a powerful tool for interactive computing, data analysis, and sharing of computational narratives.

Using the Notebook Editor

Jupyter Notebook Editor

The Jupyter Notebook editor provides an interactive environment for writing code, creating visualizations, and documenting computational workflows. It consists of a web-based interface that allows users to create and edit notebook documents containing code, text, equations, images, and interactive elements. A Jupyter Notebook provides an interface with essentially two modes of operation:

  • edit mode the mode where you edit a cells content.
  • command mode the mode where you execute the cells content.

In the more advanced version of JupyterLab you can also have a presentation mode where you can present your notebook as a slideshow.

Edit mode

Edit mode is indicated by a blue cell border and a prompt showing in the editor area when a cell is selected. You can enter edit mode by pressing Enter or using the mouse to click on a cell’s editor area.

Edit Mode

When a cell is in edit mode, you can type into the cell, like a normal text editor

Command mode

Command mode is indicated by a grey cell border with a blue left margin. When you are in command mode, you are able to edit the notebook as a whole, but not type into individual cells. Most importantly, in command mode, the keyboard is mapped to a set of shortcuts that let you perform notebook and cell actions efficiently.

Command Mode

If you have a hardware keyboard connected to your iOS device, you can use Jupyter keyboard shortcuts. The modal user interface of the Jupyter Notebook has been optimized for efficient keyboard usage. This is made possible by having two different sets of keyboard shortcuts: one set that is active in edit mode and another in command mode.

Keyboard navigation

In edit mode, most of the keyboard is dedicated to typing into the cell’s editor area. Thus, in edit mode there are relatively few shortcuts available. In command mode, the entire keyboard is available for shortcuts, so there are many more. Most important ones are:

  1. Switch command and edit mods: Enter for edit mode, and Esc or Control for command mode.
  2. Basic navigation: ↑/k, ↓/j
  3. Run or render currently selected cell: Shift+Enter or Control+Enter
  4. Saving the notebook: s
  5. Change Cell types: y to make it a code cell, m for markdown and r for raw
  6. Inserting new cells: a to insert above, b to insert below
  7. Manipulating cells using pasteboard: x for cut, c for copy, v for paste, d for delete and z for undo delete
  8. Kernel operations: i to interrupt and 0 to restart

Running code

Code cells allow you to enter and run code. Run a code cell by pressing the ▶︎ button in the bottom-right panel, or Control+Enter on your hardware keyboard.

v = 23752636
print(v)
23752636

There are a couple of keyboard shortcuts for running code:

  • Control+Enter run the current cell and enters command mode.
  • Shift+Enter runs the current cell and moves selection to the one below.
  • Option+Enter runs the current cell and inserts a new one below.

Managing the kernel

Code is run in a separate process called the kernel, which can be interrupted or restarted. You can see kernel indicator in the top-right corner reporting current kernel state: ⚪︎ means kernel is ready to execute code, and ⚫︎ means kernel is currently busy. Tapping kernel indicator will open kernel menu, where you can reconnect, interrupt or restart kernel.

Try running the following cell — kernel indicator will switch from ⚪︎ to ⚫︎, i.e. reporting kernel as “busy”. This means that you won’t be able to run any new cells until current execution finishes, or until kernel is interrupted. You can then go to kernel menu by tapping the kernel indicator and select “Interrupt”.

Markdown in Notebooks

Text can be added to Jupyter Notebooks using Markdown cells. This is extremely useful providing a complete documentation of your calculations or simulations. In this way, everything really becomes an notebook. You can change the cell type to Markdown by using the “Cell Actions” menu, or with a hardware keyboard shortcut m. Markdown is a popular markup language that is a superset of HTML. Its specification can be found here:

https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet

Markdown cells can either be rendered or unrendered.

When they are rendered, you will see a nice formatted representation of the cell’s contents.

When they are unrendered, you will see the raw text source of the cell. To render the selected cell, click the ▶︎ button or shift+ enter. To unrender, select the markdown cell, and press enter or just double click.

Markdown basics

Below are some basic markdown examples, in its rendered form. If you which to access how to create specific appearances, double click the individual cells to put the into an unrendered edit mode.

You can make text italic or bold. You can build nested itemized or enumerated lists:

Markdown lists example

  • First item
    • First subitem
      • First sub-subitem
    • Second subitem
      • First subitem of second subitem
      • Second subitem of second subitem
  • Second item
    • First subitem
  • Third item
    • First subitem

Now another list:

  1. Here we go
    1. Sublist
      1. Sublist
  2. There we go
  3. Now this

Blockquote example

Beautiful is better than ugly. Explicit is better than implicit. Simple is better than complex. Complex is better than complicated. Flat is better than nested. Sparse is better than dense. Readability counts. Special cases aren’t special enough to break the rules. Namespaces are one honking great idea – let’s do more of those!

Headings

You can add headings by starting a line with one (or multiple) # followed by a space and the title of your section. The number of # you use will determine the size of the heading

# Heading 1
# Heading 2
## Heading 2.1
## Heading 2.2
### Heading 2.2.1

Embedded code

You can embed code meant for illustration instead of execution in Python:

def f(x):
    """a docstring"""
    return x**2

LaTeX equations

Courtesy of MathJax, you can include mathematical expressions both inline: \(e^{i\pi} + 1 = 0\) and displayed:

\[e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i\]

Inline expressions can be added by surrounding the latex code with $:

$e^{i\pi} + 1 = 0$

Expressions on their own line are surrounded by $$:

$$e^x=\sum_{i=0}^\infty \frac{1}{i!}x^i$$

Images

Images may be also directly integrated into a Markdown block.

To include images use

![alternative text](url)

for example

alternative text

Videos

To include videos, we use HTML code like

<video src="mov/movie.mp4" width="320" height="200" controls preload></video>

in the Markdown cell. This works with videos stored locally.

You can embed YouTube Videos as well by using the IPython module.

from IPython.display import YouTubeVideo
YouTubeVideo('QlLx32juGzI',width=600)