xarray Logo

Introduction to Xarray


Overview

This notebook will introduce the basics of gridded, labeled data with Xarray. Since Xarray introduces additional abstractions on top of plain arrays of data, our goal is to show why these abstractions are useful and how they frequently lead to simpler, more robust code.

We’ll cover these topics:

  1. Create a DataArray, one of the core object types in Xarray

  2. Understand how to use named coordinates and metadata in a DataArray

  3. Combine individual DataArrays into a Dataset, the other core object type in Xarray

  4. Subset, slice, and interpolate the data using named coordinates

  5. Open netCDF data using XArray

  6. Basic subsetting and aggregation of a Dataset

  7. Brief introduction to plotting with Xarray

Prerequisites

Concepts

Importance

Notes

NumPy Basics

Necessary

Intermediate NumPy

Helpful

Familiarity with indexing and slicing arrays

NumPy Broadcasting

Helpful

Familiar with array arithmetic and broadcasting

Introduction to Pandas

Helpful

Familiarity with labeled data

Datetime

Helpful

Familiarity with time formats and the timedelta object

Understanding of NetCDF

Helpful

Familiarity with metadata structure

  • Time to learn: 40 minutes


Imports

Simmilar to numpy, np; pandas, pd; you may often encounter xarray imported within a shortened namespace as xr. pythia_datasets provides example data for us to work with.

from datetime import timedelta

import numpy as np
import pandas as pd
import xarray as xr
from pythia_datasets import DATASETS

Introducing the DataArray and Dataset

Xarray expands on the capabilities on NumPy arrays, providing a lot of streamlined data manipulation. It is similar in that respect to Pandas, but whereas Pandas excels at working with tabular data, Xarray is focused on N-dimensional arrays of data (i.e. grids). Its interface is based largely on the netCDF data model (variables, attributes, and dimensions), but it goes beyond the traditional netCDF interfaces to provide functionality similar to netCDF-java’s Common Data Model (CDM).

Creation of a DataArray object

The DataArray is one of the basic building blocks of Xarray (see docs here). It provides a numpy.ndarray-like object that expands to provide two critical pieces of functionality:

  1. Coordinate names and values are stored with the data, making slicing and indexing much more powerful

  2. It has a built-in container for attributes

Here we’ll initialize a DataArray object by wrapping a plain NumPy array, and explore a few of its properties.

Generate a random numpy array

For our first example, we’ll just create a random array of “temperature” data in units of Kelvin:

data = 283 + 5 * np.random.randn(5, 3, 4)
data
array([[[281.27028165, 284.76683541, 287.29821725, 288.46681859],
        [287.18752574, 280.1832731 , 288.89899017, 277.06096696],
        [286.68859442, 281.06781606, 293.93992413, 281.39713796]],

       [[297.13630747, 291.17008218, 281.27550062, 285.61047203],
        [283.11619161, 283.74250869, 274.96474131, 279.79384779],
        [279.2587802 , 277.43180491, 284.23439915, 276.66149817]],

       [[283.63163699, 281.95526702, 277.31216652, 285.52335567],
        [283.34720363, 281.32163498, 275.02414385, 271.98549934],
        [283.72291359, 288.09955405, 283.75275265, 279.52010793]],

       [[283.49580922, 285.61679409, 282.00409958, 273.13326734],
        [279.8695052 , 282.70534454, 283.45084591, 284.96013473],
        [282.00334345, 283.85055465, 284.48353433, 284.3000379 ]],

       [[288.03315907, 278.75450045, 271.64024746, 286.71692979],
        [290.78022485, 286.9451884 , 280.06509531, 287.39110824],
        [284.43435646, 274.60247373, 283.37982519, 282.41107245]]])

Wrap the array: first attempt

Now we create a basic DataArray just by passing our plain data as input:

temp = xr.DataArray(data)
temp
<xarray.DataArray (dim_0: 5, dim_1: 3, dim_2: 4)>
array([[[281.27028165, 284.76683541, 287.29821725, 288.46681859],
        [287.18752574, 280.1832731 , 288.89899017, 277.06096696],
        [286.68859442, 281.06781606, 293.93992413, 281.39713796]],

       [[297.13630747, 291.17008218, 281.27550062, 285.61047203],
        [283.11619161, 283.74250869, 274.96474131, 279.79384779],
        [279.2587802 , 277.43180491, 284.23439915, 276.66149817]],

       [[283.63163699, 281.95526702, 277.31216652, 285.52335567],
        [283.34720363, 281.32163498, 275.02414385, 271.98549934],
        [283.72291359, 288.09955405, 283.75275265, 279.52010793]],

       [[283.49580922, 285.61679409, 282.00409958, 273.13326734],
        [279.8695052 , 282.70534454, 283.45084591, 284.96013473],
        [282.00334345, 283.85055465, 284.48353433, 284.3000379 ]],

       [[288.03315907, 278.75450045, 271.64024746, 286.71692979],
        [290.78022485, 286.9451884 , 280.06509531, 287.39110824],
        [284.43435646, 274.60247373, 283.37982519, 282.41107245]]])
Dimensions without coordinates: dim_0, dim_1, dim_2

Note two things:

  1. Xarray generates some basic dimension names for us (dim_0, dim_1, dim_2). We’ll improve this with better names in the next example.

  2. Wrapping the numpy array in a DataArray gives us a rich display in the notebook! (Try clicking the array symbol to expand or collapse the view)

Assign dimension names

Much of the power of Xarray comes from making use of named dimensions. So let’s add some more useful names! We can do that by passing an ordered list of names using the keyword argument dims:

temp = xr.DataArray(data, dims=['time', 'lat', 'lon'])
temp
<xarray.DataArray (time: 5, lat: 3, lon: 4)>
array([[[281.27028165, 284.76683541, 287.29821725, 288.46681859],
        [287.18752574, 280.1832731 , 288.89899017, 277.06096696],
        [286.68859442, 281.06781606, 293.93992413, 281.39713796]],

       [[297.13630747, 291.17008218, 281.27550062, 285.61047203],
        [283.11619161, 283.74250869, 274.96474131, 279.79384779],
        [279.2587802 , 277.43180491, 284.23439915, 276.66149817]],

       [[283.63163699, 281.95526702, 277.31216652, 285.52335567],
        [283.34720363, 281.32163498, 275.02414385, 271.98549934],
        [283.72291359, 288.09955405, 283.75275265, 279.52010793]],

       [[283.49580922, 285.61679409, 282.00409958, 273.13326734],
        [279.8695052 , 282.70534454, 283.45084591, 284.96013473],
        [282.00334345, 283.85055465, 284.48353433, 284.3000379 ]],

       [[288.03315907, 278.75450045, 271.64024746, 286.71692979],
        [290.78022485, 286.9451884 , 280.06509531, 287.39110824],
        [284.43435646, 274.60247373, 283.37982519, 282.41107245]]])
Dimensions without coordinates: time, lat, lon

This is already improved upon from a NumPy array, because we have names for each of the dimensions (or axes in NumPy parlance). Even better, we can take arrays representing the values for the coordinates for each of these dimensions and associate them with the data when we create the DataArray. We’ll see this in the next example.

Create a DataArray with named Coordinates

Make time and space coordinates

Here we will use Pandas to create an array of datetime data, which we will then use to create a DataArray with a named coordinate time.

times = pd.date_range('2018-01-01', periods=5)
times
DatetimeIndex(['2018-01-01', '2018-01-02', '2018-01-03', '2018-01-04',
               '2018-01-05'],
              dtype='datetime64[ns]', freq='D')

We’ll also create arrays to represent sample longitude and latitude:

lons = np.linspace(-120, -60, 4)
lats = np.linspace(25, 55, 3)

Initialize the DataArray with complete coordinate info

When we create the DataArray instance, we pass in the arrays we just created:

temp = xr.DataArray(data, coords=[times, lats, lons], dims=['time', 'lat', 'lon'])
temp
<xarray.DataArray (time: 5, lat: 3, lon: 4)>
array([[[281.27028165, 284.76683541, 287.29821725, 288.46681859],
        [287.18752574, 280.1832731 , 288.89899017, 277.06096696],
        [286.68859442, 281.06781606, 293.93992413, 281.39713796]],

       [[297.13630747, 291.17008218, 281.27550062, 285.61047203],
        [283.11619161, 283.74250869, 274.96474131, 279.79384779],
        [279.2587802 , 277.43180491, 284.23439915, 276.66149817]],

       [[283.63163699, 281.95526702, 277.31216652, 285.52335567],
        [283.34720363, 281.32163498, 275.02414385, 271.98549934],
        [283.72291359, 288.09955405, 283.75275265, 279.52010793]],

       [[283.49580922, 285.61679409, 282.00409958, 273.13326734],
        [279.8695052 , 282.70534454, 283.45084591, 284.96013473],
        [282.00334345, 283.85055465, 284.48353433, 284.3000379 ]],

       [[288.03315907, 278.75450045, 271.64024746, 286.71692979],
        [290.78022485, 286.9451884 , 280.06509531, 287.39110824],
        [284.43435646, 274.60247373, 283.37982519, 282.41107245]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0

Set useful attributes

…and while we’re at it, we can also set some attribute metadata:

temp.attrs['units'] = 'kelvin'
temp.attrs['standard_name'] = 'air_temperature'

temp
<xarray.DataArray (time: 5, lat: 3, lon: 4)>
array([[[281.27028165, 284.76683541, 287.29821725, 288.46681859],
        [287.18752574, 280.1832731 , 288.89899017, 277.06096696],
        [286.68859442, 281.06781606, 293.93992413, 281.39713796]],

       [[297.13630747, 291.17008218, 281.27550062, 285.61047203],
        [283.11619161, 283.74250869, 274.96474131, 279.79384779],
        [279.2587802 , 277.43180491, 284.23439915, 276.66149817]],

       [[283.63163699, 281.95526702, 277.31216652, 285.52335567],
        [283.34720363, 281.32163498, 275.02414385, 271.98549934],
        [283.72291359, 288.09955405, 283.75275265, 279.52010793]],

       [[283.49580922, 285.61679409, 282.00409958, 273.13326734],
        [279.8695052 , 282.70534454, 283.45084591, 284.96013473],
        [282.00334345, 283.85055465, 284.48353433, 284.3000379 ]],

       [[288.03315907, 278.75450045, 271.64024746, 286.71692979],
        [290.78022485, 286.9451884 , 280.06509531, 287.39110824],
        [284.43435646, 274.60247373, 283.37982519, 282.41107245]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

Attributes are not preserved by default!

Notice what happens if we perform a mathematical operaton with the DataArray: the coordinate values persist, but the attributes are lost. This is done because it is very challenging to know if the attribute metadata is still correct or appropriate after arbitrary arithmetic operations.

To illustrate this, we’ll do a simple unit conversion from Kelvin to Celsius:

temp_in_celsius = temp - 273.15
temp_in_celsius
<xarray.DataArray (time: 5, lat: 3, lon: 4)>
array([[[ 8.12028165e+00,  1.16168354e+01,  1.41482173e+01,
          1.53168186e+01],
        [ 1.40375257e+01,  7.03327310e+00,  1.57489902e+01,
          3.91096696e+00],
        [ 1.35385944e+01,  7.91781606e+00,  2.07899241e+01,
          8.24713796e+00]],

       [[ 2.39863075e+01,  1.80200822e+01,  8.12550062e+00,
          1.24604720e+01],
        [ 9.96619161e+00,  1.05925087e+01,  1.81474131e+00,
          6.64384779e+00],
        [ 6.10878020e+00,  4.28180491e+00,  1.10843992e+01,
          3.51149817e+00]],

       [[ 1.04816370e+01,  8.80526702e+00,  4.16216652e+00,
          1.23733557e+01],
        [ 1.01972036e+01,  8.17163498e+00,  1.87414385e+00,
         -1.16450066e+00],
        [ 1.05729136e+01,  1.49495540e+01,  1.06027526e+01,
          6.37010793e+00]],

       [[ 1.03458092e+01,  1.24667941e+01,  8.85409958e+00,
         -1.67326555e-02],
        [ 6.71950520e+00,  9.55534454e+00,  1.03008459e+01,
          1.18101347e+01],
        [ 8.85334345e+00,  1.07005546e+01,  1.13335343e+01,
          1.11500379e+01]],

       [[ 1.48831591e+01,  5.60450045e+00, -1.50975254e+00,
          1.35669298e+01],
        [ 1.76302248e+01,  1.37951884e+01,  6.91509531e+00,
          1.42411082e+01],
        [ 1.12843565e+01,  1.45247373e+00,  1.02298252e+01,
          9.26107245e+00]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0

For an in-depth discussion of how Xarray handles metadata, start in the Xarray docs here.

The Dataset: a container for DataArrays with shared coordinates

Along with DataArray, the other key object type in Xarray is the Dataset: a dictionary-like container that holds one or more DataArrays, which can also optionally share coordinates (see docs here).

The most common way to create a Dataset object is to load data from a file (see below). Here, instead, we will create another DataArray and combine it with our temp data.

This will illustrate how the information about common coordinate axes is used.

Create a pressure DataArray using the same coordinates

This code mirrors how we created the temp object above.

pressure_data = 1000.0 + 5 * np.random.randn(5, 3, 4)
pressure = xr.DataArray(
    pressure_data, coords=[times, lats, lons], dims=['time', 'lat', 'lon']
)
pressure.attrs['units'] = 'hPa'
pressure.attrs['standard_name'] = 'air_pressure'

pressure
<xarray.DataArray (time: 5, lat: 3, lon: 4)>
array([[[1006.50780506, 1007.72603197, 1005.87317404, 1000.44671882],
        [1005.73320555,  994.68765073, 1000.39261386, 1003.3625327 ],
        [1002.67016142,  995.88524428, 1000.54798472, 1002.25397527]],

       [[ 992.83939105, 1000.0531259 , 1002.00643667, 1004.16774676],
        [ 994.31692927, 1000.4855543 , 1001.50016754, 1006.97703198],
        [1004.38650811, 1000.97590273, 1000.4674204 , 1003.42146856]],

       [[ 993.58638555, 1004.76493671,  998.90804688,  999.70373409],
        [ 990.8952302 , 1007.80801516,  995.79959262,  997.06392573],
        [ 999.69747357, 1001.50085045, 1002.56643389,  995.42679605]],

       [[1005.80462183,  991.50282979, 1003.92994436,  988.08158433],
        [ 998.34541858, 1002.86785652,  990.89472686,  998.77685021],
        [ 992.08251229,  997.18056858,  999.84024166, 1000.06604945]],

       [[ 999.15528812, 1009.14890636, 1002.19512326, 1005.41423476],
        [1007.01457376,  995.49437239,  996.48903338,  994.12658989],
        [ 997.43344619, 1005.15533615,  992.371156  , 1001.92096017]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          hPa
    standard_name:  air_pressure

Create a Dataset object

Each DataArray in our Dataset needs a name!

The most straightforward way to create a Dataset with our temp and pressure arrays is to pass a dictionary using the keyword argument data_vars:

ds = xr.Dataset(data_vars={'Temperature': temp, 'Pressure': pressure})
ds
<xarray.Dataset>
Dimensions:      (time: 5, lat: 3, lon: 4)
Coordinates:
  * time         (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
  * lat          (lat) float64 25.0 40.0 55.0
  * lon          (lon) float64 -120.0 -100.0 -80.0 -60.0
Data variables:
    Temperature  (time, lat, lon) float64 281.3 284.8 287.3 ... 283.4 282.4
    Pressure     (time, lat, lon) float64 1.007e+03 1.008e+03 ... 1.002e+03

Notice that the Dataset object ds is aware that both data arrays sit on the same coordinate axes.

Access Data variables and Coordinates in a Dataset

We can pull out any of the individual DataArray objects in a few different ways.

Using the “dot” notation:

ds.Pressure
<xarray.DataArray 'Pressure' (time: 5, lat: 3, lon: 4)>
array([[[1006.50780506, 1007.72603197, 1005.87317404, 1000.44671882],
        [1005.73320555,  994.68765073, 1000.39261386, 1003.3625327 ],
        [1002.67016142,  995.88524428, 1000.54798472, 1002.25397527]],

       [[ 992.83939105, 1000.0531259 , 1002.00643667, 1004.16774676],
        [ 994.31692927, 1000.4855543 , 1001.50016754, 1006.97703198],
        [1004.38650811, 1000.97590273, 1000.4674204 , 1003.42146856]],

       [[ 993.58638555, 1004.76493671,  998.90804688,  999.70373409],
        [ 990.8952302 , 1007.80801516,  995.79959262,  997.06392573],
        [ 999.69747357, 1001.50085045, 1002.56643389,  995.42679605]],

       [[1005.80462183,  991.50282979, 1003.92994436,  988.08158433],
        [ 998.34541858, 1002.86785652,  990.89472686,  998.77685021],
        [ 992.08251229,  997.18056858,  999.84024166, 1000.06604945]],

       [[ 999.15528812, 1009.14890636, 1002.19512326, 1005.41423476],
        [1007.01457376,  995.49437239,  996.48903338,  994.12658989],
        [ 997.43344619, 1005.15533615,  992.371156  , 1001.92096017]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          hPa
    standard_name:  air_pressure

… or using dictionary access like this:

ds['Pressure']
<xarray.DataArray 'Pressure' (time: 5, lat: 3, lon: 4)>
array([[[1006.50780506, 1007.72603197, 1005.87317404, 1000.44671882],
        [1005.73320555,  994.68765073, 1000.39261386, 1003.3625327 ],
        [1002.67016142,  995.88524428, 1000.54798472, 1002.25397527]],

       [[ 992.83939105, 1000.0531259 , 1002.00643667, 1004.16774676],
        [ 994.31692927, 1000.4855543 , 1001.50016754, 1006.97703198],
        [1004.38650811, 1000.97590273, 1000.4674204 , 1003.42146856]],

       [[ 993.58638555, 1004.76493671,  998.90804688,  999.70373409],
        [ 990.8952302 , 1007.80801516,  995.79959262,  997.06392573],
        [ 999.69747357, 1001.50085045, 1002.56643389,  995.42679605]],

       [[1005.80462183,  991.50282979, 1003.92994436,  988.08158433],
        [ 998.34541858, 1002.86785652,  990.89472686,  998.77685021],
        [ 992.08251229,  997.18056858,  999.84024166, 1000.06604945]],

       [[ 999.15528812, 1009.14890636, 1002.19512326, 1005.41423476],
        [1007.01457376,  995.49437239,  996.48903338,  994.12658989],
        [ 997.43344619, 1005.15533615,  992.371156  , 1001.92096017]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          hPa
    standard_name:  air_pressure

We’ll return to the Dataset object when we start loading data from files.

Subsetting and selection by coordinate values

Much of the power of labeled coordinates comes from the ability to select data based on coordinate names and values, rather than array indices. We’ll explore this briefly here.

NumPy-like selection

Suppose we want to extract all the spatial data for one single date: January 2, 2018. It’s possible to achieve that with NumPy-like index selection:

indexed_selection = temp[1, :, :]  # Index 1 along axis 0 is the time slice we want...
indexed_selection
<xarray.DataArray (lat: 3, lon: 4)>
array([[297.13630747, 291.17008218, 281.27550062, 285.61047203],
       [283.11619161, 283.74250869, 274.96474131, 279.79384779],
       [279.2587802 , 277.43180491, 284.23439915, 276.66149817]])
Coordinates:
    time     datetime64[ns] 2018-01-02
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

HOWEVER, notice that this requires us (the user / programmer) to have detailed knowledge of the order of the axes and the meaning of the indices along those axes!

Named coordinates free us from this burden…

Selecting with .sel()

We can instead select data based on coordinate values using the .sel() method, which takes one or more named coordinate(s) as keyword argument:

named_selection = temp.sel(time='2018-01-02')
named_selection
<xarray.DataArray (lat: 3, lon: 4)>
array([[297.13630747, 291.17008218, 281.27550062, 285.61047203],
       [283.11619161, 283.74250869, 274.96474131, 279.79384779],
       [279.2587802 , 277.43180491, 284.23439915, 276.66149817]])
Coordinates:
    time     datetime64[ns] 2018-01-02
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

We got the same result, but

  • we didn’t have to know anything about how the array was created or stored

  • our code is agnostic about how many dimensions we are dealing with

  • the intended meaning of our code is much clearer!

Approximate selection and interpolation

With time and space data, we frequently want to sample “near” the coordinate points in our dataset. Here are a few simple ways to achieve that.

Nearest-neighbor sampling

Suppose we want to sample the nearest datapoint within 2 days of date 2018-01-07. Since the last day on our time axis is 2018-01-05, this is well-posed.

.sel has the flexibility to perform nearest neighbor sampling, taking an optional tolerance:

temp.sel(time='2018-01-07', method='nearest', tolerance=timedelta(days=2))
<xarray.DataArray (lat: 3, lon: 4)>
array([[288.03315907, 278.75450045, 271.64024746, 286.71692979],
       [290.78022485, 286.9451884 , 280.06509531, 287.39110824],
       [284.43435646, 274.60247373, 283.37982519, 282.41107245]])
Coordinates:
    time     datetime64[ns] 2018-01-05
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

where we see that .sel indeed pulled out the data for date 2018-01-05.

Interpolation

Suppose we want to extract a timeseries for Boulder (40°N, 105°W). Since lon=-105 is not a point on our longitude axis, this requires interpolation between data points.

The .interp() method (see the docs here) works similarly to .sel(). Using .interp(), we can interpolate to any latitude/longitude location:

temp.interp(lon=-105, lat=40)
<xarray.DataArray (time: 5)>
array([281.93433626, 283.58592942, 281.82802714, 281.9963847 ,
       287.90394751])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 ... 2018-01-05
    lon      int64 -105
    lat      int64 40
Attributes:
    units:          kelvin
    standard_name:  air_temperature

Info

Xarray’s interpolation functionality requires the SciPy package!

Slicing along coordinates

Frequently we want to select a range (or slice) along one or more coordinate(s). We can achieve this by passing a Python slice object to .sel(), as follows:

temp.sel(
    time=slice('2018-01-01', '2018-01-03'), lon=slice(-110, -70), lat=slice(25, 45)
)
<xarray.DataArray (time: 3, lat: 2, lon: 2)>
array([[[284.76683541, 287.29821725],
        [280.1832731 , 288.89899017]],

       [[291.17008218, 281.27550062],
        [283.74250869, 274.96474131]],

       [[281.95526702, 277.31216652],
        [281.32163498, 275.02414385]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 2018-01-03
  * lat      (lat) float64 25.0 40.0
  * lon      (lon) float64 -100.0 -80.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

Info

The calling sequence for slice always looks like slice(start, stop[, step]), where step is optional.

Notice how the length of each coordinate axis has changed due to our slicing.

One more selection method: .loc

All of these operations can also be done within square brackets on the .loc attribute of the DataArray:

temp.loc['2018-01-02']
<xarray.DataArray (lat: 3, lon: 4)>
array([[297.13630747, 291.17008218, 281.27550062, 285.61047203],
       [283.11619161, 283.74250869, 274.96474131, 279.79384779],
       [279.2587802 , 277.43180491, 284.23439915, 276.66149817]])
Coordinates:
    time     datetime64[ns] 2018-01-02
  * lat      (lat) float64 25.0 40.0 55.0
  * lon      (lon) float64 -120.0 -100.0 -80.0 -60.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

This is sort of in between the NumPy-style selection

temp[1,:,:]

and the fully label-based selection using .sel()

With .loc, we make use of the coordinate values, but lose the ability to specify the names of the various dimensions. Instead, the slicing must be done in the correct order:

temp.loc['2018-01-01':'2018-01-03', 25:45, -110:-70]
<xarray.DataArray (time: 3, lat: 2, lon: 2)>
array([[[284.76683541, 287.29821725],
        [280.1832731 , 288.89899017]],

       [[291.17008218, 281.27550062],
        [283.74250869, 274.96474131]],

       [[281.95526702, 277.31216652],
        [281.32163498, 275.02414385]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 2018-01-03
  * lat      (lat) float64 25.0 40.0
  * lon      (lon) float64 -100.0 -80.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

One advantage of using .loc is that we can use NumPy-style slice notation like 25:45, rather than the more verbose slice(25,45). But of course that also works:

temp.loc['2018-01-01':'2018-01-03', slice(25, 45), -110:-70]
<xarray.DataArray (time: 3, lat: 2, lon: 2)>
array([[[284.76683541, 287.29821725],
        [280.1832731 , 288.89899017]],

       [[291.17008218, 281.27550062],
        [283.74250869, 274.96474131]],

       [[281.95526702, 277.31216652],
        [281.32163498, 275.02414385]]])
Coordinates:
  * time     (time) datetime64[ns] 2018-01-01 2018-01-02 2018-01-03
  * lat      (lat) float64 25.0 40.0
  * lon      (lon) float64 -100.0 -80.0
Attributes:
    units:          kelvin
    standard_name:  air_temperature

What doesn’t work is passing the slices in a different order:

# This will generate an error
# temp.loc[-110:-70, 25:45,'2018-01-01':'2018-01-03']

Opening netCDF data

With its close ties to the netCDF data model, Xarray also supports netCDF as a first-class file format. This means it has easy support for opening netCDF datasets, so long as they conform to some of Xarray’s limitations (such as 1-dimensional coordinates).

Access netCDF data with xr.open_dataset

Info

Here we’re getting the data from Project Pythia’s custom library of example data, which we already imported above with from pythia_datasets import DATASETS. The DATASETS.fetch() method will automatically download and cache our example data file NARR_19930313_0000.nc locally.

filepath = DATASETS.fetch('NARR_19930313_0000.nc')
Downloading file 'NARR_19930313_0000.nc' from 'https://github.com/ProjectPythia/pythia-datasets/raw/main/data/NARR_19930313_0000.nc' to '/home/runner/.cache/pythia-datasets'.

Once we have a valid path to a data file that Xarray knows how to read, we can open it like this:

ds = xr.open_dataset(filepath)
ds
<xarray.Dataset>
Dimensions:                       (time1: 1, isobaric1: 29, y: 119, x: 268)
Coordinates:
  * time1                         (time1) datetime64[ns] 1993-03-13
  * isobaric1                     (isobaric1) float32 100.0 125.0 ... 1e+03
  * y                             (y) float32 -3.117e+03 -3.084e+03 ... 714.1
  * x                             (x) float32 -3.324e+03 ... 5.343e+03
Data variables:
    u-component_of_wind_isobaric  (time1, isobaric1, y, x) float32 ...
    LambertConformal_Projection   int32 0
    lat                           (y, x) float64 17.87 17.96 ... 34.85 34.64
    lon                           (y, x) float64 -135.0 -134.8 ... -43.13 -42.91
    Geopotential_height_isobaric  (time1, isobaric1, y, x) float32 ...
    v-component_of_wind_isobaric  (time1, isobaric1, y, x) float32 ...
    Temperature_isobaric          (time1, isobaric1, y, x) float32 ...
Attributes:
    Originating_or_generating_Center:     US National Weather Service, Nation...
    Originating_or_generating_Subcenter:  North American Regional Reanalysis ...
    GRIB_table_version:                   0,131
    Generating_process_or_model:          North American Regional Reanalysis ...
    Conventions:                          CF-1.6
    history:                              Read using CDM IOSP GribCollection v3
    featureType:                          GRID
    History:                              Translated to CF-1.0 Conventions by...
    geospatial_lat_min:                   10.753308882144761
    geospatial_lat_max:                   46.8308828962289
    geospatial_lon_min:                   -153.88242040519995
    geospatial_lon_max:                   -42.666108129242815

Subsetting the Dataset

Our call to xr.open_dataset() above returned a Dataset object that we’ve decided to call ds. We can then pull out individual fields:

ds.isobaric1
<xarray.DataArray 'isobaric1' (isobaric1: 29)>
array([ 100.,  125.,  150.,  175.,  200.,  225.,  250.,  275.,  300.,  350.,
        400.,  450.,  500.,  550.,  600.,  650.,  700.,  725.,  750.,  775.,
        800.,  825.,  850.,  875.,  900.,  925.,  950.,  975., 1000.],
      dtype=float32)
Coordinates:
  * isobaric1  (isobaric1) float32 100.0 125.0 150.0 175.0 ... 950.0 975.0 1e+03
Attributes:
    units:                   hPa
    long_name:               Isobaric surface
    positive:                down
    Grib_level_type:         100
    _CoordinateAxisType:     Pressure
    _CoordinateZisPositive:  down

(recall that we can also use dictionary syntax like ds['isobaric1'] to do the same thing)

Datasets also support much of the same subsetting operations as DataArray, but will perform the operation on all data:

ds_1000 = ds.sel(isobaric1=1000.0)
ds_1000
<xarray.Dataset>
Dimensions:                       (time1: 1, y: 119, x: 268)
Coordinates:
  * time1                         (time1) datetime64[ns] 1993-03-13
    isobaric1                     float32 1e+03
  * y                             (y) float32 -3.117e+03 -3.084e+03 ... 714.1
  * x                             (x) float32 -3.324e+03 ... 5.343e+03
Data variables:
    u-component_of_wind_isobaric  (time1, y, x) float32 -5.378 -5.393 ... 7.497
    LambertConformal_Projection   int32 0
    lat                           (y, x) float64 17.87 17.96 ... 34.85 34.64
    lon                           (y, x) float64 -135.0 -134.8 ... -43.13 -42.91
    Geopotential_height_isobaric  (time1, y, x) float32 153.2 153.2 ... 203.7
    v-component_of_wind_isobaric  (time1, y, x) float32 -5.755 -5.74 ... 9.276
    Temperature_isobaric          (time1, y, x) float32 294.1 294.1 ... 289.0
Attributes:
    Originating_or_generating_Center:     US National Weather Service, Nation...
    Originating_or_generating_Subcenter:  North American Regional Reanalysis ...
    GRIB_table_version:                   0,131
    Generating_process_or_model:          North American Regional Reanalysis ...
    Conventions:                          CF-1.6
    history:                              Read using CDM IOSP GribCollection v3
    featureType:                          GRID
    History:                              Translated to CF-1.0 Conventions by...
    geospatial_lat_min:                   10.753308882144761
    geospatial_lat_max:                   46.8308828962289
    geospatial_lon_min:                   -153.88242040519995
    geospatial_lon_max:                   -42.666108129242815

And further subsetting to a single DataArray:

ds_1000.Temperature_isobaric
<xarray.DataArray 'Temperature_isobaric' (time1: 1, y: 119, x: 268)>
array([[[294.09436, 294.1256 , ..., 298.6725 , 298.71936],
        [294.07874, 294.14124, ..., 298.6256 , 298.5475 ],
        ...,
        [276.86   , 276.84436, ..., 289.11   , 289.01624],
        [277.01624, 276.82874, ..., 289.0475 , 288.96936]]], dtype=float32)
Coordinates:
  * time1      (time1) datetime64[ns] 1993-03-13
    isobaric1  float32 1e+03
  * y          (y) float32 -3.117e+03 -3.084e+03 -3.052e+03 ... 681.6 714.1
  * x          (x) float32 -3.324e+03 -3.292e+03 ... 5.311e+03 5.343e+03
Attributes:
    long_name:           Temperature @ Isobaric surface
    units:               K
    description:         Temperature
    grid_mapping:        LambertConformal_Projection
    Grib_Variable_Id:    VAR_7-15-131-11_L100
    Grib1_Center:        7
    Grib1_Subcenter:     15
    Grib1_TableVersion:  131
    Grib1_Parameter:     11
    Grib1_Level_Type:    100
    Grib1_Level_Desc:    Isobaric surface

Aggregation operations

Not only can you use the named dimensions for manual slicing and indexing of data, but you can also use it to control aggregation operations, like std (standard deviation):

u_winds = ds['u-component_of_wind_isobaric']
u_winds.std(dim=['x', 'y'])
<xarray.DataArray 'u-component_of_wind_isobaric' (time1: 1, isobaric1: 29)>
array([[ 8.673963 , 10.212325 , 11.556413 , 12.254429 , 13.372146 ,
        15.472462 , 16.091969 , 15.846294 , 15.195834 , 13.936979 ,
        12.93888  , 12.060708 , 10.972139 ,  9.722328 ,  8.853286 ,
         8.257241 ,  7.679721 ,  7.4516497,  7.2352104,  7.039894 ,
         6.883371 ,  6.7821493,  6.7088237,  6.6865997,  6.7247376,
         6.745023 ,  6.6859775,  6.5107226,  5.972262 ]], dtype=float32)
Coordinates:
  * time1      (time1) datetime64[ns] 1993-03-13
  * isobaric1  (isobaric1) float32 100.0 125.0 150.0 175.0 ... 950.0 975.0 1e+03

Info

Aggregation methods for Xarray objects operate over the named coordinate dimension(s) specified by keyword argument dim. Compare to NumPy, where aggregations operate over specified numbered axes.

Using the sample dataset, we can calculate the mean temperature profile (temperature as a function of pressure) over Colorado within this dataset. For this exercise, consider the bounds of Colorado to be:

  • x: -182km to 424km

  • y: -1450km to -990km

(37°N to 41°N and 102°W to 109°W projected to Lambert Conformal projection coordinates)

temps = ds.Temperature_isobaric
co_temps = temps.sel(x=slice(-182, 424), y=slice(-1450, -990))
prof = co_temps.mean(dim=['x', 'y'])
prof
<xarray.DataArray 'Temperature_isobaric' (time1: 1, isobaric1: 29)>
array([[215.078  , 215.76935, 217.243  , 217.82663, 215.83487, 216.10933,
        219.99902, 224.66118, 228.80576, 234.88701, 238.78503, 242.66309,
        246.44807, 249.26636, 250.84995, 253.37354, 257.0429 , 259.08398,
        260.97955, 262.98364, 264.82138, 266.5198 , 268.22467, 269.7471 ,
        271.18216, 272.66815, 274.13037, 275.54718, 276.97675]],
      dtype=float32)
Coordinates:
  * time1      (time1) datetime64[ns] 1993-03-13
  * isobaric1  (isobaric1) float32 100.0 125.0 150.0 175.0 ... 950.0 975.0 1e+03

Plotting with Xarray

Another major benefit of using labeled data structures is that they enable automated plotting with sensible axis labels.

Simple visualization with .plot()

Much like we saw in Pandas, Xarray includes an interface to Matplotlib that we can access through the .plot() method of every DataArray.

For quick and easy data exploration, we can just call .plot() without any modifiers:

prof.plot()
[<matplotlib.lines.Line2D at 0x7f848ab423d0>]
../../_images/xarray_82_1.png

Here Xarray has generated a line plot of the temperature data against the coordinate variable isobaric. Also the metadata are used to auto-generate axis labels and units.

Customizing the plot

As in Pandas, the .plot() method is mostly just a wrapper to Matplotlib, so we can customize our plot in familiar ways.

In this air temperature profile example, we would like to make two changes:

  • swap the axes so that we have isobaric levels on the y (vertical) axis of the figure

  • make pressure decrease upward in the figure, so that up is up

A few keyword arguments to our .plot() call will take care of this:

prof.plot(y="isobaric1", yincrease=False)
[<matplotlib.lines.Line2D at 0x7f848aa91c10>]
../../_images/xarray_85_1.png

Plotting 2D data

In the example above, the .plot() method produced a line plot.

What if we call .plot() on a 2D array?

temps.sel(isobaric1=1000).plot()
<matplotlib.collections.QuadMesh at 0x7f848aa1e280>
../../_images/xarray_87_1.png

Xarray has recognized that the DataArray object calling the plot method has two coordinate variables, and generates a 2D plot using the pcolormesh method from Matplotlib.

In this case, we are looking at air temperatures on the 1000 hPa isobaric surface over North America. We could of course improve this figure by using Cartopy to handle the map projection and geographic features!


Summary

Xarray brings the joy of Pandas-style labeled data operations to N-dimensional data. As such, it has become a central workhorse in the geoscience community for the analysis of gridded datasets. Xarray allows us to open self-describing NetCDF files and make full use of the coordinate axes, labels, units, and other metadata. By making use of labeled coordinates, our code is often easier to write, easier to read, and more robust.

What’s next?

Additional notebooks to appear in this section will go into more detail about

  • arithemtic and broadcasting with Xarray data structures

  • using “group by” operations

  • remote data access with OpenDAP

  • more advanced visualization including map integration with Cartopy

Resources and references

This notebook was adapated from material in Unidata’s Python Training.

The best resource for Xarray is the Xarray documentation. See in particular

Another excellent resource is this Xarray Tutorial collection.