Shifting and Frequency Conversion

Learn how to apply changes to time series data with shifting and frequency conversion.

Introduction

In this lesson, we’ll look at how we can modify the positions and frequencies of time series data with shifting and frequency conversion methods. We’ll use the truncated dataset version of Spain’s 2018 hourly energy demand data, as shown below:

Preview of Spain 2018 Hourly Energy Demand Data

Time

Generation Fossil Gas

Generation Fossil Hard Coal

2018-01-01 00:00:00

3471

996

2018-01-01 01:00:00

3269

959

2018-01-01 02:00:00

3541

1014

2018-01-01 03:00:00

3450

1043

2018-01-01 04:00:00

3318

1063

Shifting

Shifting is used to move values in time series data either forward or backward. This manipulation is often used to calculate differences over time, such as the change in value from one day to the next.

In pandas, the shift() method allows us to shift the data points by a certain number of periods as specified in the periods parameter. A positive value in the periods parameter will shift data forward in time (i.e., towards the future), while a negative value shifts the data backward (i.e., towards the past).

Note: Lag often refers specifically to shifting the data backward in time, i.e., passing a negative value into periods of shift().

For instance, we can shift forward the hourly energy demand data by three periods with the following code:

Press + to interact
# Shift forward by three periods (aka three rows in the time series data)

df_shift_fw = df.shift(periods=3)



# View output (first 5 rows)

print(df_shift_fw.head())

Because we’ve shifted the time series values three periods forward without changing the time series index, the first three rows of the output will now be replaced with NaN values. This output occurs because the index remains the same while the data values are realigned.

We can easily replace the NaN values with other values (like zeroes) using the fill_value parameter, as shown below:

Press + to interact
# Shift forward by three periods, and fill NaN with 0 value

df_shift_fw = df.shift(periods=3,

                       fill_value=0)



# View output (first 5 rows)

print(df_shift_fw.head())

If we wish to generate an output where the index is changed, but the data values ...

Get hands-on with 1400+ tech skills courses.