[Python pandas] Upsampling 변환 시 생기는 결측값 채우기(fill na), 선형 보간하기(linear interpolation)

지난번 포스팅에서는 Python pandas에서 resampling 중 Downsampling 으로 집계할 때에 왼쪽과 오른쪽 중에서 어느쪽을 포함(inclusive, closed)할 지와 어느쪽으로 라벨 이름(label)을 쓸지(https://rfriend.tistory.com/507)에 대해서 알아보았습니다. 

이번 포스팅에서는 pandas의 resampling 중 Upsampling으로 시계열 데이터 주기(frequency)를 변환(conversion) 할 때 생기는 결측값을 처리하는 두 가지 방법을 소개하겠습니다. 

(1) Upsampling 으로 주기 변환 시 생기는 결측값을 채우는 방법 (filling forward/backward)

(2) Upsampling 으로 주기 변환 시 생기는 결측값을 선형 보간하는 방법 (linear interpolation)

예제로 사용할 간단할 2개의 칼럼을 가지고 주기(frequency)가 5초(5 seconds)인 시계열 데이터 DataFrame을 만들어보겠습니다. 

import pandas as pd import numpy as np rng = pd.date_range('2019-12-31', periods=3, freq='5S') rng [Out]: DatetimeIndex(['2019-12-31 00:00:00', '2019-12-31 00:00:05', '2019-12-31 00:00:10'], dtype='datetime64[ns]', freq='5S') ts = pd.DataFrame(np.array([0, 1, 3, 2, 10, 3]).reshape(3, 2),                    index=rng,                    columns=['col_1', 'col_2']) ts [Out]: col_1 col_2 2019-12-31 00:00:00 0 1 2019-12-31 00:00:05 3 2 2019-12-31 00:00:10 10 3

이제 pandas resample() 메소드를 사용해서 주기가 5초(freq='5S')인 원래 데이터를 주기가 1초(freq='1S')인 데이터로 Upsampling 변환을 해보겠습니다. 그러면 아래처럼 새로 생긴 날짜-시간 행에 결측값(missing value)이 생깁니다ㅣ  

ts_upsample = ts.resample('S').mean() ts_upsample [Out]: col_1 col_2 2019-12-31 00:00:00 0.0 1.0 2019-12-31 00:00:01 NaN NaN 2019-12-31 00:00:02 NaN NaN 2019-12-31 00:00:03 NaN NaN 2019-12-31 00:00:04 NaN NaN 2019-12-31 00:00:05 3.0 2.0 2019-12-31 00:00:06 NaN NaN 2019-12-31 00:00:07 NaN NaN 2019-12-31 00:00:08 NaN NaN 2019-12-31 00:00:09 NaN NaN 2019-12-31 00:00:10 10.0 3.0

위에 Upsampling을 해서 생긴 결측값들을 (1) 채우기(filling), (2) 선형 보간(linear interpolation) 해보겠습니다. 

(1) Upsampling 으로 주기 변환 시 생기는 결측값을 채우기 (filling missing values)

(1-1) 앞의 값으로 뒤의 결측값 채우기 (Filling forward)

# (1) filling forward ts_upsample.ffill() ts_upsample.fillna(method='ffill') ts_upsample.fillna(method='pad') [Out]: col_1 col_2 2019-12-31 00:00:00 0.0 1.0 2019-12-31 00:00:01 0.0 1.0 2019-12-31 00:00:02 0.0 1.0 2019-12-31 00:00:03 0.0 1.0 2019-12-31 00:00:04 0.0 1.0 2019-12-31 00:00:05 3.0 2.0 2019-12-31 00:00:06 3.0 2.0 2019-12-31 00:00:07 3.0 2.0 2019-12-31 00:00:08 3.0 2.0 2019-12-31 00:00:09 3.0 2.0 2019-12-31 00:00:10 10.0 3.0

(1-2) 뒤의 값으로 앞의 결측값 채우기 (Filling backward)

# (2)filling backward ts_upsample.bfill() ts_upsample.fillna(method='bfill') ts_upsample.fillna(method='backfill') [Out]: col_1 col_2 2019-12-31 00:00:00 0.0 1.0 2019-12-31 00:00:01 3.0 2.0 2019-12-31 00:00:02 3.0 2.0 2019-12-31 00:00:03 3.0 2.0 2019-12-31 00:00:04 3.0 2.0 2019-12-31 00:00:05 3.0 2.0 2019-12-31 00:00:06 10.0 3.0 2019-12-31 00:00:07 10.0 3.0 2019-12-31 00:00:08 10.0 3.0 2019-12-31 00:00:09 10.0 3.0 2019-12-31 00:00:10 10.0 3.0

(1-3) 특정 값으로 결측값 채우기

# (3)fill Missing value with '0' ts_upsample.fillna(0) [Out]: col_1 col_2 2019-12-31 00:00:00 0.0 1.0 2019-12-31 00:00:01 0.0 0.0 2019-12-31 00:00:02 0.0 0.0 2019-12-31 00:00:03 0.0 0.0 2019-12-31 00:00:04 0.0 0.0 2019-12-31 00:00:05 3.0 2.0 2019-12-31 00:00:06 0.0 0.0 2019-12-31 00:00:07 0.0 0.0 2019-12-31 00:00:08 0.0 0.0 2019-12-31 00:00:09 0.0 0.0 2019-12-31 00:00:10 10.0 3.0

(1-4) 평균 값으로 결측값 채우기

# (4) filling with mean value # mean per column ts_upsample.mean() [Out]: col_1 4.333333 col_2 2.000000 dtype: float64 ts_upsample.fillna(ts_upsample.mean()) [Out]: col_1 col_2 2019-12-31 00:00:00 0.000000 1.0 2019-12-31 00:00:01 4.333333 2.0 2019-12-31 00:00:02 4.333333 2.0 2019-12-31 00:00:03 4.333333 2.0 2019-12-31 00:00:04 4.333333 2.0 2019-12-31 00:00:05 3.000000 2.0 2019-12-31 00:00:06 4.333333 2.0 2019-12-31 00:00:07 4.333333 2.0 2019-12-31 00:00:08 4.333333 2.0 2019-12-31 00:00:09 4.333333 2.0 2019-12-31 00:00:10 10.000000 3.0

(1-5) 결측값 채우는 행의 개수 제한하기

# (5) limit the number of filling observation ts_upsample.ffill(limit=1) [Out]: col_1 col_2 2019-12-31 00:00:00 0.0 1.0 2019-12-31 00:00:01 0.0 1.0 2019-12-31 00:00:02 NaN NaN 2019-12-31 00:00:03 NaN NaN 2019-12-31 00:00:04 NaN NaN 2019-12-31 00:00:05 3.0 2.0 2019-12-31 00:00:06 3.0 2.0 2019-12-31 00:00:07 NaN NaN 2019-12-31 00:00:08 NaN NaN 2019-12-31 00:00:09 NaN NaN 2019-12-31 00:00:10 10.0 3.0

(2) Upsampling 으로 주기 변환 시 생기는 결측값을 선형 보간하기 (linear interpolation)

# (6) Linear interpolation by values ts_upsample.interpolate(method='values') # by default [Out]: col_1 col_2 2019-12-31 00:00:00 0.0 1.0 2019-12-31 00:00:01 0.6 1.2 2019-12-31 00:00:02 1.2 1.4 2019-12-31 00:00:03 1.8 1.6 2019-12-31 00:00:04 2.4 1.8 2019-12-31 00:00:05 3.0 2.0 2019-12-31 00:00:06 4.4 2.2 2019-12-31 00:00:07 5.8 2.4 2019-12-31 00:00:08 7.2 2.6 2019-12-31 00:00:09 8.6 2.8 2019-12-31 00:00:10 10.0 3.0 ts_upsample.interpolate(method='values').plot()

# (7) Linear interpolation by time ts_upsample.interpolate(method='time') [Out]: col_1 col_2 2019-12-31 00:00:00 0.0 1.0 2019-12-31 00:00:01 0.6 1.2 2019-12-31 00:00:02 1.2 1.4 2019-12-31 00:00:03 1.8 1.6 2019-12-31 00:00:04 2.4 1.8 2019-12-31 00:00:05 3.0 2.0 2019-12-31 00:00:06 4.4 2.2 2019-12-31 00:00:07 5.8 2.4 2019-12-31 00:00:08 7.2 2.6 2019-12-31 00:00:09 8.6 2.8 2019-12-31 00:00:10 10.0 3.0

많은 도움이 되었기를 바랍니다. 

이번 포스팅이 도움이 되었다면 아래의 '공감~'를 꾹 눌러주세요. :-)