R, Python 분석과 프로그래밍의 친구 (by R Friend)

  (1) DataFrame으로 부터 특정 개수의 표본을 무작위(확률)로 추출하기 (number)

import pandas as pd

df = pd.DataFrame({'num_legs': [2, 4, 8, 0],
                   'num_wings': [2, 0, 0, 0],
                   'num_specimen_seen': [10, 2, 1, 8]},
                  index=['falcon', 'dog', 'spider', 'fish'])

df

df.sample(n=2, # number of items from axis to return.
          random_state=1004) # seed for random number generator for reproducibility

  (2) DataFrame으로 부터 특정 비율의 표본을 무작위로 추출하기 (fraction)

df.sample(frac=0.5, # fraction of axis items to return.
          random_state=1004)

## ValueError: Replace has to be set to `True` when upsampling the population `frac` > 1.
df.sample(frac=1.5,
          random_state=1004)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-45-2fcc4494d7ae> in <module>
----> 1 df.sample(frac=1.5, # fraction of axis items to return. 
      2           random_state=1004)

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/generic.py in sample(self, n, frac, replace, weights, random_state, axis)
   5326             n = 1
   5327         elif frac is not None and frac > 1 and not replace:
-> 5328             raise ValueError(
   5329                 "Replace has to be set to `True` when "
   5330                 "upsampling the population `frac` > 1."

ValueError: Replace has to be set to `True` when upsampling the population `frac` > 1.

## parameter 'n' and 'frac' cannot be used at the same time.
## ValueError: Please enter a value for `frac` OR `n`, not both
df.sample(n=2, frac=0.5)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-6-b31ebc150882> in <module>
      1 ## parameter 'n' and 'frac' cannot be used at the same time.
      2 ## ValueError: Please enter a value for `frac` OR `n`, not both
----> 3 df.sample(n=2, frac=0.5)

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/generic.py in sample(self, n, frac, replace, weights, random_state, axis)
   5335             n = int(round(frac * axis_length))
   5336         elif n is not None and frac is not None:
-> 5337             raise ValueError("Please enter a value for `frac` OR `n`, not both")
   5338 
   5339         # Check for negative sizes

ValueError: Please enter a value for `frac` OR `n`, not both

  (3) DataFrame으로 부터 복원 무작위 표본 추출하기

      (random sampling with replacement)

## replace=True: random sampling with replacement
df.sample(n=8, # or equivalently: frac=2
          replace=True, # random sampling with replacement
          random_state=1004)

## ValueError: Cannot take a larger sample than population when 'replace=False'
df.sample(n=8,
          replace=False # random sampling without replacement
)

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-42-40c76bd4c271> in <module>
      1 ## replace=True: random sampling with replacement
----> 2 df.sample(n=8, # or equivalently: frac=2
      3           replace=False # random sampling without replacement
      4 )

~/opt/anaconda3/lib/python3.8/site-packages/pandas/core/generic.py in sample(self, n, frac, replace, weights, random_state, axis)
   5343             )
   5344 
-> 5345         locs = rs.choice(axis_length, size=n, replace=replace, p=weights)
   5346         return self.take(locs, axis=axis)
   5347 

mtrand.pyx in numpy.random.mtrand.RandomState.choice()

ValueError: Cannot take a larger sample than population when 'replace=False'

  (4) DataFrame으로 부터 가중치를 부여하여 표본 추출하기 (weights)

## Using a DataFrame column as weights.
## Rows with larger value in the num_specimen_seen column are more likely to be sampled.
df.sample(n=2,
          weights='num_specimen_seen'

)

  (5) DataFrame으로 부터 칼럼에 대해 무작위 표본 추출하기 (axis=1, axis='column)

## Axis to sample: by column
df.sample(n=2,
          random_state=1004,
          axis=1) # or equivalently, axis='column'

## Axis to sample: by index
df.sample(n=2,
          random_state=1004,
          axis=0) # or equivalently, axis='index', default

  (6) DataFrame으로 부터 특정 칼럼에 대해 무작위 표본 추출한 결과를

       numpy array로 할당하기

## Sampling only for a column
df['num_legs'].sample(n=2, random_state=1004)

[Out] 
falcon    2
fish      0
Name: num_legs, dtype: int64

df['num_specimen_seen'].sample(n=2, random_state=1004)

[Out] 
falcon    10
fish       8
Name: num_specimen_seen, dtype: int64

## Assigning sampling results as Series
samp_Series = df['num_legs'].sample(n=2)
type(samp_Series)

[Out] pandas.core.series.Series

## Assigning sampling results as numpy array
import numpy as np
samp_array = np.array(df['num_legs'].sample(n=2))
type(samp_array)

[Out] numpy.ndarray

samp_array

[Out] array([0, 2])

  (1) DataFrame.iterrows() : 행에 대해 순환 반복
      (Iterate over DataFrame rows as (index, Series) pairs.)

import pandas as pd

df = pd.DataFrame(
    {'price': [100, 200, 300],
     'weight': [20.3, 15.1, 25.9]},
    index=['idx_a', 'idx_b', 'idx_c'])

df

## DataFrame.iterrows()
for idx, row in df.iterrows():
    print("** index name:", idx)
    print(row)
    print("------"*5)

[Out]
** index name: idx_a
price     100.0
weight     20.3
Name: idx_a, dtype: float64
------------------------------
** index name: idx_b
price     200.0
weight     15.1
Name: idx_b, dtype: float64
------------------------------
** index name: idx_c
price     300.0
weight     25.9
Name: idx_c, dtype: float64
------------------------------

## accessing to column of each rows by indexing
for idx, row in df.iterrows():
    print(idx)
    print(row['price']) # or print(row[0])
    print("-----")

[Out]
idx_a
100.0
-----
idx_b
200.0
-----
idx_c
300.0
-----

## DataFrame.iterrows() returns a Series for each row,
## it does not preserve dtypes across the rows.
print('Data type of df price:', df['price'].dtype) # int
print('Data type of row price:', row['price'].dtype) # float

[Out]
Data type of df price: int64
Data type of row price: float64

  (2) DataFrame.iteritems() : 열에 대해 순환 반복
      (Iterate over DataFrame (column name, Series) pairs.)

df

for col, item in df.iteritems():
    print("** column name:", col)
    print(item) # = print(item, sep='\n')
    print("-----"*5)

[Out]
** column name: price
idx_a    100
idx_b    200
idx_c    300
Name: price, dtype: int64
-------------------------
** column name: weight
idx_a    20.3
idx_b    15.1
idx_c    25.9
Name: weight, dtype: float64
-------------------------

for col, item in df.iteritems():
    print(col)
    print(item[0]) # = print(item['idx_a'])

[Out]
price
100
weight
20.3

  (3) DataFrame.itertuples() : 이름이 있는 튜플 (인덱스, 행, 열) 에 대해 순환 반복

df

for row in df.itertuples():
    print(row)

[Out] 
Pandas(Index='idx_a', price=100, weight=20.3)
Pandas(Index='idx_b', price=200, weight=15.1)
Pandas(Index='idx_c', price=300, weight=25.9)

## By setting the indx=False, we can remove the index as the first element of the tuple.
for row in df.itertuples(index=False):
    print(row)

[Out] 
Pandas(price=100, weight=20.3)
Pandas(price=200, weight=15.1)
Pandas(price=300, weight=25.9)

## Setting a custom name for the yielded namedtuples.
for row in df.itertuples(name='Product'):
    print(row)

[Out]
Product(Index='idx_a', price=100, weight=20.3)
Product(Index='idx_b', price=200, weight=15.1)
Product(Index='idx_c', price=300, weight=25.9)

## DataFrame.itertuples() preserves dtypes, returning namedtuples of the values.
print('Data type of df price:', df['price'].dtype) # int
print('Data type of row price:', type(row.price)) # int

[Out] 
Data type of df price: int64
Data type of row price: <class 'int'>

import pandas as pd

tr = pd.DataFrame({

    'ts': ['2020-06-01', '2020-06-02', '2020-06-03', '2020-06-01', '2020-06-03'], 

    'id': [1, 1, 1, 2, 3], 

    'amt': [100, 300, 50, 200, 150]})

tr

date_id_idx = pd.MultiIndex.from_product((set(tr.ts), set(tr.id)))

date_id_idx

MultiIndex([('2020-06-01', 1),
            ('2020-06-01', 2),
            ('2020-06-01', 3),
            ('2020-06-02', 1),
            ('2020-06-02', 2),
            ('2020-06-02', 3),
            ('2020-06-03', 1),
            ('2020-06-03', 2),
            ('2020-06-03', 3)],
           )

tr_tsformat = tr.set_index(['ts', 'id']).reindex(date_id_idx)

tr_tsformat

tr_tsformat = tr.set_index(['ts', 'id']).reindex(date_id_idx, fill_value=0)

tr_tsformat

tr_tsformat.reset_index(inplace=True)

tr_tsformat

tr_tsformat.columns = ['ts', 'id', 'amt']

tr_tsformat

  (1) 동일 간격의 시계열 데이터 Series 만들기 (fixed frequency time series pandas Series)

import pandas as pd

# generate dates from 2019-12-01 to 2019-12-10

date_idx = pd.date_range('2019-12-01', periods=10)

date_idx

[Out]:
DatetimeIndex(['2019-12-01', '2019-12-02', '2019-12-03', '2019-12-04',
               '2019-12-05', '2019-12-06', '2019-12-07', '2019-12-08',
               '2019-12-09', '2019-12-10'],
              dtype='datetime64[ns]', freq='D')

# drop 2 dates from DatetimeIndex

date_idx = date_idx.drop(pd.DatetimeIndex(['2019-12-04', '2019-12-08']))

date_idx

[Out]:

DatetimeIndex(['2019-12-01', '2019-12-02', '2019-12-03', '2019-12-05',
               '2019-12-06', '2019-12-07', '2019-12-09', '2019-12-10'],
              dtype='datetime64[ns]', freq=None)

# Time Series with missing dates index

series_ts_missing = pd.Series(range(len(date_idx))

                              , index=date_idx)

series_ts_missing

[Out]:

2019-12-01    0
2019-12-02    1
2019-12-03    2
2019-12-05    3
2019-12-06    4
2019-12-07    5
2019-12-09    6
2019-12-10    7
dtype: int64

# Create a 1 day Fixed Frequency Time Series using resample('D')

series_ts_fixed_freq = series_ts_missing.resample('D')

series_ts_fixed_freq.first()

[Out]:

2019-12-01    0.0
2019-12-02    1.0
2019-12-03    2.0
2019-12-04    NaN <---
2019-12-05    3.0
2019-12-06    4.0
2019-12-07    5.0
2019-12-08    NaN <---
2019-12-09    6.0
2019-12-10    7.0
Freq: D, dtype: float64

# fill missing value with '0'

series_ts_fixed_freq.first().fillna(0)

[Out]:

2019-12-01    0.0
2019-12-02    1.0
2019-12-03    2.0
2019-12-04    0.0 <---
2019-12-05    3.0
2019-12-06    4.0
2019-12-07    5.0
2019-12-08    0.0 <---
2019-12-09    6.0
2019-12-10    7.0
Freq: D, dtype: float64

# resampling with 10 minutes frequency (interval)

series_ts_missing.resample('10T').first()

[Out]:
2019-12-01 00:00:00    0.0
2019-12-01 00:10:00    NaN
2019-12-01 00:20:00    NaN
2019-12-01 00:30:00    NaN
2019-12-01 00:40:00    NaN
                      ... 
2019-12-09 23:20:00    NaN
2019-12-09 23:30:00    NaN
2019-12-09 23:40:00    NaN
2019-12-09 23:50:00    NaN
2019-12-10 00:00:00    7.0
Freq: 10T, Length: 1297, dtype: float64

  (2) 동일 간격의 시계열 데이터 DataFrame 만들기 

       (fixed frequency time series pandas DataFrame)

import pandas as pd

# generate dates from 2019-12-01 to 2019-12-10

date_idx = pd.date_range('2019-12-01', periods=10)

# drop 2 dates from DatetimeIndex

date_idx = date_idx.drop(pd.DatetimeIndex(['2019-12-04', '2019-12-08']))

date_idx

[Out]:
DatetimeIndex(['2019-12-01', '2019-12-02', '2019-12-03', '2019-12-05',

               '2019-12-06', '2019-12-07', '2019-12-09', '2019-12-10'],
              dtype='datetime64[ns]', freq=None)

df_ts_missing = pd.DataFrame(range(len(date_idx))

                             , columns=['col']

                             , index=date_idx)

df_ts_missing

[Out]:

df_ts_fixed_freq = df_ts_missing.resample('D').first()

df_ts_fixed_freq

[Out]:

# fill missing value with '0'

df_ts_fixed_freq = df_ts_fixed_freq.fillna(0)

df_ts_fixed_freq

  (1) pandas Series에서 시계열 데이터 indexing, slicing, selection, truncation 하는 방법

import pandas as pd

from datetime import datetime

# DatetimeIndex

ts_days_idx = pd.date_range('2019-11-25', periods=10)

ts_days_idx

[Out]:
DatetimeIndex(['2019-11-25', '2019-11-26', '2019-11-27', '2019-11-28',
               '2019-11-29', '2019-11-30', '2019-12-01', '2019-12-02',
               '2019-12-03', '2019-12-04'],
              dtype='datetime64[ns]', freq='D')

# Series with time series index

series_ts = pd.Series(range(len(ts_days_idx))

                      , index=ts_days_idx)

series_ts

[Out]:

2019-11-25    0
2019-11-26    1
2019-11-27    2
2019-11-28    3
2019-11-29    4
2019-11-30    5
2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

series_ts.index

[Out]: 

DatetimeIndex(['2019-11-25', '2019-11-26', '2019-11-27', '2019-11-28',
               '2019-11-29', '2019-11-30', '2019-12-01', '2019-12-02',
               '2019-12-03', '2019-12-04'],
              dtype='datetime64[ns]', freq='D')

series_ts.index[6]

[Out]: Timestamp('2019-12-01 00:00:00', freq='D')

import pandas as pd

pd.date_range(start='2019-11-25', end='2019-12-04')

[Out]:

DatetimeIndex(['2019-11-25', '2019-11-26', '2019-11-27', '2019-11-28',
               '2019-11-29', '2019-11-30', '2019-12-01', '2019-12-02',
               '2019-12-03', '2019-12-04'],
              dtype='datetime64[ns]', freq='D')

# 10 timeseries data points by Second(freq='S')

pd.date_range('2019-11-25 00:00:00', periods=10, freq='S')

[Out]: 

DatetimeIndex(['2019-11-25 00:00:00', '2019-11-25 00:00:01',
               '2019-11-25 00:00:02', '2019-11-25 00:00:03',
               '2019-11-25 00:00:04', '2019-11-25 00:00:05',
               '2019-11-25 00:00:06', '2019-11-25 00:00:07',
               '2019-11-25 00:00:08', '2019-11-25 00:00:09'],
              dtype='datetime64[ns]', freq='S')

# 10 timeseries data points by 10 Seconds (freq='10S')

pd.date_range('2019-11-25 00:00:00', periods=10, freq='10S')

[Out]:
DatetimeIndex(['2019-11-25 00:00:00', '2019-11-25 00:00:10',
               '2019-11-25 00:00:20', '2019-11-25 00:00:30',
               '2019-11-25 00:00:40', '2019-11-25 00:00:50',
               '2019-11-25 00:01:00', '2019-11-25 00:01:10',
               '2019-11-25 00:01:20', '2019-11-25 00:01:30'],
              dtype='datetime64[ns]', freq='10S')

import pandas as pd

from datetime import datetime

# (a) indexing with index number

series_ts[6]

[Out]: 6

# (b) indexing with index number using iloc

series_ts.iloc[6]

[Out]: 6

# (c) indexing with string ['year-month-day']

series_ts['2019-12-01']

[Out]: 6

# (d) indexing with string ['month/day/year']

series_ts['12/01/2019']

[Out]: 6

# (f) indexing with datetime.datetime(year, month, day)

series_ts[datetime(2019, 12, 1)]

[Out]: 6

import pandas as pd

from datetime import datetime

# (a) slicing with position

series_ts[6:]

[Out]:

2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# (b) slicing with position using iloc

series_ts.iloc[6:]

[Out]:

2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# (c) slicing with string

series_ts['2019-12-01':'2019-12-10']

[Out]:

2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# (d) slicing with string

series_ts['12/01/2019':'12/10/2019']

[Out]:

2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# (e) slicing with datetime

series_ts[datetime(2019, 12, 1):datetime(2019, 12, 10)]

[Out]:

2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# selection with year string

series_ts['2019']

[Out]:
2019-11-25    0
2019-11-26    1
2019-11-27    2
2019-11-28    3
2019-11-29    4
2019-11-30    5
2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# selection with year-month string

series_ts['2019-12']

[Out]:

2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# truncate before

series_ts.truncate(before='2019-12-01')

[Out]:

2019-12-01    6
2019-12-02    7
2019-12-03    8
2019-12-04    9
Freq: D, dtype: int64

# truncate after

series_ts.truncate(after='2019-11-30')

[Out]:

2019-11-25    0
2019-11-26    1
2019-11-27    2
2019-11-28    3
2019-11-29    4
2019-11-30    5
Freq: D, dtype: int64

  (2) pandas DataFrame에서 시계열 데이터 indexing, slicing, selection, truncation 하는 방법

import pandas as pd

from datetime import datetime

# DatetimeIndex

ts_days_idx = pd.date_range('2019-11-25', periods=10)

ts_days_idx

[Out]:

DatetimeIndex(['2019-11-25', '2019-11-26', '2019-11-27', '2019-11-28',
               '2019-11-29', '2019-11-30', '2019-12-01', '2019-12-02',
               '2019-12-03', '2019-12-04'],
              dtype='datetime64[ns]', freq='D')

# DataFrame with DatetimeIndex

df_ts = pd.DataFrame(range(len(ts_days_idx))

                     , columns=['col']

                     , index=ts_days_idx)

df_ts

# (a) indexing with index position integer using iloc[]

df_ts.iloc[6]

[Out]:

col    6
Name: 2019-12-01 00:00:00, dtype: int64

# (b) indexing with index labels ['year-month-day']

df_ts.loc['2019-12-01']

[Out]:

col    6
Name: 2019-12-01 00:00:00, dtype: int64

# (c) indexing with index labels ['month/day/year']

df_ts.loc['12/01/2019']

[Out]:

col    6
Name: 2019-12-01 00:00:00, dtype: int64

# (a) slicing DataFrame with position integer

df_ts[6:10]

# (b) silcing using date strings 'year-month-day'

df_ts.loc['2019-12-01':'2019-12-10']

# (c) slicing using date strings 'month/day/year'

df_ts.loc['12/01/2019':'12/10/2019']

# (d) slicing using datetime objects

from datetime import datetime

df_ts.loc[datetime(2019, 12, 1):datetime(2019, 12, 10)]

# selection of year '2019'

df_ts.loc['2019'] # df_ts['2019']

# selection of year-month '2019-12'

df_ts.loc['2019-12']

# truncate before

df_ts.truncate(before='2019-12-01') # '2019-12-01' is not removed

# truncate after

df_ts.truncate(after='2019-11-30') # '2019-11-30' is not removed

 (1) pandas DataFrame 내 문자열 칼럼을 구분자로 분리하여 여러개의 칼럼 만들기

import pandas as pd

df = pd.DataFrame({'col': ['a:1:20.3:S', 'b:2:10.5:C', 'c:3:51.9:A']})

df

df[['group', 'id', 'value', 'grade']] = pd.DataFrame(df.col.str.split(':', 3).tolist())

df

df2 = pd.DataFrame(df.col.str.split(':', 3).tolist(), 

                   columns = ['group', 'id', 'value', 'grade'])

df2

df2.dtypes

group    object
id       object
value    object
grade    object
dtype: object

 (2) PostgreSQL, GPDB에서 문자열 칼럼을 구분자로 분리하여 여러개 칼럼 만들기

-- make a table

DROP TABLE IF EXISTS grp_val_grade;

CREATE TABLE grp_val_grade (

col varchar(100) NOT NULL

);

INSERT INTO grp_val_grade VALUES ('a:1:20.3:S');

INSERT INTO grp_val_grade VALUES('b:2:10.5:C');

INSERT INTO grp_val_grade VALUES('c:3:51.9:A');

SELECT * FROM grp_val_grade; 

-- split a column by delimeter and make 4 columns

DROP TABLE IF EXISTS grp_val_grade2;

CREATE TABLE grp_val_grade2 AS (

SELECT 

col

, split_part(col, ':', 1) AS group

, split_part(col, ':', 2) AS id 

, split_part(col, ':', 3) AS value

, split_part(col, ':', 4) AS grade

FROM grp_val_grade

);

SELECT * FROM grp_val_grade2;

import numpy as np

import pandas as pd

from pandas import DataFrame

# making DataFrame with 4 random variables

np.random.seed(123) # for reproducibility

df= DataFrame(np.random.randn(10, 4), 

                    columns=['a', 'b', 'c', 'd'])

# setting index with 2 group, 'grp1' and 'grp2'

df['group'] = ['grp1', 'grp1', 'grp1', 'grp1', 'grp1', 

                  'grp2', 'grp2', 'grp2', 'grp2', 'grp2']

df = df.set_index('group')

df

  (1) 'd' 변수와 나머지 모든 변수 간 그룹 별 상관계수 구하기 : x.corrwith(x['d'])

# correlation with columns: corrwith()

corr_with_d = lambda x: x.corrwith(x['d'])

grouped = df.groupby('group')

grouped.apply(corr_with_d) 

  (2) 'a'변수와 'd'변수 간 그룹 별 상관계수 구하기 : g['a'].corr[g['d'])

# inter-column correlation: corr()

corr_a_d = lambda g: g['a'].corr(g['d'])

grouped = df.groupby('group')

DataFrame(grouped.apply(corr_a_d))

import numpy as np

import pandas as pd

from pandas import DataFrame

np.random.seed(123)

df = DataFrame({'col_1': np.random.randint(20, size=20), 

                      'col_2': np.random.randn(20)})

df

  (1) pd.cut : 동일 길이로 나누어서 범주 만들기(equal-length buckets categorization)

factor_col_1 = pd.cut(df.col_1, 4)

factor_col_1

0 (9.5, 14.25]

1     (-0.019, 4.75]
2     (-0.019, 4.75]
3        (4.75, 9.5]
4      (14.25, 19.0]
5      (14.25, 19.0]
6       (9.5, 14.25]
7     (-0.019, 4.75]
8     (-0.019, 4.75]
9      (14.25, 19.0]
10     (14.25, 19.0]
11       (4.75, 9.5]
12    (-0.019, 4.75]
13      (9.5, 14.25]
14    (-0.019, 4.75]
15     (14.25, 19.0]
16     (14.25, 19.0]
17      (9.5, 14.25]
18    (-0.019, 4.75]
19    (-0.019, 4.75]
Name: col_1, dtype: category
Categories (4, interval[float64]): [(-0.019, 4.75] < (4.75, 9.5] < (9.5, 14.25] < (14.25, 19.0]]

grouped_col_1 = df.col_1.groupby(factor_col_1)

grouped_col_1.agg(['count', 'mean', 'std', 'min', 'max'])

def summary_func(group):

    return {'count': group.count(), 

             'mean': group.mean(), 

             'std': group.std(), 

             'min': group.min(), 

             'max': group.max()}

grouped_col_1.apply(summary_func)

col_1                
(-0.019, 4.75]  count     8.000000
                max       4.000000
                mean      1.125000
                min       0.000000
                std       1.457738
(4.75, 9.5]     count     2.000000
                max       9.000000
                mean      7.500000
                min       6.000000
                std       2.121320
(9.5, 14.25]    count     4.000000
                max      14.000000
                mean     12.750000
                min      10.000000
                std       1.892969
(14.25, 19.0]   count     6.000000
                max      19.000000
                mean     17.000000
                min      15.000000
                std       1.788854
Name: col_1, dtype: float64

grouped_col_1.apply(summary_func).unstack()

  (2) pd.qcut() : 동일 개수로 나누어서 범주 만들기 (equal-size buckets categorization)

bucket_qcut_col_2 = pd.qcut(df.col_2, 4, labels=False)

bucket_qcut_col_2

0     3
1     3
2     0
3     2
4     1
5     3
6     2
7     0
8     1
9     3
10    3
11    0
12    0
13    1
14    1
15    1
16    2
17    2
18    0
19    2
Name: col_2, dtype: int64

bucket_qcut_label_col_2 = pd.qcut(df.col_2, 4, labels=np.arange(4, 0, -1))

bucket_qcut_label_col_2

0     1
1     1
2     4
3     2
4     3
5     1
6     2
7     4
8     3
9     1
10    1
11    4
12    4
13    3
14    3
15    3
16    2
17    2
18    4
19    2

Name: col_2, dtype: category
Categories (4, int64): [4 < 3 < 2 < 1]

grouped = df.col_2.groupby(bucket_qcut_label_col_2)

grouped.apply(summary_func).unstack()

import pandas as pd

import numpy as np

df = pd.DataFrame({'grp_col' : ['a', 'a', 'a', 'a', 'a', 'b', 'b', 'b', 'b', 'b'], 

                   'val' : np.arange(10)+1,                   

                   'weight' : [0.0, 0.1, 0.2, 0.3, 0.4, 0.0, 0.1, 0.2, 0.3, 0.4]})

df

  (1) GroupBy 를 활용하여 그룹 별 가중 평균 구하기

# group weighted average by category

grouped = df.groupby('grp_col')

weighted_avg_func = lambda g:np.average(g['val'], weights=g['weight'])

grouped.apply(weighted_avg_func)

grp_col
a    4.0
b    9.0
dtype: float64

 (2) 수작업으로 그룹 별 가중 평균 구하기 (Split -> Apply -> Combine) 

# split

df_a = df[df['grp_col']=='a']

df_b = df[df['grp_col']=='b']