R, Python 분석과 프로그래밍의 친구 (by R Friend)

[ UCI Machine Learning Repository ]

• Abalone Data Set 설명: http://archive.ics.uci.edu/ml/datasets/Abalone
• Abalone Data Set: http://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data
• Variables

Name		Data Type   Meas.   Description
----		---------   -----   -----------
Sex		nominal		    M, F, and I (infant)
Length		continuous   mm	    Longest shell measurement
Diameter	continuous   mm	    perpendicular to length
Height		continuous   mm	    with meat in shell
Whole weight	continuous   grams  whole abalone
Shucked weight	continuous   grams  weight of meat
Viscera weight	continuous   grams  gut weight (after bleeding)
Shell weight	continuous   grams  after being dried
Rings		integer		    +1.5 gives the age in years

# Importing common libraries

import numpy as np

import pandas as pd

# Import Abalone data set from UCI machine learning repository directly

import csv

import urllib2

url = 'https://archive.ics.uci.edu/ml/machine-learning-databases/abalone/abalone.data'

downloaded_data  = urllib2.urlopen(url)

abalone = pd.read_csv(downloaded_data, 

                      names = ['sex', 'length', 'diameter', 'height', 

                               'whole_weight', 'shucked_weight', 'viscera_weight', 

                               'shell_weight', 'rings'], 

                      header = None)

abalone.head()

abalone['length_cat'] = np.where(abalone.length > np.median(abalone.length), 

                                 'length_long', # True

                                 'length_short') # False

abalone[['length', 'length_cat']][:10]

 (1) 함수 리스트(List)를 사용하여 다수의 GroupBy 집계 함수를 동일한 칼럼에 적용하기

grouped_ww = abalone.groupby(['sex', 'length_cat'])['whole_weight']

grouped_ww

<pandas.core.groupby.SeriesGroupBy object at 0x10a7e0290> 

(방법1) GroupBy methods

(방법2) grouped.agg(function)

grouped_ww.mean() # Series

sex  length_cat  
F    length_long     1.261330
     length_short    0.589702
I    length_long     0.923215
     length_short    0.351234
M    length_long     1.255182
     length_short    0.538157
Name: whole_weight, dtype: float64

grouped_ww.agg('mean') # Series

sex  length_cat  
F    length_long     1.261330
     length_short    0.589702
I    length_long     0.923215
     length_short    0.351234
M    length_long     1.255182
     length_short    0.538157
Name: whole_weight, dtype: float64

grouped_ww.agg(['size', 'mean', 'std', 'min', 'max']) # DataFrame

function_list = ['size', 'mean', 'std', 'min', 'max']

grouped_ww.agg(function_list)

grouped = abalone.groupby(['sex', 'length_cat'])

function_list = ['size', 'mean', 'std']

groupby_result = grouped['whole_weight', 'shell_weight'].agg(function_list)

groupby_result

groupby_result['shell_weight']

groupby_result['shell_weight'][['size', 'mean']]

groupby_result.loc['M']

groupby_result.loc['M', 'shell_weight']

 (2) 칼럼과 함수를 매핑한 Dict를 사용하여 칼럼별로 특정 GroupBy 집계 함수를 적용하기 

def range_func(x):

    max_val = np.max(x)

    min_val = np.min(x)

    range_val = max_val - min_val

    return range_val

def iqr_func(x):

    q3, q1 = np.percentile(x, [75, 25])

    iqr = q3 - q1

    return iqr

grouped.agg({'whole_weight': ['size', 'mean', 'std'], # put method's name as a string

            'shell_weight': [range_func, iqr_func]}) # UDF name

 (3) (이름, 함수)의 튜플 (Tuples of (name, function))을 사용하여 GroupBy 집계 함수에 이름 부여하기

# (name, function) tuples

grouped.agg({'whole_weight': ['size', 'mean', 'std'], 

            'shell_weight': [('Range', range_func),  # (name, function) tuple

                                    ('Inter-Quartile_Range', iqr_func)]}) # (name, function) tuple

# Importing common libraries

import numpy as np

import pandas as pd

# sample DataFrame

df = pd.DataFrame({'group': ['a', 'a', 'a', 'b', 'b', 'b'], 

                  'value_1': np.arange(6), 

                 'value_2': np.random.randn(6)})

df

# Making GroupBy object

grouped = df.groupby('group')

grouped

<pandas.core.groupby.DataFrameGroupBy object at 0x11136f550>

  (1) GroupBy 메소드를 이용한 집계 (GroupBy aggregation using methods)

count(): 그룹 내 non-NA 개수 

sum(): 그룹 내 non-NA 합 

grouped.count()

grouped.sum() # DataFrame

grouped.sum()['value_2'] # Series

group

a   -0.012473
b    0.072545 

Name: value_2, dtype: float64

pd.DataFrame(grouped.sum()['value_2']) # DataFrame 

min(): 그룹 내 non-NA 값 중 최소값 

max(): 그룹 내 non-NA 값 중 최대값 

grouped.min()

grouped.max()

mean(): 그룹 내 non-NA 값들의 평균값 

median(): 그룹 내 non-NA 값들의 중앙값 

grouped.mean()

grouped.median()

std(): 그룹 내 표준편차

var(): 그룹 내 분산

quantile(): 그룹 내 분위수 

grouped.std() 

grouped.var()

 # interpolation

grouped.quantile(0.1) 

first(): 그룹 내 non-NA 값 중 첫번째 값 

last(): 그룹 내 non-NA 값 중 마지막 값 

grouped.first()

grouped.last()

describe(): 그룹 별 기술통계량 

- 옆으로 길게

 describe().T: 그룹 별 기술통계량 

- 세로로 길게

grouped.describe()['value_1']

grouped.describe()['value_1'].T

  (2) 함수를 이용한 GroupBy 집계: grouped.agg(function)

def iqr_func(x):

    q3, q1 = np.percentile(x, [75, 25])

    iqr = q3 - q1

    return iqr

grouped.aggregate(function) 

grouped.agg(function) 

grouped.aggregate(iqr_func)

grouped.agg(iqr_func)

grouped.quantile([0.75, 0.25])