R, Python 분석과 프로그래밍의 친구 (by R Friend)

In [1]: import pandas as pd

In [2]: df = pd.DataFrame({'id': [1, 2, 10, 20, 100, 200], 

   ...:                    'name': ['aaa', 'bbb', 'ccc', 'ddd', 'eee', 'fff']})

In [3]: df

Out[3]: 

    id name

0    1  aaa

1    2  bbb

2   10  ccc

3   20  ddd

4  100  eee

5  200  fff

  (1) 'id' 변수가 전체 5개 자리가 되도록 왼쪽에 비어있는 부분에 '0'을 채워서 새로운 변수 'id_2' 만들기

     (Left padding with zeros so that make 5 positions)

In [4]: df['id_2'] = df['id'].apply(lambda x: "{:0>5d}".format(x))

In [5]: df

Out[5]: 

    id      name   id_2

0      1   aaa      00001

1      2   bbb     00002

2    10   ccc     00010

3    20  ddd    00020

4  100  eee     00100

5  200  fff      00200

 (2) 새로 만든 'id_2' 변수와 'name' 변수를 각 원소별로 합쳐서 데이터프레임 안에

새로운 변수 'id_name' 만들기

    (element-wise string concatenation with multiple inputs array in pandas DataFrame)

In [6]: df['id_name'] = df[['id_2', 'name']].apply(lambda x: '_'.join(x), axis=1)

In [7]: df

Out[7]: 

       id    name   id_2         id_name

0      1    aaa      00001    00001_aaa

1      2    bbb     00002    00002_bbb

2    10    ccc     00010     00010_ccc

3    20   ddd    00020    00020_ddd

4  100   eee    00100     00100_eee

5  200   fff     00200     00200_fff 

(3) 'id' 변수의 값을 소숫점 두번째 자리까지 나타내는 새로운 변수 'id_3' 만들기

In [8]: df['id_3'] = df['id'].apply(lambda x: "{:.2f}".format(x))

In [9]: df

  (4) 'name' 변수의 문자열을 전부 대문자로 바꾼 새로운 변수 'name_3' 만들기

In [10]: df['name_3'] = df['name'].apply(lambda x: x.upper())

In [11]: df

  (5) 데이터프레임 안의 'id_3'와 'name_3' 변수를 합쳐서 새로운 변수 'id_name_3' 만들기

In [14]: df['id_name_3'] = df[['id_3', 'name_3']].apply(lambda x: ':'.join(x), axis=1)

In [15]: df

import pandas as pd

from pandas import DataFrame

df = DataFrame({'name': ['kim', 'KIM', 'Kim', 'lee', 'LEE', 'Lee', 'wang', 'hong'], 

                'value': [1, 2, 3, 4, 5, 6, 7, 8], 

                'value_2': [100, 300, 200, 100, 100, 300, 50, 80]

               })

df

  (1) 범주형 변수의 항목 매핑/변환에 사용할 기준 정보를 dict 자료형으로 만들기

name_mapping = {

    'KIM': 'kim',

    'Kim': 'kim', 

    'LEE': 'lee', 

    'Lee': 'lee', 

    'wang': 'others', 

    'hong': 'others'

}

name_mapping

 {'KIM': 'kim',

 'Kim': 'kim',
 'LEE': 'lee',
 'Lee': 'lee',
 'hong': 'others',
 'wang': 'others'}

  (2) dict.get() 함수를 이용하여 매핑/변환에 사용할 사용자 정의 함수 만들기

func = lambda x: name_mapping.get(x, x)

  (3) map() 함수로 매핑용 사용자 정의 함수를 DataFrame의 범주형 변수에 적용하여 매핑/변환하기

df['name_2'] = df.name.map(func)

df

  (4) groupby() 로 범주형 변수의 그룹별로 집계하기

# aggregation by name

df.groupby('name_2').sum()

df.groupby(['name_2', 'name'])['value_2'].sum()

name_2  name
kim     KIM     300
        Kim     200
        kim     100
lee     LEE     100
        Lee     300
        lee     100
others  hong     80
        wang     50
Name: value_2, dtype: int64

import numpy as np

import pandas as pd

abalone = pd.read_csv("/Users/ihongdon/Documents/Python/abalone.txt", 

                      sep=",", 

                      names = ['sex', 'length', 'diameter', 'height', 

                               'whole_weight', 'shucked_weight', 'viscera_weight', 

                               'shell_weight', 'rings'], 

                      header = None)

abalone['length_cat'] = np.where(abalone.length > np.median(abalone.length), 

                                 'length_long', 

                                 'length_short')

abalone.head()

for sex, group_data in abalone[['sex', 'length_cat', 'whole_weight', 'rings']].groupby('sex'):

    print sex, 

    print group_data[:5]

F    sex    length_cat  whole_weight  rings

2    F  length_short        0.6770      9
6    F  length_short        0.7775     20
7    F  length_short        0.7680     16
9    F   length_long        0.8945     19
10   F  length_short        0.6065     14

I    sex    length_cat  whole_weight  rings
4    I  length_short        0.2050      7
5    I  length_short        0.3515      8
16   I  length_short        0.2905      7
21   I  length_short        0.2255     10
42   I  length_short        0.0700      5

M    sex    length_cat  whole_weight  rings
0    M  length_short        0.5140     15
1    M  length_short        0.2255      7
3    M  length_short        0.5160     10
8    M  length_short        0.5095      9
11   M  length_short        0.4060     10

for (sex, length_cat), group_data in abalone[['sex', 'length_cat', 'whole_weight', 'rings']]\

.groupby(['sex', 'length_cat']):

    print sex, length_cat

    print group_data[:5]

F length_long
   sex   length_cat  whole_weight  rings
9    F  length_long        0.8945     19
22   F  length_long        0.9395     12
23   F  length_long        0.7635      9
24   F  length_long        1.1615     10
25   F  length_long        0.9285     11

F length_short
   sex    length_cat  whole_weight  rings
2    F  length_short        0.6770      9
6    F  length_short        0.7775     20
7    F  length_short        0.7680     16
10   F  length_short        0.6065     14
13   F  length_short        0.6845     10

I length_long
    sex   length_cat  whole_weight  rings
509   I  length_long        0.8735     16
510   I  length_long        1.1095     10
549   I  length_long        0.8750     11
550   I  length_long        1.1625     17
551   I  length_long        0.9885     13

I length_short
   sex    length_cat  whole_weight  rings
4    I  length_short        0.2050      7
5    I  length_short        0.3515      8
16   I  length_short        0.2905      7
21   I  length_short        0.2255     10
42   I  length_short        0.0700      5

M length_long
   sex   length_cat  whole_weight  rings
27   M  length_long        0.9310     12
28   M  length_long        0.9365     15
29   M  length_long        0.8635     11
30   M  length_long        0.9975     10
32   M  length_long        1.3380     18

M length_short
   sex    length_cat  whole_weight  rings
0    M  length_short        0.5140     15
1    M  length_short        0.2255      7
3    M  length_short        0.5160     10
8    M  length_short        0.5095      9
11   M  length_short        0.4060     10

abalone_sex_group = dict(list(abalone[:10][['sex', 'length_cat', 'whole_weight', 'rings']]

                              .groupby('sex')))

abalone_sex_group

{'F':   sex    length_cat  whole_weight  rings
 2   F  length_short        0.6770      9
 6   F  length_short        0.7775     20
 7   F  length_short        0.7680     16
 9   F   length_long        0.8945     19,

 'I':   sex    length_cat  whole_weight  rings
 4   I  length_short        0.2050      7
 5   I  length_short        0.3515      8,

 'M':   sex    length_cat  whole_weight  rings
 0   M  length_short        0.5140     15
 1   M  length_short        0.2255      7
 3   M  length_short        0.5160     10
 8   M  length_short        0.5095      9}

abalone_sex_group['M'] 

  (1) Series 생성 및 Series 원소 선택 (element selection, indexing)

# importing library

In [1]: import numpy as np

In [2]: import pandas as pd

# pd.Series with ndarrary data

In [3]: Seri = pd.Series([0., 1., 2., 3., 4.])

In [4]: Seri

Out[4]:

0    0.0
1    1.0
2    2.0
3    3.0
4    4.0
dtype: float64

# Slicing pd.Series like ndarray-like

In [5]: Seri[0]

Out[5]: 0.0

In [6]: Seri[:3]

Out[6]:

0    0.0
1    1.0
2    2.0
dtype: float64

In [7]: Seri[Seri >= Seri.mean()]

Out[7]:

2    2.0
3    3.0
4    4.0
dtype: float64

In [8]: Seri[[4, 2, 0]]

Out[8]:

4    4.0
2    2.0
0    0.0
dtype: float64

# pd.Series with index name passed

In [9]: Seri_ix = pd.Series([0., 1., 2., 3., 4.], index=['a', 'b', 'c', 'd', 'e'])

In [10]: Seri_ix

Out[10]:

a    0.0
b    1.0
c    2.0
d    3.0
e    4.0
dtype: float64

# Slicing with index label

In [11]: Seri_ix[['a', 'b', 'e']]

Out[11]:

a    0.0
b    1.0
e    4.0
dtype: float64

In [12]: Seri_ix.get(['a', 'b', 'e']) # get() method

Out[12]:

a    0.0
b    1.0
e    4.0
dtype: float64

# set values by index label
In [13]: Seri_ix['a'] = 100

In [14]: Seri_ix

Out[14]:

a    100.0
b      1.0
c      2.0
d      3.0
e      4.0
dtype: float64

# check index label whether it is or is'not in Series

In [15]: 'a' in Seri_ix

Out[15]: True

In [16]: 'x' in Seri_ix

Out[16]: False

  (2) DataFrame 행과 열 생성, 선택, 삭제 (creation, selection, drop of row and column)

# importing library and making an example DataFrame

In [18]: df = DataFrame({'C1': [0., 1., 2., 3.],

    ...: 'C2': [4., 5., 6., 7.],

    ...: 'C3': [8., 9., 10., np.nan]},

    ...: index=['R1', 'R2', 'R3', 'R4'])

    ...:

In [19]: df

Out[19]:

     C1   C2    C3
R1  0.0  4.0   8.0
R2  1.0  5.0   9.0
R3  2.0  6.0  10.0
R4  3.0  7.0   NaN

# the row and column labels

In [20]: df.index # row labels

Out[20]: Index(['R1', 'R2', 'R3', 'R4'], dtype='object')

In [21]: df.columns # column labels

Out[21]: Index(['C1', 'C2', 'C3'], dtype='object')

In [22]: df_R1R3 = DataFrame(df, index=['R1', 'R3'])

In [23]: df_R1R3

Out[23]:

     C1   C2    C3
R1  0.0  4.0   8.0
R3  2.0  6.0  10.0

In [24]: df_C1C3 = DataFrame(df, columns=['C1', 'C3'])

In [25]: df_C1C3

Out[25]:

     C1    C3
R1  0.0   8.0
R2  1.0   9.0
R3  2.0  10.0
R4  3.0   NaN

In [26]: df_R3R1_C3C1 = DataFrame(df, index=['R3', 'R1'], columns=['C3', 'C1'])

In [27]: df_R3R1_C3C1

Out[27]:

      C3   C1
R3  10.0  2.0
R1   8.0  0.0

# selecting columns from DataFrame

In [28]: df

Out[28]:

     C1   C2    C3
R1  0.0  4.0   8.0
R2  1.0  5.0   9.0
R3  2.0  6.0  10.0
R4  3.0  7.0   NaN

In [29]: df[['C1', 'C2']]

Out[29]:

     C1   C2
R1  0.0  4.0
R2  1.0  5.0
R3  2.0  6.0
R4  3.0  7.0

# (1) making a new column

In [30]: df['C4'] = df['C1'] + df['C2']

In [31]: df

Out[31]:

     C1   C2    C3    C4
R1  0.0  4.0   8.0   4.0
R2  1.0  5.0   9.0   6.0
R3  2.0  6.0  10.0   8.0
R4  3.0  7.0   NaN  10.0

# (2-1) assign() method

In [32]: df = df.assign(C5 = df['C1']*df['C2'])

In [33]: df

Out[33]:

     C1   C2    C3    C4    C5
R1  0.0  4.0   8.0   4.0   0.0
R2  1.0  5.0   9.0   6.0   5.0
R3  2.0  6.0  10.0   8.0  12.0
R4  3.0  7.0   NaN  10.0  21.0

# (2-2) the same with the above

In [34]: df.assign(C5 = lambda x: x.C1*x.C2)

Out[34]:

     C1   C2    C3    C4    C5
R1  0.0  4.0   8.0   4.0   0.0
R2  1.0  5.0   9.0   6.0   5.0
R3  2.0  6.0  10.0   8.0  12.0
R4  3.0  7.0   NaN  10.0  21.0

# drop 'C3' column : DataFrame.drop('Column', 1)

In [35]: df_drop_C4C5 = df.drop(['C4', 'C5'], 1)

In [36]: df_drop_C4C5

Out[36]:

     C1   C2    C3
R1  0.0  4.0   8.0
R2  1.0  5.0   9.0
R3  2.0  6.0  10.0
R4  3.0  7.0   NaN

# delete a column from original DataFrame : del DataFrame['column']

In [37]: df

Out[37]:

     C1   C2    C3    C4    C5
R1  0.0  4.0   8.0   4.0   0.0
R2  1.0  5.0   9.0   6.0   5.0
R3  2.0  6.0  10.0   8.0  12.0
R4  3.0  7.0   NaN  10.0  21.0

In [38]: del df['C4']  # delete 'C4' column from the original DataFrame df directly

In [39]: del df['C5']  # delete 'C5' column from the original DataFrame df directly

In [40]: df

Out[40]:

     C1   C2    C3
R1  0.0  4.0   8.0
R2  1.0  5.0   9.0
R3  2.0  6.0  10.0
R4  3.0  7.0   NaN

In [42]: df

Out[42]:

     C1   C2    C3
R1  0.0  4.0   8.0
R2  1.0  5.0   9.0
R3  2.0  6.0  10.0
R4  3.0  7.0   NaN

# selecting column form DataFrame

In [43]: df['C1']

Out[43]:

R1    0.0
R2    1.0
R3    2.0
R4    3.0
Name: C1, dtype: float64

In [44]: df.C1

Out[44]:

R1    0.0
R2    1.0
R3    2.0
R4    3.0
Name: C1, dtype: float64

# selecting row from DataFrame

In [45]: df[0:2]

Out[45]:

     C1   C2   C3
R1  0.0  4.0  8.0
R2  1.0  5.0  9.0

# indexing 'column' and 'row' from DataFrame

In [46]: df['C1'][0:2]

Out[46]:

R1    0.0
R2    1.0
Name: C1, dtype: float64

In [47]: df.C1[0:2]

Out[47]:

R1    0.0
R2    1.0
Name: C1, dtype: float64

# Select row by label : df.loc[label]

In [48]: df.loc['R1']

Out[48]:

C1    0.0
C2    4.0
C3    8.0
Name: R1, dtype: float64

In [49]: df.loc[['R1', 'R2']]

Out[49]:

     C1   C2   C3
R1  0.0  4.0  8.0
R2  1.0  5.0  9.0

# TypeError: cannot do label indexing on with these indexers [0] of <class 'int'>

In [50]: df.loc[0] # TypeError

TypeError: cannot do label indexing on <class 'pandas.indexes.base.Index'> with these indexers [0] of <class 'int'>

# Select row by interger location : df.iloc[loc]

In [51]: df.iloc[0]

Out[51]:

C1    0.0
C2    4.0
C3    8.0
Name: R1, dtype: float64

In [52]: df.iloc[0:2]

Out[52]:

     C1   C2   C3
R1  0.0  4.0  8.0
R2  1.0  5.0  9.0

# KeyError: 0

KeyError: 0

# Select rows : df[0:2]

In [54]: df[0:2]

Out[54]:

     C1   C2   C3
R1  0.0  4.0  8.0
R2  1.0  5.0  9.0

# Select rows by boolean vector : df[bool_vec]

Out[55]:

     C1   C2   C3
R1  0.0  4.0  8.0
R2  1.0  5.0  9.0

# Select columns by column vector : df[col_bool_vec]

In [56]: df_col_selector = ['C1', 'C2']

In [57]: df[df_col_selector]

Out[57]:

     C1   C2
R1  0.0  4.0
R2  1.0  5.0
R3  2.0  6.0
R4  3.0  7.0

  (1) DataFrame 정렬 : DataFrame.sort_values()

In [1]: import pandas as pd

In [2]: personnel_df = pd.DataFrame({'sequence': [1, 3, 2],

   ...: 'name': ['park', 'lee', 'choi'],

   ...: 'age': [30, 20, 40]})

In [3]: personnel_df

Out[3]:

   age  name  sequence
0   30  park         1
1   20   lee         3
2   40  choi         2

# sorting index of DataFrame by a specific column : axis=0, columns

In [4]: personnel_df.sort_values(by=['sequence'], axis=0)

Out[4]:

   age  name  sequence
0   30  park         1
2   40  choi         2
1   20   lee          3

# sorting index of dataFrame in descending order : ascending=False

In [5]: personnel_df.sort_values(by=['sequence'], axis=0, ascending=False)

Out[5]:

   age  name  sequence
1   20   lee         3
2   40  choi         2
0   30  park         1

# sorting columns of DataFrame : axis=1

In [6]: personnel_df.sort(axis=1)

Out[6]:

   age  name  sequence
0   30  park         1
1   20   lee         3
2   40  choi         2

# sorting columns of DataFrame in descending order : axis=1, ascending=False

In [7]: personnel_df.sort(axis=1, ascending=False) 

Out[7]:

   sequence  name  age
0         1  park   30
1         3   lee   20
2         2  choi   40

In [8]: personnel_df

Out[8]:

age name sequence

0 30 park 1

1 20 lee 3

2 40 choi 2

# sorting DataFarme in-place : inplace=True

In [9]: personnel_df.sort_values(by=['sequence'], axis=0, inplace=True)

In [10]: personnel_df

Out[10]:

age name sequence

0 30 park 1

2 40 choi 2

1 20 lee 3

# putting NaN to DataFrame

In [11]: import numpy as np

In [12]: personnel_df = pd.DataFrame({'sequence': [1, 3, np.nan],

    ...: 'name': ['park', 'lee', 'choi'],

    ...: 'age': [30, 20, 40]})

    ...:

In [13]: personnel_df

Out[13]:

   age  name  sequence
0   30  park       1.0
1   20   lee       3.0
2   40  choi       NaN

# first puts NaNs at the beginning : na_position='first'

In [14]: personnel_df.sort_values(by=['sequence'], axis=0, na_position='first')

Out[14]:

   age  name  sequence
2   40  choi       NaN
0   30  park       1.0
1   20   lee       3.0

# last puts NaNs at the end : na_position='last'

In [15]: personnel_df.sort_values(by=['sequence'], axis=0, na_position='last')

Out[15]:

   age  name  sequence
0   30  park       1.0
1   20   lee       3.0
2   40  choi       NaN

  (2) Tuple 정렬하기 : sorted(tuple, key) method

# making a tuple

In [16]: personnel_tuple = [(1, 'park', 30),

    ...: (3, 'lee', 20),

    ...: (2, 'choi', 40)]

In [17]: personnel_tuple

Out[17]: [(1, 'park', 30), (3, 'lee', 20), (2, 'choi', 40)]

# use 'key' parameter to specify a function to be called on

# sort by sequence number

In [18]: sorted(personnel_tuple, key=lambda personnel: personnel[0])

Out[18]: [(1, 'park', 30), (2, 'choi', 40), (3, 'lee', 20)]

# sort by name

In [19]: sorted(personnel_tuple, key=lambda personnel: personnel[1])

Out[19]: [(2, 'choi', 40), (3, 'lee', 20), (1, 'park', 30)]

# sort by age

In [20]: sorted(personnel_tuple, key=lambda personnel: personnel[2])

Out[20]: [(3, 'lee', 20), (1, 'park', 30), (2, 'choi', 40)]

# sorting tuple in descending order by age : reverse=True

In [21]: sorted(personnel_tuple, reverse=True, key=lambda personnel: personnel[2])

Out[21]: [(2, 'choi', 40), (1, 'park', 30), (3, 'lee', 20)]

  (3) List 정렬하기 : sorted(list), or list.sort()

# making a list

In [23]: my_list = [0, 1, 2, 3, 4, 9, 8, 7, 6, 5]

# (1) sorting a list : sort(list) function

In [24]: sorted(my_list)

Out[24]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

# (2) sorting a list : list.sort() method

In [25]: my_list.sort()

In [26]: my_list

Out[26]: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9]

# sorting a list in descending order : reverse=True

In [27]: sorted(my_list, reverse=True)

Out[27]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

In [28]: my_list.sort(reverse=True)

In [29]: my_list

Out[29]: [9, 8, 7, 6, 5, 4, 3, 2, 1, 0]

In [1]: import pandas as pd

In [2]: from pandas import DataFrame

In [3]: data = DataFrame({'id': ['id1', 'id1', 'id1', 'id2', 'id2', 'id3'],

   ...: 'fac_1': ['a', 'a', 'a', 'b', 'b', 'b'],

   ...: 'fac_2': ['d', 'd', 'd', 'c', 'c', 'd']})

In [4]: data

Out[4]:

    fac_1   fac_2    id
0     a       d       id1
1     a       d       id1
2     a       d       id1
3     b       c       id2
4     b       c       id2
5     b       d       id3

  (1) 교차표(contingency table, frequency table) 만들기 : pd.crosstab(index, columns)

# cross tabulations using pd.crosstab => contingency table

In [5]: pd.crosstab(data.fac_1, data.fac_2)

Out[5]:
fac_2  c  d
fac_1     
a      0  3
b      2  1

In [6]: pd.crosstab(data.id, data.fac_1)

Out[6]: 
fac_1  a  b
id        
id1    3  0
id2    0  2
id3    0  1

In [7]: pd.crosstab(data.id, data.fac_2)

Out[7]:
fac_2  c  d
id        
id1    0  3
id2    2  0
id3    0  1

  (2) Multi-index, Multi-level로 교차표 만들기 : pd.crosstab([id1, id2], [col1, col2])

# cross tabulations using pd.crosstab with Multi-level columns

In [8]: pd.crosstab(data.id, [data.fac_1, data.fac_2])

Out[8]:

fac_1  a  b  
fac_2  d  c  d
id           
id1    3  0  0
id2    0  2  0
id3    0  0  1

In [9]: pd.crosstab([data.fac_1, data.fac_2], data.id)

Out[9]:

id           id1  id2  id3
fac_1 fac_2              
a     d        3    0    0
b     c        0    2    0
      d        0    0    1

  (3) 교차표의 행 이름, 열 이름 부여 : pd.crosstab(rownames=['xx'], colnames=['aa'])

# pd.crosstab(rownames, colnames) : giving rownames, colnames

In [10]: pd.crosstab(data.id, [data.fac_1, data.fac_2],

    ...: rownames=['id_num'],

    ...: colnames=['a_b', 'c_d'])

Out[10]:

a_b     a  b  
c_d     d  c  d
id_num        
id1     3  0  0
id2     0  2  0
id3     0  0  1

  (4) 교차표의 행 합, 열 합 추가하기 : pd.crosstab(margins=True)

# pd.crosstab(margins=True) : adding row/column margins

In [11]: pd.crosstab(data.id, [data.fac_1, data.fac_2],

    ...: margins=True)

Out[11]:

fac_1  a  b    All
fac_2  d  c  d   
id               
id1    3  0  0   3
id2    0  2  0   2
id3    0  0  1   1
All    3  2  1   6

  (5) 구성비율로 교차표 만들기 : pd.crosstab(normalize=True)

# pd.corsstab(normalize=True)
# : Normalize by dividing all values by the sum of values

In [12]: pd.crosstab(data.id, [data.fac_1, data.fac_2],

    ...: normalize=True)

Out[12]:

fac_1    a         b         
fac_2    d         c         d
id                           
id1    0.5  0.000000  0.000000
id2    0.0  0.333333  0.000000
id3    0.0  0.000000  0.166667

In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: from pandas import DataFrame

In [4]: mul_index = pd.MultiIndex.from_tuples([('cust_1', '2015'), ('cust_1', '2016'),

   ...: ('cust_2', '2015'), ('cust_2', '2016')])

   ...:

In [5]: data = DataFrame(data=np.arange(16).reshape(4, 4),

   ...: index=mul_index,

   ...: columns=['prd_1', 'prd_2', 'prd_3', 'prd_4'],

   ...: dtype='int')

   ...:

In [6]: data

Out[6]:

                 prd_1  prd_2  prd_3  prd_4
cust_1 2015      0       1      2      3
         2016      4       5      6      7
cust_2 2015      8       9     10     11
         2016     12     13     14     15

  (1) pd.DataFrame.stack(level=-1, dropna=True)

# stack()

# DataFrame.stack() => returns Series

In [8]: data_stacked

Out[8]:

cust_1  2015  prd_1     0
                  prd_2     1
                  prd_3     2
                  prd_4     3
          2016  prd_1     4
                  prd_2     5
                  prd_3     6
                  prd_4     7
cust_2  2015  prd_1     8
                  prd_2     9
                  prd_3    10
                  prd_4    11
          2016  prd_1    12
                  prd_2    13
                  prd_3    14
                  prd_4    15

dtype: int32

# MultiIndex(levels) after stack()

In [9]: data_stacked.index

Out[9]:

MultiIndex(levels=[['cust_1', 'cust_2'], ['2015', '2016'], ['prd_1', 'prd_2', 'prd_3', 'prd_4']],

labels=[[0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1], [0, 0, 0, 0, 1, 1, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1], [0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3, 0, 1, 2, 3]])

# indexing

In [10]: data_stacked['cust_2']['2015'][['prd_1', 'prd_2']]

Out[10]:

prd_1 8

prd_2 9

dtype: int32

# # putting NaN to DataFrame

In [11]: data.ix['cust_2', 'prd_4'] = np.nan

In [12]: data

Out[12]:

             prd_1  prd_2  prd_3  prd_4
cust_1 2015      0      1      2    3.0
         2016      4      5      6    7.0
cust_2 2015      8      9     10    NaN
         2016     12     13     14    NaN

# stack with 'dropna=False' argument

In [13]: data.stack(dropna=False)

Out[13]:

cust_1  2015  prd_1     0.0
                  prd_2     1.0
                  prd_3     2.0
                  prd_4     3.0
          2016  prd_1     4.0
                  prd_2     5.0
                  prd_3     6.0
                  prd_4     7.0
cust_2  2015  prd_1     8.0
                  prd_2     9.0
                  prd_3    10.0
                  prd_4     NaN
          2016  prd_1    12.0
                  prd_2    13.0
                  prd_3    14.0
                  prd_4     NaN

dtype: float64

# stack with 'dropna=True' argument

In [14]: data.stack(dropna=True) # by default

Out[14]:

cust_1  2015  prd_1     0.0
                  prd_2     1.0
                  prd_3     2.0
                  prd_4     3.0
          2016  prd_1     4.0
                  prd_2     5.0
                  prd_3     6.0
                  prd_4     7.0
cust_2  2015  prd_1     8.0
                  prd_2     9.0
                  prd_3    10.0
          2016  prd_1    12.0
                  prd_2    13.0
                  prd_3    14.0

dtype: float64

  (2) pd.DataFrame.unstack(level=-1, fill_value=None)

In [15]: data_stacked

Out[15]:

cust_1  2015  prd_1     0
                  prd_2     1
                  prd_3     2
                  prd_4     3
          2016  prd_1     4
                  prd_2     5
                  prd_3     6
                  prd_4     7
cust_2  2015  prd_1     8
                  prd_2     9
                  prd_3    10
                  prd_4    11
          2016  prd_1    12
                  prd_2    13
                  prd_3    14
                  prd_4    15

dtype: int32

In [16]: data_stacked.unstack(level=-1)

Out[16]:

                 prd_1  prd_2  prd_3  prd_4
cust_1 2015      0      1      2      3
         2016      4      5      6      7
cust_2 2015      8      9     10     11
         2016     12     13     14     15

In [17]: data_stacked.unstack(level=0)

Out[17]:

                cust_1  cust_2
2015 prd_1       0       8
       prd_2       1       9
       prd_3       2      10
       prd_4       3      11
2016 prd_1       4      12
       prd_2       5      13
       prd_3       6      14
       prd_4       7      15

In [18]: data_stacked.unstack(level=1)

Out[18]:

                  2015  2016
cust_1 prd_1     0     4
         prd_2     1     5
         prd_3     2     6
         prd_4     3     7
cust_2 prd_1     8    12
         prd_2     9    13
         prd_3    10    14
         prd_4    11    15

# converting Series to DataFrame

In [19]: data_stacked_unstacked = data_stacked.unstack(level=-1)

In [20]: data_stacked_unstacked

Out[20]:

                prd_1  prd_2  prd_3  prd_4
cust_1 2015      0      1      2      3
         2016      4      5      6      7
cust_2 2015      8      9     10     11
         2016     12     13     14     15

# converting index to columns

In [21]: data_stacked_unstacked_df = data_stacked_unstacked.reset_index()

# changing columns' name

In [22]: data_stacked_unstacked_df.rename(columns={'level_0' : 'custID',

    ...: 'level_1' : 'year'}, inplace=True)

    ...:

In [23]: data_stacked_unstacked_df

Out[23]:

    custID  year  prd_1  prd_2  prd_3  prd_4
0  cust_1  2015      0      1      2      3
1  cust_1  2016      4      5      6      7
2  cust_2  2015      8      9     10     11
3  cust_2  2016     12     13     14     15

# importing libraries

In [1]: import numpy as np

In [2]: import pandas as pd

In [3]: from pandas import DataFrame

# making an example DataFrame

In [4]: data = DataFrame({'cust_id': ['c1', 'c1', 'c1', 'c2', 'c2', 'c2', 'c3', 'c3', 'c3'],

   ...: 'prod_cd': ['p1', 'p2', 'p3', 'p1', 'p2', 'p3', 'p1', 'p2', 'p3'],

   ...: 'grade' : ['A', 'A', 'A', 'A', 'A', 'A', 'B', 'B', 'B'],

   ...: 'pch_amt': [30, 10, 0, 40, 15, 30, 0, 0, 10]})

   ...:

In [5]: data

Out[5]:

  cust_id grade  pch_amt prod_cd
0      c1     A       30      p1
1      c1     A       10      p2
2      c1     A        0      p3
3      c2     A       40      p1
4      c2     A       15      p2
5      c2     A       30      p3
6      c3     B        0      p1
7      c3     B        0      p2
8      c3     B       10      p3

  (1) 데이터 재구조화 : data.pivot(index, columns, values)

# reshaping DataFrame by pivoting

In [6]: data_pivot = data.pivot(index='cust_id', columns='prod_cd', values='pch_amt')

In [7]: data_pivot

Out[7]:

prod_cd  p1  p2  p3
cust_id           
c1       30  10   0
c2       40  15  30
c3        0   0  10

  (2) 데이터 재구조화 : pd.pivot_table(data, index, columns, values, aggfunc)

# pd.pivot_table(data, index, columns, values, aggfunc)

In [8]: pd.pivot_table(data, index='cust_id', columns='prod_cd', values='pch_amt')

Out[8]:

prod_cd  p1  p2  p3
cust_id           
c1       30  10   0
c2       40  15  30
c3        0   0  10

# pivot() with 2 indices :ValueError

In [9]: data.pivot(index=['cust_id', 'grade'], columns='prod_cd', values='pch_amt')

ValueError: Wrong number of items passed 9, placement implies 2

# pd.pivot_table() with 2 indices : works well!

In [10]: pd.pivot_table(data, index=['cust_id', 'grade'], columns='prod_cd', values='pch_amt')

Out[10]:

prod_cd        p1  p2  p3
cust_id grade           
c1      A      30  10   0
c2      A      40  15  30
c3      B       0   0  10

# pivot() with 2 columns : KeyError

In [11]: data.pivot(index='cust_id', columns=['grade', 'prod_cd'], values='pch_amt')

KeyError: 'Level grade not found'

# pd.pivot_table() with 2 columns : works well!

Out[12]:

grade A B

grade       A                B          
prod_cd    p1    p2    p3   p1   p2    p3
cust_id                                 
c1       30.0  10.0   0.0  NaN  NaN   NaN
c2       40.0  15.0  30.0  NaN  NaN   NaN
c3        NaN   NaN   NaN  0.0  0.0  10.0

# pivot() with index which contains duplicate entries: ValueError

In [13]: data.pivot(index='grade', columns='prod_cd', values='pch_amt')

ValueError: Index contains duplicate entries, cannot reshape

# pd.pivot_table() with aggfunc : works well!

In [14]: pd.pivot_table(data, index='grade', columns='prod_cd',

    ...: values='pch_amt', aggfunc=np.sum)

Out[14]:

prod_cd  p1  p2  p3
grade             
A        70  25  30
B         0   0  10

In [15]: pd.pivot_table(data, index='grade', columns='prod_cd',

    ...: values='pch_amt', aggfunc=np.mean)

Out[15]:

prod_cd    p1    p2    p3
grade                   
A        35.0  12.5  15.0
B         0.0   0.0  10.0

# pivot_table(aggfunc=np.mean), by default

In [16]: pd.pivot_table(data, index='grade', columns='prod_cd', values='pch_amt')

Out[16]:

prod_cd    p1    p2    p3
grade                   
A        35.0  12.5  15.0
B         0.0   0.0  10.0

# pd.pivot_table : margins=True
  # special All columns and rows will be added with partial group aggregates
  # across the categories on the rows and columns

In [17]: pd.pivot_table(data, index='grade', columns='prod_cd',

    ...: values='pch_amt', aggfunc=np.sum, margins=True)

Out[17]:

prod_cd    p1    p2    p3    All
grade                          
A        70.0  25.0  30.0  125.0
B         0.0   0.0  10.0   10.0
All      70.0  25.0  40.0  135.0

In [18]: pd.pivot_table(data, index='grade', columns='prod_cd',

    ...: values='pch_amt', aggfunc=np.mean, margins=True)

Out[18]:

prod_cd         p1         p2         p3        All
grade                                             
A        35.000000  12.500000  15.000000  20.833333
B         0.000000   0.000000  10.000000   3.333333
All      23.333333   8.333333  13.333333  15.000000

In [1]: import pandas as pd

In [2]: import numpy as np

In [3]: df = pd.DataFrame({'A': ['A1', 'A1', 'A2', 'A2', 'A3', 'A3'],

   ...: 'B': ['B1', 'B1', 'B1', 'B1', 'B2', np.nan],

   ...: 'C': [1, 1, 3, 4, 4, 4]})

In [4]: df

Out[4]:

    A    B  C
0  A1   B1  1
1  A1   B1  1
2  A2   B1  3
3  A2   B1  4
4  A3   B2  4
5  A3  NaN  4

  (1) 유일한 값 찾기 : pd.Series.unique()

In [5]: df['A'].unique()

Out[5]: array(['A1', 'A2', 'A3'], dtype=object)

In [6]: df['B'].unique()

Out[6]: array(['B1', 'B2', nan], dtype=object)

In [7]: df['C'].unique()

Out[7]: array([1, 3, 4], dtype=int64)

  (2) 유일한 값별로 개수 세기 : pd.Series.value_counts()

# eturns object containing counts of unique values

pd.Series.value_counts(normalize=False, # False면 개수, True면 상대 비율 구함

                              sort=True, # True면 개수 기준 정렬, False면 유일한 값 기준 정렬

                              ascending=False, # False면 내림차순 정렬, True면 오름차순 정렬

                          bins=None, # None이면 유일값 기준 개수, None아니면 Bins Group별 개수

                              dropna=True # True면 NaN 무시, False면 유일값에 NaN 포함)

In [4]: df

Out[4]:

    A    B  C
0  A1   B1  1
1  A1   B1  1
2  A2   B1  3
3  A2   B1  4
4  A3   B2  4
5  A3  NaN  4

# returns of pd.Series.value_counts() by default setting

In [8]: df['A'].value_counts()

Out[8]:

A3    2
A2    2
A1    2

Name: A, dtype: int64

In [9]: df['B'].value_counts()

Out[9]:

B1    4
B2    1

Name: B, dtype: int64

In [10]: df['C'].value_counts()

Out[10]:

4    3
1    2
3    1

Name: C, dtype: int64

  (2-1) 유일 값 별 상대적 비율 : pd.Series.value_counts(normalize=True)

In [11]: df['C'].value_counts(normalize=True)

Out[11]:

4    0.500000
1    0.333333
3    0.166667

Name: C, dtype: float64

  (2-2) 유일한 값 기준 정렬 : pd.Series.value_counts(sort=True, ascending=True)

In [4]: df

Out[4]:

    A    B   C
0  A1   B1  1
1  A1   B1  1
2  A2   B1  3
3  A2   B1  4
4  A3   B2  4
5  A3  NaN  4

# sort descending order by value_counts

In [12]: df['C'].value_counts(sort=True, ascending=False) # by default

Out[12]:

4    3
1    2
3    1

Name: C, dtype: int64

# sort ascending order by value_counts

In [13]: df['C'].value_counts(sort=True, ascending=True)

Out[13]:

3    1
1    2
4    3

Name: C, dtype: int64

# Don't sort by value_counts, but sort by unique value

In [14]: df['C'].value_counts(sort=False)

Out[14]:

1    2
3    1
4    3

Name: C, dtype: int64

  (2-3) 결측값을 유일한 값에 포함할지 여부 : pd.Series.value_counts(dropna=True)

In [4]: df

Out[4]:

    A    B   C
0  A1   B1  1
1  A1   B1  1
2  A2   B1  3
3  A2   B1  4
4  A3   B2  4
5  A3  NaN  4

# dropna=True : Don’t include counts of NaN

In [15]: df['B'].value_counts(dropna=True) # by default

Out[15]:

B1    4
B2    1

Name: B, dtype: int64

# dropna=False : Include counts of NaN

In [16]: df['B'].value_counts(dropna=False)

Out[16]:

B1     4
B2     1
NaN    1

Name: B, dtype: int64

  (2-4) Bins Group별 값 개수 세기 : pd.Series.value_counts(bins=[ , , ,])

In [4]: df

Out[4]:

    A    B   C
0  A1   B1  1
1  A1   B1  1
2  A2   B1  3
3  A2   B1  4
4  A3   B2  4
5  A3  NaN  4

In [17]: df['C'].value_counts(bins=[0, 1, 2, 3, 4, 5], sort=False)

Out[17]:

0 2

1 0

2 1

3 3

4 0

Name: C, dtype: int64

In [18]: out = pd.cut(df['C'], bins=[0, 1, 2, 3, 4, 5])

In [19]: pd.value_counts(out)

Out[19]:

(3, 4] 3

(0, 1] 2

(2, 3] 1

(4, 5] 0

(1, 2] 0

Name: C, dtype: int64