'PL/Python을 도커 이미지를 이용하여 환경구성 하기'에 해당되는 글 1건

  1. 2018.08.13 [Greenplum DB] Greenplum DB, MADlib, PL/R, PL/Python을 Docker Image를 이용하여 환경구성 하기

이번 포스팅에서는 도커 허브(Docker Hub)에서 Greenplum Database(이하 GPDB)에 MADlib, PL/R, PL/Python이 설치된 Docker Image를 내려받아 분석 환경을 구성하는 방법을 소개하겠습니다. 


이번 포스팅에서 소개하는 gpdb-analytics 도커 이미지는 개인이 집이나 회사에서 GPDB로 MADlib, PL/R, PL/Python 사용하면서 테스트해보고 공부할 때 간편하게 사용할 수 있는 용도로 만든 것입니다. 



[사전 준비] Dokcer Install


Docker Image를 이용하여 GPDB+분석툴을 설치할 것이므로, 먼저 Docker Hub (https://hub.docker.com/)에서 회원가입을 하고, https://www.docker.com/products/docker-desktop 사이트에서 자신의 OS에 맞는 Docker를 다운로드 받아 설치하시기 바랍니다. 




단, Windows OS 사용자의 경우는 (1) Windows 10 Professional or Enterprise 64-bit 의 경우 'Docker CE for Windows'를 다운받아 설치하시구요, 그 이전 버전 혹은 Home Edition 버전 Windows OS 이용하시는 분의 경우는 'Docker Toolbox'를 다운로드 받아서 설치하시기 바랍니다. 





[ Docker Hub에서 gpdb-analytics 도커 이미지 내려받아서 GPDB 분석 환경 구성하기 ]


1. Docker Hub에서 gpdb-analytics 도커 이미지 내려받기 (docker pull)





(터미널 사용)

## Docker 이미지 내려 받기

$ docker pull hdlee2u/gpdb-analytics

## Docker 이미지 확인 하기

$ docker images

REPOSITORY               TAG                 IMAGE ID            CREATED             SIZE
centos                     7                   d123f4e55e12        9 months ago        197MB
hdlee2u/gpdb-base        latest              bfe4e63b8e81         2 years ago           1.17GB

hdlee2u/gpdb-analytics  latest           3be773a1a7e1        About a minute ago   4.93GB

 




2. 도커 이미지를 실행하여 Docker Container 를 생성하고, GPDB 분석 환경 시작하기



## Docker 이미지를 실행/ 5432 기본 포트로, ssh 2022포트를 사용하여 접근 가능하도록 Docker 컨테이너 생성

docker run -i -d -p 5432:5432 -p 28080:28080 --name gpdb-ds --hostname mdw hdlee2u/gpdb-analytics /usr/sbin/sshd -D


## Docker 컨테이너 목록 확인

$ docker ps

CONTAINER ID    IMAGE                   COMMAND                   CREATED             STATUS              PORTS                   NAMES

7518fd48450a        575a7d45999d        "/bin/bash"              1 minute ago         Up 6 hours                  0.0.0.0:5432->5432/tcp, 0.0.0.0:28080->28080/tcp   gpdb-ds



## Start GPDB and Use psql


$ docker exec -it gpdb-ds /bin/bash

[root@mdw /]# su - gpadmin
[gpadmin@mdw ~]$ gpstart -a

20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Starting gpstart with args: -a
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Gathering information and validating the environment...
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Greenplum Binary Version: 'postgres (Greenplum Database) 5.10.2 build commit:b3c02f3acd880e2d676dacea36be015e4a3826d4'
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Greenplum Catalog Version: '301705051'
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[WARNING]:-postmaster.pid file exists on Master, checking if recovery startup required
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Commencing recovery startup checks
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Have lock file /tmp/.s.PGSQL.5432 but no process running on port 5432
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-No Master instance process, entering recovery startup mode
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Clearing Master instance lock files
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Clearing Master instance pid file
20180821:04:45:08:000043 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20180821:04:45:10:000043 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20180821:04:45:10:000043 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20180821:04:45:10:000043 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20180821:04:45:10:000043 gpstart:mdw:gpadmin-[INFO]:-Commencing forced instance shutdown
20180821:04:45:12:000043 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance in admin mode
20180821:04:45:13:000043 gpstart:mdw:gpadmin-[INFO]:-Obtaining Greenplum Master catalog information
20180821:04:45:13:000043 gpstart:mdw:gpadmin-[INFO]:-Obtaining Segment details from master...
20180821:04:45:13:000043 gpstart:mdw:gpadmin-[INFO]:-Setting new master era
20180821:04:45:13:000043 gpstart:mdw:gpadmin-[INFO]:-Master Started...
20180821:04:45:13:000043 gpstart:mdw:gpadmin-[INFO]:-Shutting down master
20180821:04:45:14:000043 gpstart:mdw:gpadmin-[INFO]:-Commencing parallel segment instance startup, please wait...
...
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-Process results...
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-   Successful segment starts                                            = 2
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-   Failed segment starts                                                = 0
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-   Skipped segment starts (segments are marked down in configuration)   = 0
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-Successfully started 2 of 2 segment instances
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-----------------------------------------------------
20180821:04:45:17:000043 gpstart:mdw:gpadmin-[INFO]:-Starting Master instance mdw directory /data/master/gpseg-1
20180821:04:45:18:000043 gpstart:mdw:gpadmin-[INFO]:-Command pg_ctl reports Master mdw instance active
20180821:04:45:18:000043 gpstart:mdw:gpadmin-[INFO]:-No standby master configured.  skipping...
20180821:04:45:18:000043 gpstart:mdw:gpadmin-[INFO]:-Database successfully started
[gpadmin@mdw ~]$
[gpadmin@mdw ~]$

[gpadmin@mdw ~]$ psql
psql (8.3.23)
Type "help" for help.

gpadmin=# \dn
List of schemas
Name | Owner
--------------------+---------
gp_toolkit | gpadmin
information_schema | gpadmin
madlib | gpadmin
pg_aoseg | gpadmin
pg_bitmapindex | gpadmin
pg_catalog | gpadmin
pg_toast | gpadmin
public | gpadmin
(8 rows)

gpadmin=# \q




3. PGAdmin IV 로 GPDB 연결하기


GPDB 5.x 버전에서 SQL Query를 할 때 PGAdmin IV 를 사용합니다. (GPDB 5.x 버전에서는 PGAdmin III 는 작동하지 않으며, 반대로 GPDB 4.x 버전에서는 PGAdmin IV가 작동하지 않고 대신 PGAdmin III만 작동합니다)


PGAdmin IV 는 https://www.pgadmin.org/download/ 에서 다운로드 하여 설치하면 됩니다. 


  • Host : localhost
  • Port : 5432
  • Maintenance DB : gpadmin
  • Username : gpadmin
  • Password : pivotal
  • Group: Servers
  • Ternel port: 22






4. Jupyter Notebook으로 GPDB 연결하기


4-0. Python 2.7 version의 Anaconda 설치


https://www.anaconda.com/download 에서 자신의 OS에 맞는 Python 2.7 version의 Anaconda를 설치합니다. GPDB 5.x 버전은 Python 2.7 version을 지원합니다. 




Anaconda를 설치하였으면, Anaconda Navigator를 실행한 후 base(root) 환경(즉, python 2.7)에서 'Jupyter Notebook'의 Launch 단추를 눌러서 Jupyter Notebook을 실행합니다. 





터미널을 이용해서 가상환경을 조회, 선택하고 Jupyter Notebook을 실행할 수도 있습니다. 




# 가상 환경 리스트 조회

$ conda env list

 

# 가상 환경 선택 (가상환경 리스트에서 python 2.7 버전 선택, windows의 경우: activate env_name)

$ source activate env_name


# Jupyter Notebook 실행

$ jupyter notebook






4-1. pip install 로 추가로 필요한 Python 패키지 설치하기



(터미널에서)

$ pip install --upgrade pip


$ pip install psycopg2

$ pip install sqlalchemy

$ pip install sql_magic

$ pip install math

$ pip install textwrap

$ pip install os

$ pip install Ipython


$ pip install ipywidgets

$ jupyter nbextension enable --py widgetsnbextension

$ pip install pygraphviz





4-2.  Jupyter Notebook에서 DB Connection 설정하기

4-2-1. Python packages importing



# Common modules

import numpy as np

import pandas as pd

from pandas import DataFrame

from pandas import Series

import sklearn

import math

import textwrap as tw


# For DB Connecton

import psycopg2

from sqlalchemy import create_engine

import sql_magic


# For reproducibility

np.random.seed(2622)


# Directory

import os

 



4-2-2. Visualization Parms Setup



import matplotlib as mpl

import matplotlib.pyplot as plt

import seaborn as sns


# To draw plots in jupyter notebook

#%matplotlib inline

%pylab inline


from pylab import rcParams

rcParams['figure.figsize'] = 12, 8

rcParams['axes.labelsize'] = 14

rcParams['xtick.labelsize'] = 12

rcParams['ytick.labelsize'] = 12


pd.set_option('display.max_columns', None)

pd.set_option('display.max_colwidth', 1000)


# Display

import ipywidgets as widgets

import IPython.display as ipd

from IPython.display import display


# interpret string as markdown

def printmd(string):

    ipd.display(ipd.Markdown(string))


# seaborn style

sns.set(style="darkgrid")




4-2-3. Greenplum Database Connection Setup



# put your own GPDB information

user = 'gpadmin'

password = 'pivotal'

host = 'localhost'

db = 'gpadmin'


connection_string = "postgresql://{user}:{password}@{host}/{db}".\

    format(user=user, 

           password=password, 

           host=host, 

           db=db)

    

conn = psycopg2.connect(connection_string)

cur = conn.cursor()

conn.autocommit = True




# helper function

def query_gpdb(query): 


    cur.execute(query)


    colnames = [desc[0] for desc in cur.description]

    return DataFrame(cur.fetchall(), columns=colnames)

 




4-2-4. sql_magic Setup

https://github.com/pivotal-legacy/sql_magic

sql_magic is Jupyter magic for writing SQL to interact with Greenplum/PostgreSQL database, Hive and Spark. Query results are saved directly to a Pandas dataframe.



# sql_magic package and ext sql_magic to query GPDB

%load_ext sql_magic

#%reload_ext sql_magic


# sql_magic

postgres_engine = create_engine(connection_string)


%config SQL.conn_name = 'postgres_engine'


# '%execsql' for sql execution, 

# '%read_sql' for reading table as a DataFrame format

from IPython.core.magic import (register_line_magic, register_cell_magic, register_line_cell_magic)

@register_cell_magic

def execsql(line, cell):

       _ = postgres_engine.execute(cell)

       return

 




드디어 GPDB를 개인 컴퓨터에서 테스트, 공부용으로 간편하게(? ^^;) 설치하여 보았습니다. 수고 많으셨습니다. 




Jupyter Notebook 에서 sql 매직 언어로 DB를 조회할 수 있습니다. 


# GPDB 버전 확인

%read_sql select version();



# GPDB instance 확인 (하나의 서버에 1개 master, 2개 segment가 설치된 경우임)

%read_sql select * from pg_catalog.gp_segment_configuration 



# MADlib version 확인

%read_sql select madlib.version();



# PL/Languages 확인


sql query가 두 줄 이상일 경우 %%read_sql 처럼 % 두개를 앞에 써줍니다. (sql query 가 한 줄일 경우 %read_sql)

%%read_sql 

select * 

    from pg_catalog.pg_language;



# Table 생성

%%execsql

drop table if exists tmp;

create table tmp (

    id int, 

    var1 varchar(10)

    );




5. Docker Container 중단, 재시작, 작동 중인 컨테이너 목록 확인



## Docker  컨테이너 중단, 재시작, 목록 확인

$ docker stop gpdb-ds

$ docker start gpdb-ds

$ docker ps

CONTAINER ID    IMAGE                   COMMAND                   CREATED             STATUS              PORTS                   NAMES
7518fd48450a        575a7d45999d        "/bin/bash"              2 minutes ago         Up 6 hours                  0.0.0.0:5432->5432/tcp, 0.0.0.0:28080->28080/tcp   gpdb-ds

 



많은 도움이 되었기를 바랍니다. 



---------------

혹시 아래와 같은 에러가 발생하는 경우 http://rfriend.tistory.com/396 포스팅을 참고하세요. 



Error response from daemon: driver failed programming external connectivity on endpoint gpdb-ds (d519c381360008f0ac0e8d756e97aeb0538075ee1b7e35862a0eaedf887181f1): Error starting userland proxy: Bind for 0.0.0.0:5432 failed: port is already allocated

Error: failed to start containers: gpdb-ds 





728x90
반응형
Posted by Rfriend
,