멋지게 놀아라

브롬톤 일렉트릭 국토종주 (남한강, 새재)

★깜장별★ — Wed, 9 Oct 2024 15:51:06 +0900

첫째 군대 가기 전 제주도 일주를 생각하던 중 브롬톤 일렉을 알게 되었고, 자전거를 몇 년동안 타지 않았던 터라 일렉 1대, 일반 1대를 구매해서 3박4일로 제주도 일주를 가려고 마음을 먹었었지요.

그런데 첫 째 놈이 체력이 안될 것 같다고 해서 제주도 일주는 무산되고, 브롬톤 일렉 한대만 구매해서 국토 종주나 해 보자 해서 구매를 했습니다.

워낙 자전거를 안 탔었어서, 업힐에서 도움을 받을 요량으로 구매를 한 것입니다.

구매는 판교 에잇플레이스에서 구매를 하였으며, 구매 시 여행용 큰 가방도 같이 구매를 하였습니다.

사장님이 매우 친절하십니다. 설명도 친절히 해 주시고 ~~

스펙상으로는 70km 정도를 서포트 해 준다고 하는데, 일단 체력도 올려야 하고 실제 점검도 해야 해서

집에서 20, 30, 40, 60 키로 정도를 타면서 체력과 베터리 테스트를 진행해 보았습니다.

1단은 거의 느끼지 못할 정도이고, 2단 정도는 폐달링을 쉬었다가 재 출발시 평지에서 도움은 확실히 느낄 정도입니다.

주행 거리 테스트는 집에서 잠실 왕복 60km 가 조금 넘는데 2단 서포트로 탈 경우 약 15km 정도에서 1칸씩 줄어 든다는 느낌이 들었습니다. 집에 도착하면 2칸이 남는 상태 (기어비는 내장3단 외장 1,2단을 사용. 밑에서는 내장기어-외장기어 비를 3-1, 3-2로 표현할 예정)

탄천 자전거 도로에서 집에 올때 살짝 경사도가 있는데 3-1을 사용하면 도움이 살짝 덜하고, 3-2를 해야 확실히 도움을 느낄 수 있습니다.

일단 주행 거리 테스트는 끝났다고 생각해서 국토 종주를 가보았습니다.

복장은 후드티 + 트레이닝 반바지 조합입니다. (바람 저항 따위는 개나 줘 벼려 복장입니다.)

첫날 : 양평역 ~ 탄금대

둘째날 : 탄금대 ~ 이화령 ~ 충주역

둘째날 이화령에서 다시 충주역으로 복귀하는 루틴을 짠 것은 충주역에서 판교역까지 KTX가 있어서라는 단순한 이유 때문입니다. (고속버스 짐칸에 아직 브롬톤을 태우기에는 ~~)

베터리 + 앞 가방이 출발 전 무게를 재어 보니 약 7kg 정도, 브롬톤 무게까지 합하면 대략 자전거 무게가 23kg 정도 되고, 몸무게가 62kg 정도이니 모터가 85kg을 끌어야 하네요 ~

첫날 계획은 이렇게 설정하고 갔습니다.

대략 60km를 타도 2칸 정도 남으니, 전체 거리가 약 110km 정도 되니

1. 양평에서 이포보 가기 전 처음 업힐에 베터리 사용

2. 양평 ~ 강천보 (대략 40km 조금 넘음) 까지는 그냥 라이딩, 이후 체력보고 베터리 사용

위와 같이 단순한 계산을 하고 라이딩을 시작하였는데, 베터리가 생각보다 빨리 소모 되는 것을 느껴서 강천보 이후 계속 키고 달리는 것은 오산이였습니다.

베터리 도움을 받은 구간 (숙소 도착 시 1칸 남음)

1. 양평 이후 첫 업힐 (3-2)

2. 중간 중간 자잘한 짧은 업힐 (3-2) - 아래 가민 로그에서 뽈록 솟은 구간들

3. 강천보에서 비내섬 가는 구간 베터리 켜고 가다가 순식간에 2칸 다는걸 경험하고 바로 끔 (3-2)

4. 목행교 전 지루한 구간에서 힘들기도 하고 빨리 탈출하고 싶어서 (3-1)

5. 충주댐 업힐 양방향 (3-2)

실제 주행을 해 보니 짧은 업힐이지만 모터 개입이 평지와 다르게 업힐의 경우 베터리 소모가 더 되는 것 같습니다.

첫날 탄금대 까지 도장찍고 예약했던 숙소에 도착하니 오후 2시 40분 정도 되었습니다. (출발은 6시 40분 정도)

둘째 날은 7시30분쯤 숙소를 나와서 이화령 까지 왕복 라이딩을 생각하고 KTX는 5시 표로 예매를 해 놓고 라이딩을 합니다.

어제 베터리 소모를 경험해 봐서 오늘도 역시 왠만한 평지는 그냥 타고, 공도에서 힘들 경우 , 이화령 전 고개 업힐 시 , 이화령 업힐 시 베터리를 사용하려고 마음을 먹고 라이딩 합니다.

그러나 역시 공도는 호락호락 하지 않지요 ....

베터리 도움을 받은 구간

1. 수안보 가는 중 계속되는 은근한 경사도 구간 (1~2% 경사도 구간)

2. 이름 모를 첫번째 업힐 구간

3. 이화령 업힐

4. 자잘한 짧은 경사

위와 같이 탔음에도 이화령을 올라가니 베터리가 2칸 남았습니다. (정확하게는 이호령 업힐 중간 즈음에서 2칸으로 변경)

이화령은 3-2로 올라기지 않고, 그나마 베터리를 적게 사용하고자 3-1로 올랐는데 꾸역 꾸역 발로 밟으면서 올라갔습니다.

왜냐하면 첫번째 업힐 이후 상당한 내리막을 경험해서, 역방향으로 다시 충주를 가야 하는데 업힐에도 쓸 베터리를 아껴야 했기 때문입니다.

그렇게 아낀 베터리는 다시 넘어야 하는 고개 3/5 지점에서 고개 넘는데 사용하고, 수주팔봉 야영지 부터 유주막 다리 밑 까지 켜고 끄고를 반복 하면서 충주역에 1시 30분쯤 베터리 하나(라이딩 중 한칸으로 변경 됨) 남기고 도착.

2시20분 KTX로 변경 후 복귀 하였습니다.

라이딩 중 느낀 것은 베터리 인디케이터가 켜 있으면, 모터 어시스트가 1,2,3 이 아니어도 내리막에서 저항이 있는거 같은 느낌이 들기는 했습니다.

뒷 바퀴 폴딩 후 다시 폈을 때는 인디케이터가 꺼지게 되는데 이화령에서 다운 시 좀더 부드러운 다운힐이 었다면, 첫번째 작은 고개 업힐 후 다운 시에는 쉬지 않고 업힐 후 다운을 했기에 인디케이터는 켜 있고, 어시스트는 꺼 있는 상태였는데 이때는 다운 시에 저항이 생기는 느낌이었습니다.

그래서 이번 라이딩 중에 느낀 건데, 기본으로 베터리를 장착하고 라이딩을 하게 되면 인디케이터가 점등이 되는데(어시스트를 켜지 않아도) , 이 인디케이터 자체를 라이딩 하면서 끌수 있었으면 하는 생각이 들더라구요.

그렇다고 라이딩 전 베터리 팩을 장착하고, 뒷 바퀴를 폴딩 시켜서 인디케이터가 꺼 지는 것을 보고 다시 언 폴딩 해서 라이딩을 할 수는 없으니 ...

물론 개인적인 느낌일 수 있겠지만, 저는 그렇게 느꼈습니다.

다운 시에 속도 유지가 확실 히 잘 안된는 것 같다는 .... 그런 느낌이 생겨서 인지 평지 라이딩시에도 24km 부터는 왠지 저항이 생기는 거 같은 느낌적인 느낌이 자꾸 듭니다.

어시스트 도움(2단)을 받으면 22km 까지 정도는 확실히 초반에 밀어 주는데, 24~25 넘을려면 힘을 좀 써야 합니다.

물론 제가 자전거를 오랜만에 타서 체력이 없어서 그럴 수도 있지만 .... 어시스트가 25km 까지는 지원을 한다고 하는데 ...

약간 잡는 느낌이 있지 않나... 하는 (개인적인 라이딩 후 느낌입니다)

하지만 업힐에서는 도움이 되는 것은 확실하니 ~~

혹시 모르니 베터리 인디케이터(전원?) 자체도 끄고 켤수 있게 해 주면 좋겠네요 ^^

라이딩 중에 밥 먹는것을 싫어해서 이번 라이딩을 위해서 오랜만에 파워젤 12개를 사서 6개 들고 갔는데

하루에 3개씩 먹은 것도 나름 도움이 되었던 것 같네요 ~~

첫날은 쉬면서 먹어 주고 , 둘째 날은 KTX 타야 하니 달리면서 먹어주고 ~~

그런데 파워젤 먹을때 마다 소세지 처럼 그냥 둥글고 길게 해 놓으면 안되나 하는 생각이 자꾸 드네요..

이게 달리며서 짜 먹기가 힘듭니다. 남은거 쪽쪽 빨아도 안 올라오고 ㅋㅋㅋ

넓게 되어 있어서 손으로 그냥 짜 먹으면 깔끔하게 먹어지지 않고 .. 뭐 여튼 그랬습니다.

혹시 참고들 하시라고 사진 한장 올립니다. (제가 써칭해서 산 제품입니다. 맛도 괜찮았어요 오랜만에 파워젤 먹는데 ~~)

바인드 변수 쿼리 플랜 생성 로직

★깜장별★ — Mon, 10 Jun 2024 14:36:34 +0900

bind 변수를 이용하는 쿼리를 사용시 plan_cache_mode 설정 값에 따라서 플랜을 생성하는 것이 다르다.

plan_cache_mode의 기본 값은 auto이고, 줄 수 있는 옵션은 다음과 같다

force_generic_plan : 바인드 값을 참고하지 않고 일반적인 plan 생성

force_custom_plan : 오라클의 bind peeking이나 SQL Server의 sniffed parameter 처럼 바인드 값에 따라서 다르게 플랜을 생성한다.

auto : 동일 세션에서 수행되는 바인드 SQL인 경우 5번 까지는 force_custom_plan 처럼 바인드 변수 값을 이용해서 플랜을 생성한다. 이후 6번째 수행 부터는 바인드 값을 참조 하지 않는 플랜과 기존 바인드 값을 참조한 플랜들의 코스트를 비교해서 플랜을 생성하며, generic plan의 코스트가 낮은 경우(generic plan이 선택) 이후 부터는 cost 비교를 하지 않고 generic plan으로 고정 됨

테스트 환경 구성

테스트 데이터 생성

create table test
as
select (row_number() over()) as id ,
        md5(random()::text) as name ,
        gs as input_date
from   generate_series('2020-01-01 00:00'::timestamp, '2024-06-10 23:00'::timestamp, '1 minute') AS gs;

create index test_input_date on test(input_date) ;

데이터 확인

Bind SQL

prepare stmt ( timestamp ) as select * from test where input_date > $1 ;

테스트 #1

파라미터 확인

test=# show plan_cache_mode ;
+-----------------+
| plan_cache_mode |
+-----------------+
| auto |
+-----------------+

test=# explain execute stmt ('2025-06-01 00:00') ;
+--------------------------------------------------------------------------------+
|                                   QUERY PLAN                                   |
+--------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..4.45 rows=1 width=49)    |
|   Index Cond: (input_date > '01-JUN-25 00:00:00'::timestamp without time zone) |
+--------------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2022-01-01 00:00') ;
+---------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                       |
+---------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..49951.72 rows=1287902 width=49) |
|   Index Cond: (input_date > '01-JAN-22 00:00:00'::timestamp without time zone)        |
+---------------------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2020-01-01 00:00') ;
+----------------------------------------------------------------------------+
|                                 QUERY PLAN                                 |
+----------------------------------------------------------------------------+
| Seq Scan on test (cost=0.00..53307.26 rows=2337060 width=49)              |
|   Filter: (input_date > '01-JAN-20 00:00:00'::timestamp without time zone) |
+----------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2021-01-01 00:00') ;
+----------------------------------------------------------------------------+
|                                 QUERY PLAN                                 |
+----------------------------------------------------------------------------+
| Seq Scan on test (cost=0.00..53307.26 rows=1818857 width=49)              |
|   Filter: (input_date > '01-JAN-21 00:00:00'::timestamp without time zone) |
+----------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2021-01-01 00:00') ;
+----------------------------------------------------------------------------+
|                                 QUERY PLAN                                 |
+----------------------------------------------------------------------------+
| Seq Scan on test (cost=0.00..53307.26 rows=1818857 width=49)              |
|   Filter: (input_date > '01-JAN-21 00:00:00'::timestamp without time zone) |
+----------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2021-01-01 00:00') ;
+--------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                      |
+--------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..30216.28 rows=779020 width=49) |
|   Index Cond: (input_date > $1)                                                      |
+--------------------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2021-01-01 00:00') ;
+--------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                      |
+--------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..30216.28 rows=779020 width=49) |
|   Index Cond: (input_date > $1)                                                      |
+--------------------------------------------------------------------------------------+

테스트 #2

위의 동일 세션에서 plan_cache_mode 값을 변경

force_custom_plan

test=# set plan_cache_mode = force_custom_plan ;
test=# show plan_cache_mode ;
+-------------------+
| plan_cache_mode |
+-------------------+
| force_custom_plan |
+-------------------+
(1 row)

test=# explain execute stmt ('2021-01-01 00:00') ;
+----------------------------------------------------------------------------+
|                                 QUERY PLAN                                 |
+----------------------------------------------------------------------------+
| Seq Scan on test (cost=0.00..53307.26 rows=1818857 width=49)              |
|   Filter: (input_date > '01-JAN-21 00:00:00'::timestamp without time zone) |
+----------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2021-01-01 00:00') ;
+----------------------------------------------------------------------------+
|                                 QUERY PLAN                                 |
+----------------------------------------------------------------------------+
| Seq Scan on test (cost=0.00..53307.26 rows=1818857 width=49)              |
|   Filter: (input_date > '01-JAN-21 00:00:00'::timestamp without time zone) |
+----------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2022-01-01 00:00') ;
+---------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                       |
+---------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..49951.72 rows=1287902 width=49) |
|   Index Cond: (input_date > '01-JAN-22 00:00:00'::timestamp without time zone)        |
+---------------------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2023-01-01 00:00') ;
+--------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                      |
+--------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..29507.31 rows=760736 width=49) |
|   Index Cond: (input_date > '01-JAN-23 00:00:00'::timestamp without time zone)       |
+--------------------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2024-01-01 00:00') ;
+-------------------------------------------------------------------------------------+
|                                     QUERY PLAN                                      |
+-------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..9065.17 rows=233642 width=49) |
|   Index Cond: (input_date > '01-JAN-24 00:00:00'::timestamp without time zone)      |
+-------------------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2020-01-01 00:00') ;
+----------------------------------------------------------------------------+
|                                 QUERY PLAN                                 |
+----------------------------------------------------------------------------+
| Seq Scan on test (cost=0.00..53307.26 rows=2337060 width=49)              |
|   Filter: (input_date > '01-JAN-20 00:00:00'::timestamp without time zone) |
+----------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2021-01-01 00:00') ;
+----------------------------------------------------------------------------+
|                                 QUERY PLAN                                 |
+----------------------------------------------------------------------------+
| Seq Scan on test (cost=0.00..53307.26 rows=1818857 width=49)              |
|   Filter: (input_date > '01-JAN-21 00:00:00'::timestamp without time zone) |
+----------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2022-01-01 00:00') ;
+---------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                       |
+---------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..49951.72 rows=1287902 width=49) |
|   Index Cond: (input_date > '01-JAN-22 00:00:00'::timestamp without time zone)        |
+---------------------------------------------------------------------------------------+
(2 rows)

force_generic_plan

test=# set plan_cache_mode = force_generic_plan ;
test=# show plan_cache_mode ;
+--------------------+
| plan_cache_mode   |
+--------------------+
| force_generic_plan |
+--------------------+
(1 row)

test=# explain execute stmt ('2022-01-01 00:00') ;
+--------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                      |
+--------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..30216.28 rows=779020 width=49) |
|   Index Cond: (input_date > $1)                                                      |
+--------------------------------------------------------------------------------------+
(2 rows)

test=# explain execute stmt ('2020-01-01 00:00') ;
+--------------------------------------------------------------------------------------+
|                                      QUERY PLAN                                      |
+--------------------------------------------------------------------------------------+
| Index Scan using test_input_date on test (cost=0.43..30216.28 rows=779020 width=49) |
|   Index Cond: (input_date > $1)                                                      |
+--------------------------------------------------------------------------------------+
(2 rows)

log file Rotation

★깜장별★ — Wed, 22 May 2024 17:39:21 +0900

MongoDB log file rotation

MonogDB는 Log가 자동으로 로테이션 되지 않기 때문에 쉘이나 크론탭에 등록해서 로그를 로테이션 하는 명령을 수행해 줘야 한다
해당 명령은 rs.oplog에 기록되지 않기 때문에 Replica Set이면 모든 맴버 서버에서 각각 수행해 줘야 한다
로그 Rotate 명령을 수행하면 원래 로그파일을 rename 하고, 새로운 로그 파일을 생성한다

명령어

db.adminCommand( { logRotate : "server" } )

명령어를 수행하게 되면, 사용하던 로그 파일의 이름을 명령어가 수행된 시간으로 파일명을 변경 시키고, 신규 로그파일을 사용한다

mongod.log.2024-05-22T01-45-38
mongod.log.2024-05-22T01-45-47
mongod.log.2024-05-22T01-54-43
mongod.log <--- 현재 current log file

oplog.rs 내용 확인

★깜장별★ — Wed, 22 May 2024 17:26:15 +0900

Replica set 의 모든 Secondary 멤버는 Primary 멤버로 부터 변경이 기록된 로그(oplog.rs)를 가져와서 적용 하면서, Primary 와 Secondary 간의 데이터를 동기화를 하게 됩니다.

아래 테스트를 보면 확인 할 수 있듯이, 수행된 MQL이 저장되지 않고 실제 저장될 document 내용이 oplog.rs에 저장 됩니다

그래서 현재 시간을 사용할 경우 쿼리를 이용한다면 Primary, secondary의 내용이 다를 수 있지만, mongodb는 실 데이터를 가져가서 반영 시키므로 데이터가 달라지지는 않습니다.

##################

# 테스트 시나리오

##################

insert , update, delete , drop 명령 수행 후 oplog.rs 내용 확인

use test

db.t2.insertOne({col1 : Date() , name : 'hong gil dong' , city : 'seoul' , meta : { gu : 'jongno-gu' , other : 'jong-ro 33-gil'}})

db.t2.insertOne({col1 : Date() , name : 'Yi Sun-sin' , city : 'Gyeongsangnam-do'})

db.t2.update(

{},

{$set:{col1:Date()}},

{multi: true}

)

db.t2.remove(

{name : 'hong gil dong'}

)

##########################

# oplog.rs field 값 설명

##########################

ts (timestamp)

: Unix Epoch 시간 값

이 값을 기준으로 Secondary에서 데이터를 요청 (기존에 반영한 이후의 데이터를 찾는 용도)

op (operation type)

: i : insert

d : delete

u : update

c : command

n : no operation

c : 데이터베이스 생성, 컬렉션 생성, 컬렉션 삭제, 컬렉션 속성 변경 등이 발생 할 경우

n : 단순한 정보 메세지를 저장하는 경우

ns (namespace)

: 대상 컬렉션의 네임스페이스

네임스페이스 = "데이터베이스.컬렉션명"

o (operation)

: 실제 Document에 반영 된 값이 저장

insert : insert된 Document값

update : update된 filed 값

delete : delete된 document의 PK값 (_id filed값)

create, drop : 명령어 + 컬렉션 명

o2 (operation 2)

: insert, update, delete : 영향을 받은 Document PK값 (_id field값)

##################

# PSS 정보

##################

PRIMARY : 'rhel:26001'

SECONDARY : 'rhel:26002'

SECONDARY : 'rhel:26003'

#####################

# Primary 에서 Insert

#####################

insert MQL을 수행하면, MQL로 인해서 실제 저장되는 Document 내용이 저장 됨

최초 insert 발생 시 자동으로 collection이 생성되는 것도 확인 할 수 있다

test

comu [primary] test> use test
comu [direct: primary] test> db.t2.insertOne({col1 : Date() , name : 'hong gil dong' , city : 'seoul' , meta : { gu : 'jongno-gu' , other : 'jong-ro 33-gil'}})
{
acknowledged: true,
insertedId: ObjectId('6646f4d1986dc1b24cc934df')
}
comu [direct: primary] test> db.t2.insertOne({col1 : Date() , name : 'Yi Sun-sin' , city : 'Gyeongsangnam-do'})
{
acknowledged: true,
insertedId: ObjectId('6646f4e0986dc1b24cc934e0')
}
comu [direct: primary] test> db.t2.find()
[
{
    _id: ObjectId('6646f4d1986dc1b24cc934df'),
    col1: 'Fri May 17 2024 15:10:25 GMT+0900 (Korean Standard Time)',
    name: 'hong gil dong',
    city: 'seoul',
    meta: { gu: 'jongno-gu', other: 'jong-ro 33-gil' }
},
{
    _id: ObjectId('6646f4e0986dc1b24cc934e0'),
    col1: 'Fri May 17 2024 15:10:40 GMT+0900 (Korean Standard Time)',
    name: 'Yi Sun-sin',
    city: 'Gyeongsangnam-do'
}
]

local

comu [direct: primary] test> use local
comu [direct: primary] local> db.oplog.rs.find({op : 'i'}).sort({$natural: -1}).limit(2).pretty()
[
{
    lsid: {
      id: UUID('5756aba4-9615-4139-922a-9b7a25d45e6e'),
      uid: Binary.createFromBase64('Y5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fg=', 0)
    },
    txnNumber: Long('3'),
    op: 'i',
    ns: 'test.t2',
    ui: UUID('2a11f34d-ec5b-4bde-baef-db13f903fe18'),
    o: {
      _id: ObjectId('6646f4e0986dc1b24cc934e0'),
      col1: 'Fri May 17 2024 15:10:40 GMT+0900 (Korean Standard Time)',
      name: 'Yi Sun-sin',
      city: 'Gyeongsangnam-do'
    },
    o2: { _id: ObjectId('6646f4e0986dc1b24cc934e0') },
    stmtId: 0,
    ts: Timestamp({ t: 1715926240, i: 1 }),
    t: Long('10'),
    v: Long('2'),
    wall: ISODate('2024-05-17T06:10:40.409Z'),
    prevOpTime: { ts: Timestamp({ t: 0, i: 0 }), t: Long('-1') }
},
{
    lsid: {
      id: UUID('5756aba4-9615-4139-922a-9b7a25d45e6e'),
      uid: Binary.createFromBase64('Y5mrDaxi8gv8RmdTsQ+1j7fmkr7JUsabhNmXAheU0fg=', 0)
    },
    txnNumber: Long('2'),
    op: 'i',
    ns: 'test.t2',
    ui: UUID('2a11f34d-ec5b-4bde-baef-db13f903fe18'),
    o: {
      _id: ObjectId('6646f4d1986dc1b24cc934df'),
      col1: 'Fri May 17 2024 15:10:25 GMT+0900 (Korean Standard Time)',
      name: 'hong gil dong',
      city: 'seoul',
      meta: { gu: 'jongno-gu', other: 'jong-ro 33-gil' }
    },
    o2: { _id: ObjectId('6646f4d1986dc1b24cc934df') },
    stmtId: 0,
    ts: Timestamp({ t: 1715926225, i: 2 }),
    t: Long('10'),
    v: Long('2'),
    wall: ISODate('2024-05-17T06:10:25.827Z'),
    prevOpTime: { ts: Timestamp({ t: 0, i: 0 }), t: Long('-1') }
}
]

comu [direct: primary] local> db.oplog.rs.find({op : 'c' , "o.create" : 't2'}).sort({$natural: -1}).limit(1).pretty()
[
{
    op: 'c',
    ns: 'test.$cmd',
    ui: UUID('2a11f34d-ec5b-4bde-baef-db13f903fe18'),
    o: { create: 't2', idIndex: { v: 2, key: { _id: 1 }, name: '_id_' } },
    ts: Timestamp({ t: 1715926225, i: 1 }),
    t: Long('10'),
    v: Long('2'),
    wall: ISODate('2024-05-17T06:10:25.827Z')
}
]

#####################

# Secondary에서 조회

#####################

Secondary 1

secondary 2

comu [direct: primary] local> db = connect('rhel:26002/admin','root','root')
comu [direct: secondary] admin> use test
comu [direct: secondary] test> db.t2.find()
[
{
    _id: ObjectId('6646f4d1986dc1b24cc934df'),
    col1: 'Fri May 17 2024 15:10:25 GMT+0900 (Korean Standard Time)',
    name: 'hong gil dong',
    city: 'seoul',
    meta: { gu: 'jongno-gu', other: 'jong-ro 33-gil' }
},
{
    _id: ObjectId('6646f4e0986dc1b24cc934e0'),
    col1: 'Fri May 17 2024 15:10:40 GMT+0900 (Korean Standard Time)',
    name: 'Yi Sun-sin',
    city: 'Gyeongsangnam-do'
}
]

comu [direct: secondary] test> db = connect('rhel:26003/admin','root','root')
comu [direct: secondary] admin> use test
comu [direct: secondary] test> db.t2.find()
[
{
    _id: ObjectId('6646f4d1986dc1b24cc934df'),
    col1: 'Fri May 17 2024 15:10:25 GMT+0900 (Korean Standard Time)',
    name: 'hong gil dong',
    city: 'seoul',
    meta: { gu: 'jongno-gu', other: 'jong-ro 33-gil' }
},
{
    _id: ObjectId('6646f4e0986dc1b24cc934e0'),
    col1: 'Fri May 17 2024 15:10:40 GMT+0900 (Korean Standard Time)',
    name: 'Yi Sun-sin',
    city: 'Gyeongsangnam-do'
}
]

########################

# Primary에서 update 수행

########################

update에 의해서 변경되는 부분만 저장. 해당 데이터는 PK인 _id 값이 같이 저장되므로 해당 값을 이용해서 변경

test

comu [direct: secondary] test> db = connect('rhel:26001/admin','root','root')
comu [direct: primary] admin> use test
comu [direct: primary] test> db.t2.update(
... {},
... {$set:{col1:Date()}},
... {multi: true}
... )
{
acknowledged: true,
insertedId: null,
matchedCount: 2,
modifiedCount: 2,
upsertedCount: 0
}
comu [direct: primary] test> db.t2.find()
[
{
    _id: ObjectId('6646f4d1986dc1b24cc934df'),
    col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)',
    name: 'hong gil dong',
    city: 'seoul',
    meta: { gu: 'jongno-gu', other: 'jong-ro 33-gil' }
},
{
    _id: ObjectId('6646f4e0986dc1b24cc934e0'),
    col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)',
    name: 'Yi Sun-sin',
    city: 'Gyeongsangnam-do'
}
]

local

comu [direct: primary] test> use local
comu [direct: primary] local> db.oplog.rs.find({op : 'u'}).sort({$natural: -1}).limit(2).pretty()
[
{
    op: 'u',
    ns: 'test.t2',
    ui: UUID('2a11f34d-ec5b-4bde-baef-db13f903fe18'),
    o: {
      '$v': 2,
      diff: {
        u: {
          col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)'
        }
      }
    },
    o2: { _id: ObjectId('6646f4e0986dc1b24cc934e0') },
    ts: Timestamp({ t: 1715926905, i: 2 }),
    t: Long('10'),
    v: Long('2'),
    wall: ISODate('2024-05-17T06:21:45.353Z')
},
{
    op: 'u',
    ns: 'test.t2',
    ui: UUID('2a11f34d-ec5b-4bde-baef-db13f903fe18'),
    o: {
      '$v': 2,
      diff: {
        u: {
          col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)'
        }
      }
    },
    o2: { _id: ObjectId('6646f4d1986dc1b24cc934df') },
    ts: Timestamp({ t: 1715926905, i: 1 }),
    t: Long('10'),
    v: Long('2'),
    wall: ISODate('2024-05-17T06:21:45.353Z')
}
]

#####################

# Secondary 에서 조회

#####################

Secondary 1

secondary 2

comu [direct: primary] local> db = connect('rhel:26002/admin','root','root')
comu [direct: secondary] admin> use test
comu [direct: secondary] test> db.t2.find()
[
{
    _id: ObjectId('6646f4d1986dc1b24cc934df'),
    col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)',
    name: 'hong gil dong',
    city: 'seoul',
    meta: { gu: 'jongno-gu', other: 'jong-ro 33-gil' }
},
{
    _id: ObjectId('6646f4e0986dc1b24cc934e0'),
    col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)',
    name: 'Yi Sun-sin',
    city: 'Gyeongsangnam-do'
}
]

comu [direct: secondary] test> db = connect('rhel:26003/admin','root','root')
comu [direct: secondary] admin> use test
comu [direct: secondary] test> db.t2.find()
[
{
    _id: ObjectId('6646f4d1986dc1b24cc934df'),
    col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)',
    name: 'hong gil dong',
    city: 'seoul',
    meta: { gu: 'jongno-gu', other: 'jong-ro 33-gil' }
},
{
    _id: ObjectId('6646f4e0986dc1b24cc934e0'),
    col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)',
    name: 'Yi Sun-sin',
    city: 'Gyeongsangnam-do'
}
]

#####################

# Primary에서 삭제 진행

#####################

delete 진행 시 지워 진 Document의 pk값인 _id만 저장

drop 진행 시 명령어와 collection명 확인 가능

test	comu [direct: secondary] test> db = connect('rhel:26001/admin','root','root') comu [direct: primary] admin> use test comu [direct: primary] test> db.t2.remove( ... {name : 'hong gil dong'} ... ) DeprecationWarning: Collection.remove() is deprecated. Use deleteOne, deleteMany, findOneAndDelete, or bulkWrite. { acknowledged: true, deletedCount: 1 } comu [direct: primary] test> db.t2.find() [ { _id: ObjectId('6646f4e0986dc1b24cc934e0'), col1: 'Fri May 17 2024 15:21:45 GMT+0900 (Korean Standard Time)', name: 'Yi Sun-sin', city: 'Gyeongsangnam-do' } ]
local	comu [direct: primary] local> db.oplog.rs.find({op : 'd'}).sort({$natural: -1}).limit(1).pretty() [ { op: 'd', ns: 'test.t2', ui: UUID('2a11f34d-ec5b-4bde-baef-db13f903fe18'), o: { _id: ObjectId('6646f4d1986dc1b24cc934df') }, ts: Timestamp({ t: 1715927604, i: 1 }), t: Long('10'), v: Long('2'), wall: ISODate('2024-05-17T06:33:24.014Z') } ]
test	comu [direct: primary] local> use test comu [direct: primary] test> db.t2.drop() comu [direct: primary] test> db.t2.find()
local	comu [direct: primary] test> use local comu [direct: primary] local> db.oplog.rs.find({op : 'c' , "o.drop" : 't2'}).sort({$natural: -1}).limit(1).pretty() [ { op: 'c', ns: 'test.$cmd', ui: UUID('2a11f34d-ec5b-4bde-baef-db13f903fe18'), o: { drop: 't2' }, o2: { numRecords: 1 }, ts: Timestamp({ t: 1715927871, i: 1 }), t: Long('10'), v: Long('2'), wall: ISODate('2024-05-17T06:37:51.638Z') } ]

Reog 대상 확인

★깜장별★ — Wed, 24 Jan 2024 14:41:51 +0900

PostgreSQL에서 reog 대상 테이블 선정 및 반환되는 사이즈 확인

0. 결론

- 정확하게 반환되는 사이즈는 확인 불가 (free 공간 비슷하게 공간 반환)

--> 데이터가 block에 저장 될 때 저장가능 모든 공간을 사용할 수 있는 데이터 사이즈 일 경우 free size가 다 반환되나, 남은 공간이 하나의 Row를 저장할 공간이 되지 않을 경우 , 새로운 block에 저장하므로 free 공간이 생길 수 밖에 없으므로 free 공간이 다 반환된다고 볼 수는 없다

- pgstattuple은 전체 테이블에 대한 full scan을 수행 한다

1. 테스트 데이터 생성

테이블 생성	drop table if exists test_reog ; create table test_reog ( item_code char(12) , order_id char(12) , order_flag char(1) , order_date timestamp , check_flag char(1) , check_date timestamp , delivery_flag char(1) , delivery_date timestamp , delivery_complite_flag char(1) , delivery_complite_date timestamp ) ;
데이터 삽입	do $insert$ declare i int; begin for i in 1..10000000 loop insert into test_reog values ( concat('item',lpad(i::bpchar,8,'0')) , 'admin' , 'Y' , now() , NULL , NULL , NULL , NULL , NULL , NULL ) ; end loop; end $insert$ ;
사이즈 확인	select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze , pg_size_pretty(pg_relation_size( 'test_reog' )) as table_size from pg_stat_all_tables where relname= 'test_reog' ; schemaname \| relname \| n_live_tup \| n_tup_upd \| n_dead_tup \| last_autovacuum \| last_analyze \| table_size ------------+-----------+------------+-----------+------------+-----------------+--------------+------------ scott \| test_reog \| 10000000 \| 0 \| 0 \| \| \| 730 MB

2. 데이터 변경

전체 데이터 중에서 1/100만 변경

update 속도 향상을 위해서 인덱스 생성

인덱스 생성	create index test_reog_item_code on test_reog(item_code) ;
사이즈 확인	select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze , pg_size_pretty(pg_table_size('test_reog')) as table_size , pg_size_pretty(pg_indexes_size('test_reog')) as idx_size from pg_stat_all_tables where relname= 'test_reog' ; schemaname \| relname \| n_live_tup \| n_tup_upd \| n_dead_tup \| last_autovacuum \| last_analyze \| table_size \| idx_size ------------+-----------+------------+-----------+------------+-------------------------------+--------------+------------+---------- scott \| test_reog \| 10000006 \| 0 \| 0 \| 2024-01-24 13:21:02.058931+09 \| \| 730 MB \| 301 MB
데이터 변경	do $update$ declare i int; begin for i in 1..10000000 loop if mod(i,100) = 0 then update test_reog set check_flag = 'Y' , check_date = now() where item_code = concat('item',lpad(i::bpchar,8,'0'))::bpchar ; end if; end loop; end $update$ ;
사이즈 확인	select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze , pg_size_pretty(pg_table_size('test_reog')) as table_size , pg_size_pretty(pg_indexes_size('test_reog')) as idx_size from pg_stat_all_tables where relname= 'test_reog' ; schemaname \| relname \| n_live_tup \| n_tup_upd \| n_dead_tup \| last_autovacuum \| last_analyze \| table_size \| idx_size ------------+-----------+------------+-----------+------------+-------------------------------+--------------+------------+---------- scott \| test_reog \| 10000006 \| 100000 \| 100000 \| 2024-01-24 13:21:02.058931+09 \| \| 739 MB \| 301 MB

3. 데이터 확인

현재 테이블의 데이터 중에서 free 공간이 얼마나 되는지 확인해 보자

테스트 1

평균 tuple size와 tuple count를 이용해서 계산해 본다
테이블의 평균 칼럼 사이즈(pg_stats.avg_width)를 합한 후에 전체 테이블 사이즈와 비교 작업 진행

analyze test_reog ;

select a.tablename , pg_size_pretty(a.row_cnt * b.avg_row_byte) as tb_avg_used_size ,
       pg_size_pretty(pg_table_size('test_reog')) as tb_real_size ,
((a.row_cnt * b.avg_row_byte)::float/pg_table_size('test_reog')::float)*100 as tb_used_per
from (select 'test_reog' as tablename , count(*) as row_cnt from test_reog ) a ,
       (select 'test_reog' as tablename , sum(avg_width) as avg_row_byte from pg_stats where tablename ='test_reog' ) b
where a.tablename = b.tablename ;

tablename | tb_avg_used_size | tb_real_size |    tb_used_per
-----------+------------------+--------------+-------------------
test_reog | 591 MB           | 739 MB       | 79.98350709122421

전체 table size : 739 MB
tuple이 사용하는 size : 591 MB
대략 141 MB의 공간이 비어 있는 것으로 나옴

테스트 2

테스트 #1, 테스트 #2의 결과가 너무 다르므로 실제 vacuum full을 수행해 본다

4. vacuum full 수행

vacuum full 수행

acuum full test_reog ;

사이즈 확인

select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze ,
       pg_size_pretty(pg_table_size('test_reog')) as table_size ,
       pg_size_pretty(pg_indexes_size('test_reog')) as idx_size
from pg_stat_all_tables where relname= 'test_reog' ;

schemaname | relname | n_live_tup | n_tup_upd | n_dead_tup |        last_autovacuum        |         last_analyze          | table_size | idx_size
------------+-----------+------------+-----------+------------+-------------------------------+-------------------------------+------------+----------
scott      | test_reog |    9999426 |    100000 |     100000 | 2024-01-24 13:21:02.058931+09 | 2024-01-24 13:32:09.581285+09 | 732 MB     | 301 MB

5. 재 테스트

예상으로는 free size인 10 MB가 줄어서 729 MB가 되기를 기대 하였으나 테이블 사이즈가 732 MB로 약 7 MB정도 줄어 들었다

그래서 다시 auto vacuum이 수행되지 않도록 테이블에 설정 후 다시 테스트 진행

테이블 생성	drop table if exists test_reog ; create table test_reog ( item_code char(12) , order_id char(12) , order_flag char(1) , order_date timestamp , check_flag char(1) , check_date timestamp , delivery_flag char(1) , delivery_date timestamp , delivery_complite_flag char(1) , delivery_complite_date timestamp ) ; alter table test_reog set (autovacuum_enabled = false) ;
데이터 입력	do $insert$ declare i int; begin for i in 1..10000000 loop insert into test_reog values ( concat('item',lpad(i::bpchar,8,'0')) , 'admin' , 'Y' , now() , NULL , NULL , NULL , NULL , NULL , NULL ) ; end loop; end $insert$ ; select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze , pg_size_pretty(pg_relation_size( 'test_reog' )) as table_size from pg_stat_all_tables where relname= 'test_reog' ; schemaname \| relname \| n_live_tup \| n_tup_upd \| n_dead_tup \| last_autovacuum \| last_analyze \| table_size ------------+-----------+------------+-----------+------------+-----------------+--------------+------------ scott \| test_reog \| 10000000 \| 0 \| 0 \| \| \| 730 MB create index test_reog_item_code on test_reog(item_code) ; select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze , pg_size_pretty(pg_table_size('test_reog')) as table_size , pg_size_pretty(pg_indexes_size('test_reog')) as idx_size from pg_stat_all_tables where relname= 'test_reog' ; schemaname \| relname \| n_live_tup \| n_tup_upd \| n_dead_tup \| last_autovacuum \| last_analyze \| table_size \| idx_size ------------+-----------+------------+-----------+------------+-----------------+--------------+------------+---------- scott \| test_reog \| 10000000 \| 0 \| 0 \| \| \| 730 MB \| 301 MB
데이터 변경	do $update$ declare i int; begin for i in 1..10000000 loop if mod(i,100) = 0 then update test_reog set check_flag = 'Y' , check_date = now() where item_code = concat('item',lpad(i::bpchar,8,'0'))::bpchar ; end if; end loop; end $update$ ; select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze , pg_size_pretty(pg_table_size('test_reog')) as table_size , pg_size_pretty(pg_indexes_size('test_reog')) as idx_size from pg_stat_all_tables where relname= 'test_reog' ; schemaname \| relname \| n_live_tup \| n_tup_upd \| n_dead_tup \| last_autovacuum \| last_analyze \| table_size \| idx_size ------------+-----------+------------+-----------+------------+-----------------+--------------+------------+---------- scott \| test_reog \| 10000000 \| 100000 \| 100000 \| \| \| 739 MB \| 301 MB
psstattuple	select pg_size_pretty(table_len) as table_size , pg_size_pretty(tuple_len) as tuple_size , pg_size_pretty(free_space) as free_size , tuple_count , dead_tuple_count , tuple_percent , free_percent from pgstattuple('scott.test_reog'); table_size \| tuple_size \| free_size \| tuple_count \| dead_tuple_count \| tuple_percent \| free_percent ------------+------------+-----------+-------------+------------------+---------------+-------------- 739 MB \| 688 MB \| 3002 kB \| 10000000 \| 100000 \| 93.12 \| 0.4 현재는 vacuum이 수행되지 않아서 dead tuple이 공간을 차지 하고 있으므로 free 공간이 3MB임을 알 수 있다
vacuum 후 pgstattuple	vacuum test_reog ; select pg_size_pretty(table_len) as table_size , pg_size_pretty(tuple_len) as tuple_size , pg_size_pretty(free_space) as free_size , tuple_count , dead_tuple_count , tuple_percent , free_percent from pgstattuple('scott.test_reog'); table_size \| tuple_size \| free_size \| tuple_count \| dead_tuple_count \| tuple_percent \| free_percent ------------+------------+-----------+-------------+------------------+---------------+-------------- 739 MB \| 688 MB \| 10036 kB \| 10000000 \| 0 \| 93.12 \| 1.33 vacuum을 수행 후 dead tuple이 있던 공간이 사용가능 공간으로 변경 되어서 free 공간이 10MB로 늘어 남
vacuum full	vacuum full test_reog ; select schemaname , relname , n_live_tup , n_tup_upd, n_dead_tup , last_autovacuum , last_analyze , pg_size_pretty(pg_table_size('test_reog')) as table_size , pg_size_pretty(pg_indexes_size('test_reog')) as idx_size from pg_stat_all_tables where relname= 'test_reog' ; schemaname \| relname \| n_live_tup \| n_tup_upd \| n_dead_tup \| last_autovacuum \| last_analyze \| table_size \| idx_size ------------+-----------+------------+-----------+------------+-----------------+--------------+------------+---------- scott \| test_reog \| 10000000 \| 100000 \| 0 \| \| \| 732 MB \| 301 MB 역시나 10 MB의 공간이 반환 되기를 기대 하였으나, 7 MB의 공간만이 줄어 들었다 select pg_size_pretty(table_len) as table_size , pg_size_pretty(tuple_len) as tuple_size , pg_size_pretty(free_space) as free_size , tuple_count , dead_tuple_count , tuple_percent , free_percent from pgstattuple('scott.test_reog'); table_size \| tuple_size \| free_size \| tuple_count \| dead_tuple_count \| tuple_percent \| free_percent ------------+------------+-----------+-------------+------------------+---------------+-------------- 732 MB \| 688 MB \| 2969 kB \| 10000000 \| 0 \| 94.05 \| 0.4 pgstattuple로 다시 정보를 조회해 보면 free 공간이 약 3 MB 존재 한다 테이블 사이즈 739 MB에서 10 MB가 줄어든 729 MB를 기대 하였으나 732 MB의 테이블 사이즈를 가지며, 약 3 MB의 free 공간이 있는 상태이다 그러면 732 MB - 3 MB 하면 729 MB로 예상했던 사이즈로 생각해 볼 수가 있다 ?
3MB ?	vacuum full을 해서 모든 데이터를 다시 insert 했음에도 약 3MB의 free 공간이 존재하는 이유를 찾아보자 select relpages from pg_class where relname = 'test_reog' ; relpages ---------- 93660 전체 블록 개수는 93660 select 2969 *1024 / 93660 ; ?column? ---------- 32 free 공간(2969 kb)을 블로 개수로 나누면 한 블록당 약 32 byte의 여유 공간이 존재 한다는 뜻 블록을 살펴 보자 !!! CREATE EXTENSION pageinspect; 테이블의 블록을 처음부터 확인해 보자 select lower , upper , special , pagesize from page_header(get_raw_page('test_reog',0)) union all select lower , upper , special , pagesize from page_header(get_raw_page('test_reog',1)) union all select lower , upper , special , pagesize from page_header(get_raw_page('test_reog',2)) union all select lower , upper , special , pagesize from page_header(get_raw_page('test_reog',3)) union all select lower , upper , special , pagesize from page_header(get_raw_page('test_reog',93658)) union all select lower , upper , special , pagesize from page_header(get_raw_page('test_reog',93659)) ; lower \| upper \| special \| pagesize -------+-------+---------+---------- 452 \| 488 \| 8192 \| 8192 452 \| 488 \| 8192 \| 8192 452 \| 488 \| 8192 \| 8192 452 \| 488 \| 8192 \| 8192 452 \| 488 \| 8192 \| 8192 332 \| 2648 \| 8192 \| 8192 블록의 lower ~ upper까지의 공간이 free 공간인데, 맨 마지막 블록을 제외하고 488-452 하면 약 36 byte의 free 공간을 가지고 있는 것이 확인된다 전체 free 공간을 더해 보자 do $select$ declare i int ; free_sum int := 0 ; free int := 0 ; begin for i in 0..93658 loop select upper - lower into free from page_header(get_raw_page('test_reog', i )) ; free_sum := free_sum + free ; free := 0 ; end loop; RAISE NOTICE 'free : % ' , free_sum ; end $select$ ; NOTICE: free : 3412564 select pg_size_pretty(3412564::numeric) ; pg_size_pretty ---------------- 3333 kB 정확하게 일치 하지 않지만, 비슷한 사이즈의 free 공간이 있음을 확인 할 수 있다 하나의 tuple의 평균 사이즈를 확인해 보면 analyze test_reog ; select 'test_reog' as tablename , sum(avg_width) as avg_row_byte from pg_stats where tablename ='test_reog' ; tablename \| avg_row_byte -----------+-------------- test_reog \| 62 하나의 tuple의 평균 사이즈가 62 byte 이므로 하나의 tuple을 저장할 공간으로 62 byte가 되지 않을 경우 다른 블록에 데이터를 insert 하므로 블록 당 free 공간이 존재 함을 확인 할 수 있다 정확하게 free 공간이 일치 하였으면 깔끔했을 것 같으나, 대략적인 원인 정도만 파악을 해서 약간 아쉬움(?)이 남는다

6. Reog 대상 확인 쿼리

아래 쿼리는 테이블 크기가 500MB 이상인 테이블을 대상으로 freespace가 많은 테이블들 순으로 조회 하는 쿼리 입니다

select schemaname , relname , pg_size_pretty(tb_byte) tb_byte , pg_size_pretty(free_byte) free_byte , free_per
from   ( select schemaname ,
                relname ,
                (pgstattuple(concat(schemaname,'.',relname))).table_len tb_byte,
                (pgstattuple(concat(schemaname,'.',relname))).free_space free_byte,
                (pgstattuple(concat(schemaname,'.',relname))).free_percent free_per
         from   pg_stat_user_tables ) user_tb
where   tb_byte > 500000000
order by free_byte desc ;

schemaname | relname | tb_byte | free_byte | free_per
------------+-----------+---------+-----------+----------
scott      | test_reog | 732 MB | 2969 kB   |      0.4

dead lock

★깜장별★ — Tue, 16 Jan 2024 14:51:54 +0900

dead lock은 세션에서 대기하는 lock이 dead lock인지 아닌지 검사하는 시간을 lock을 대기 하자 마자 체크하지 않고, 일정 시간이 지난 후에 체크 합니다.

이유는 해당 lock이 서로 참조하고 있는지를 체크 하기 위해서는 내부적으로 체크하는 로직이 수행되어야 하므로, 매번 lock 대기시에 체크를 하게 되면 불 필요한 리소스를 사용하게 되고, 해당 lock 대기가 금방 끝날 경우 dead lock 체크를 위한 불필요한 리소스 사용을 하지 않게 하기 위함입니다.

그래서 lock을 대기 하고 일정 시간이 지난 후에 dead lock인지 여부를 체크 합니다.

아래의 테스트를 보시면 이해가 되시리라 생각 됩니다.

오라클 테스트 #1 : 세션1의 id = 2 업데이트가 lock 대기를 먼저 하고, 해당 lock 대기가 dead lock인지를 체크하는 시간이 먼저 도달 하기 때문에 세션 1이 dead lock으로 rollback

PostgreSQL 테스트 #1 : dead lock 체크가 1초 이므로, lock 대기 후 1초가 지난 후에 dead lock을 체크 하는데, 세션 1의 id = 2 쿼리 후에 세션2에서 id = 1 업데이트 문 수행 전에 1초가 지나 가므로, 세션 1의 dead lock을 체크하는 시기에는 dead lock 상태가 아니였으므로 세션2가 dead lock 으로 rollback

PostgreSQL 테스트 #2 : dead lock 체크를 10초로 변경 하였으므로, 세션 1에서 id = 2 쿼리 수행(lock 대기) 후 세션2에서 id =1을 할 때가 10초가 지나지 않은 시점이므로 세션 1의 dead lock 체크 시점에 세션2 업데이트로 인해서 dead lock 상태이므로 먼저 검사 대상인 세션 1이 rollback

postgreSQL 테스트 #3 : 세션 1의 업데이트 이후 10초가 지나서 세션1에 대한 dead lock 검사가 지난 시점에 세션 2에서 update를 수행하게 되면 세션2의 마지막 업데이트만 dead lock 체크 대상이 되므로 세션 2의 업데이트가 dead lock으로 인한 rollback

오라클 테스트 #2 : 위와 마찬가지 이유로 세션2가 dead lock으로 인한 rollback

Patroni + etcd + HAproxy를 이용한 HA구

★깜장별★ — Wed, 10 Jan 2024 15:35:42 +0900

Patroni를 이용하여 PostgreSQL HA 구성

0. 구성환경

ip	hostname	설치 제품
192.168.56.201	pgserver1	Cent7 minimal, PostgreSQL 13, patroni
192.168.56.202	pgserver2	Cent7 minimal, PostgreSQL 13, patroni
192.168.56.203	pgserver3	Cent7 minimal, etcd, HAproxy

기본적으로 위의 정보는 /etc/hosts와 /etc/hostname에 설정이 되어 있어야 하고, SELINUX=disabled 된 상태 (sed -i s/SELINUX=enforcing/SELINUX=disabled/ /etc/selinux/config)

PG 서버는 source 버전을 이용하여 컴파일 하여 사용

patroni : PostgreSQL의 관리를 하는 python 기반의 프로그램. Auto failover, pg_rewind를 이용한 replication 재 구성

etcd : key-value구조를 가지는 데이터베이스로 patroni에서 사용하는 메타 데이터와 patroni에서 전송되는 상태 값 저장. 이 값을 patroni에서 참조하여 장애 조치 또는 auto failover를 수행 함

HAproxy : patroni와 통신 하면서 현재의 Primary에 대한 정보를 받아와서 Client로의 연결을 Primary로 연결 시켜주는 라우터 역할

repmgr 또는 EFM은 각각의 데몬끼리 서로 통신하면서 상태값을 체크하면서 auto fail-over를 수행하는 반면, patroni는 상태값을 etcd에 저장하고 이 값을 참조하여 장애 조치나 failover를 수행

patroni 초기 기동 시 etcd와의 연결이 되지 않으면 initdb도 수행되지 않음

그래서 etcd를 3개 이상으로 구성하는 것을 권장함

patroni는 PostgreSQL 유저와 동일한 O/S로 기동

etcd, HAproxy는 동일 유저 아니어도 상관 없음

여기 테스트에서는 1개의 etcd, HAproxy로 구성하여 테스트 진행

1. PostgreSQL 설치 (pgserver1, pgserver2)

PostgreSQL source 버전을 다운 받아서 설치 진행

yum -y install epel-release
yum -y install gcc gcc-c++ make autoconf readline readline-devel zlib zlib-devel openssl openssl-devel gettext gettext-devel python python-devel wget

useradd postgres

mkdir /engine /data
chwon -R postgres:postgres /engine /data

postgres 유저로 소스파일 압축 푼 디렉터리로 이동하여 컴파일 진행
./configure --prefix=/engine/pg13
make
make world
make install
make install-world

postgres 유저의 .bash_profile에 /engine/pg13/bin 패스 추가

2. patroni 설치 (pgserver1, pgserver2)

root유저로 설치를 진행하는데, 파이썬에서 사용하는 psycopg2 컴파일 시에 postgresql 라이브러리를 참조하여서 컴파일이 진행 되므로

root유저의 path에 /engine/pg13/bin:/engine/pg13/lib:/engine/pg13/include 추가 후 적용

적용 후 which psql로 정상적으로 PATH가 적용 되었는지 확인 후 진행

yum -y install groupinstall "Development Tools"
yum -y install gcc python3 python3-devel python3-pip gcc libpq-devel

which pip3로 설치 확인

pip3 install patroni
pip3 install python-etcd
pip3 install psycopg2

which patroni로 patroni 설치 확인 후
patroni --version 으로 patroni가 정상 실행 되는지 확인
--> 여기서 에러 발생 할 경우
--> pip3 uninstall psycopg2
--> yum -y install postgresqlxx-libs (postgreSQL 버전에 맞는 파일 설치)
--> pip3 install psycopg2
--> patroni --version 으로 다시 확인

3. etcd 설치

pgserver3 서버에 root로 etcd 설치 진행

etcd가 patroni 보다 먼저 기동되어 있어야 함

dnf를 이용하여 설치 진행 시	파일을 다운받아서 설치 진행 시
yum -y install epel-release yum -y install dnf dnf install -y etcd which etcd etcd --version	https://github.com/etcd-io/etcd/releases/tag/v3.5.11 에서 다운로드 후 tar xvfz etcd-v3.5.11-linux-amd64.tar.gz mv etcd-v3.5.11-linux-amd64 etcd-v3.5.11 /engine/etcd-v3.5.11/etcd --version etcd Version: 3.5.11 Git SHA: 3b252db4f Go Version: go1.20.12 Go OS/Arch: linux/amd64

etcd config 파일 작성

echo "name: 'pgserver3'
data-dir: '/data/etcd/pgserver3.etcd'
listen-peer-urls: 'http://192.168.56.203:2380,http://127.0.0.1:2380'
listen-client-urls: 'http://127.0.0.1:2379,http://192.168.56.203:2379'
advertise-client-urls: 'http://192.168.56.203:2379'
initial-cluster: 'pgserver3=http://192.168.56.203:2380'
initial-advertise-peer-urls: 'http://192.168.56.203:2380'
initial-cluster-state: 'new'
initial-cluster-token: 'node1'
enable-v2: true" > /engine/etcd-v3.5.11/etcd.yml

etcd 기동

/engine/etcd-v3.5.11/etcd --config-file /engine/etcd-v3.5.11/etcd.yml > /engine/etcd-v3.5.11/etcd.log 2>&1 &
또는
etcd --config-file /engine/etcd-v3.5.11/etcd.yml > /tmp/etcd.log 2>&1 &

etcd 기동 확인

4. patroni 기동

pgserver1, pgserver2에서 postgres 유저로 기동

postgreSQL서버를 rpm으로 설치 후 서비스가 기동되어 있으면, 해당 서비스 종료 후, 서비스 disable 한 후 진행

데이터 디렉터리 여기서는 /data/pg13인데 /data 디렉터리 까지만 생성해 놓고 권한을 postgres 유저에게 주고서 실행해야 한다

/data/pg13까지 만들어 놓을 경우 patroni가 initial을 진행 할때 해당 디렉터리 /data/pg13의 권한을 변경하려고 하면서 제대로 변경이 되지 않는 경우 에러 발생

/data 까지만 만들어 놓고 chown -R postgres:postgres /data로 해 놓은 상태에서 패트로니 초기 기동

scope : patroni cluster이름( 여기서는 postgres로 지정, 이후 명령어에 postgres가 나오는 부분은 클러스터명을 지정한 것임)

name : patroni cluster내에서 유니크한 이름

etcd : etcd 서버 정보. etcd 서버가 여러대의 클러서터로 구성 되어 있으면 컴마로 구분하여 여러 IP 지정

pg_hba 부분에서는 해당 버전에 맞는 값으로 변경 md5 --> scram-sha-256

pgserver1	pgserver2
echo "scope: postgres name: pgsvr1 restapi: listen: '192.168.56.201:8008' connect_address: '192.168.56.201:8008' etcd: hosts: '192.168.56.203:2379' bootstrap: dcs: ttl: 30 loop_wait: 10 retry_timeout: 10 maximum_lag_on_failover: 1048576 postgresql: use_pg_rewind: true parameters: max_wal_size: '1G' max_wal_senders: 8 initdb: - encoding: UTF8 - data-checksums pg_hba: - host replication repl 192.168.56.201/32 md5 - host replication repl 192.168.56.202/32 md5 - host all all 0.0.0.0/0 md5 users: admin: password: admin options: - createrole - createdb postgresql: listen: '192.168.56.201:5413' connect_address: '192.168.56.201:5413' data_dir: /data/pg13 pgpass: /tmp/pgpass0 authentication: replication: username: repl password: repl superuser: username: postgres password: postgres rewind: username: rewind_user password: rewind_user parameters: unix_socket_directories: . tags: nofailover: false noloadbalance: false clonefrom: false nosync: false" > /engine/patroni.yml	echo "scope: postgres name: pgsvr2 restapi: listen: '192.168.56.202:8008' connect_address: '192.168.56.202:8008' etcd: hosts: '192.168.56.203:2379' bootstrap: dcs: ttl: 30 loop_wait: 10 retry_timeout: 10 maximum_lag_on_failover: 1048576 postgresql: use_pg_rewind: true parameters: max_wal_size: '1G' max_wal_senders: 8 initdb: - encoding: UTF8 - data-checksums pg_hba: - host replication repl 192.168.56.201/32 md5 - host replication repl 192.168.56.202/32 md5 - host all all 0.0.0.0/0 md5 users: admin: password: admin options: - createrole - createdb postgresql: listen: '192.168.56.202:5413' connect_address: '192.168.56.202:5413' data_dir: /data/pg13 pgpass: /tmp/pgpass0 authentication: replication: username: repl password: repl superuser: username: postgres password: postgres rewind: username: rewind_user password: rewind_user parameters: unix_socket_directories: . tags: nofailover: false noloadbalance: false clonefrom: false nosync: false" > /engine/patroni.yml
/usr/local/bin/patroni /engine/patroni.yml > /engine/patroni.log 2>&1 &	/usr/local/bin/patroni /engine/patroni.yml > /engine/patroni.log 2>&1 &

etcd와 patroni가 정상적으로 연결이 되면, patroni는 설정 파일 값을 읽어서 PostgreSQL replication을 자동으로 구성 한다

etcd와 통신이 되지 않으면 patroni는 아무런 동작을 하지 못 한다

patroni 상태 체크

state 값이 starting 또는 TL 값이 Reader와 동일하지 않은 경우 해당 서버의 동기화 상태가 이상이 있는 것이므로 PostgreSQL로그를 확인하여 해당 노드를 수동으로 재 구성해 주어야 할 수도 있다

실제 Patroni를 강제 종료 하고, PostgreSQL을 강제 종료 하는 등의 테스트를 진행하다 보니, Promotion 되면서 wal log들이 없다고 하면서 동기화가 깨지는 경우들이 발생 함

물론 운영 환경에서는 그런 경우가 없겠지만...

해당 경우에는 pause 명령을 이용하여 관리를 수동으로 전환한 후에 문제가 있는 서버의 PostgreSQL을 수동으로 종료 후 $PGDATA 디렉터리를 삭제 하고 resume 명령을 수행하면 된다

(이 경우는 내부적으로 pg_basebackup을 사용하여 복구)

4.1 switch over 수행

patroni를 이용하여 switch over 수행

reder나 replica 아무 곳에서 수행해도 됨

/usr/local/bin/patronictl -c /engine/patroni.yml switchover

Reader와 Replica Role이 바뀌면서 TL이 2로 변경 된 것 확인 가능

4.2 Reader 변경 히스토리 확인

patroni가 자동으로 failover를 수행했거나, 수동 switchover를 수행하여서 Reader가 변경 된 경우

해당 변경 히스토리를 확인 할 수 있다

/usr/local/bin/patronictl -c /engine/patroni.yml history postgres
+----+----------+------------------------------+----------------------------------+------------+
| TL |      LSN | Reason                       | Timestamp                        | New Leader |
+----+----------+------------------------------+----------------------------------+------------+
| 1 | 50549776 | no recovery target specified | 2024-01-08T13:00:31.621876+09:00 | pgsvr2     |
| 2 | 50717520 | no recovery target specified | 2024-01-08T13:14:54.865611+09:00 | pgsvr1    |
| 3 | 51162816 | no recovery target specified | 2024-01-08T13:43:41.631876+09:00 | pgsvr2     |
| 4 | 51163096 | no recovery target specified | 2024-01-08T13:46:24.821611+09:00 | pgsvr1     |
| 5 | 51272632 | no recovery target specified | 2024-01-09T10:22:37.616876+09:00 | pgsvr2     |
| 6 | 67109024 | no recovery target specified |                                  |            |
+----+----------+------------------------------+----------------------------------+------------+

4.3 파라미터 변경

patroni.yml에 여러 설정을 해 줄 수 없으므로 PostgreSQL 클러스터가 구성 된 이후에 변경 가능

필요한 내용을 postgresql 밑의 parameter 부분에 추가 한 후에 저장 후 반영 (reload, restart)

파라미터 = 값 이 아니라 파라미터: 값으로 설정 해야 함

/usr/local/bin/patronictl -c /engine/patroni.yml edit-config
loop_wait: 10
maximum_lag_on_failover: 1048576
postgresql:
parameters:
max_wal_senders: 8
max_wal_size: 1G
use_pg_rewind: true
retry_timeout: 10
ttl: 30

+++
@@ -4,6 +4,23 @@
   parameters:
     max_wal_senders: 8
     max_wal_size: 1G
+    max_connections: 500
+    port: 5413
+    shared_buffers: 1G
+    logging_collector: on
+    log_destination: 'stderr'
+    log_directory: 'log'
+    log_filename: 'postgresql-%Y-%m-%d.log'
+    log_checkpoints: on
+    log_connections: on
+    log_disconnections: on
+    log_lock_waits: on
+    log_temp_files: 0
+    log_min_duration_statement: 500
+    log_line_prefix: '%t [%p]: db=%d,user=%u,app=%a,client=%h '
+    shared_preload_libraries: 'pg_stat_statements'
+    pg_stat_statements.max: 10000
+    pg_stat_statements.track: all
   use_pg_rewind: true
retry_timeout: 10
ttl: 30

Apply these changes? [y/N]: y
Configuration changed

4.4 pause / resume

일시적으로 patroni가 PostgreSQL에 대한 유지 관리를 중지 하고, 수동으로 PostgreSQL에 대한 관리를 진행하고자 할 경우에 사용

다음의 예는 서버가 현재 동기화 문제가 발생하고 있는 상태에서 조치를 취하기 위하여 pause/resume 사용

/usr/local/bin/patronictl -c /engine/patroni.yml list
+ Cluster: postgres (7321565912410233191) ----------+----+-----------+
| Member | Host                | Role    | State    | TL | Lag in MB |
+--------+---------------------+---------+----------+----+-----------+
| pgsvr1 | 192.168.56.201:5413 | Replica | starting |    |   unknown |
| pgsvr2 | 192.168.56.202:5413 | Leader | running | 8 |           |
+--------+---------------------+---------+----------+----+-----------+
--pgsvr1이 동기화가 되고 있지 않는 상태

/usr/local/bin/patronictl -c /engine/patroni.yml pause postgres
Success: cluster management is paused

/usr/local/bin/patronictl -c /engine/patroni.yml list
+ Cluster: postgres (7321565912410233191) ----------+----+-----------+
| Member | Host                | Role    | State    | TL | Lag in MB |
+--------+---------------------+---------+----------+----+-----------+
| pgsvr1 | 192.168.56.201:5413 | Replica | starting |    |   unknown |
| pgsvr2 | 192.168.56.202:5413 | Leader | running | 8 |           |
+--------+---------------------+---------+----------+----+-----------+
Maintenance mode: on

-- 수동 관리 모드 이므로 문제가 있는 pgsvr1 서버에서 PostgreSQL 종료 후 $PGDATA삭제

ps -ef | grep postgres
postgres 3507 3092 0 09:14 pts/0    00:00:07 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml
postgres 4019     1 0 09:27 ?        00:00:00 postgres -D /data/pg13 --config-file=/data/pg13/postgresql.conf --listen_addresses=192.168.56.201 --port=5413 --cluster_name=postgres --wal_level=replica --hot_standby=on --max_connections=500 --max_wal_senders=8 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on
postgres 4021 4019 0 09:27 ?        00:00:00 postgres: postgres: logger
postgres 4022 4019 0 09:27 ?        00:00:00 postgres: postgres: startup recovering 000000060000000000000003

pg_ctl stop -D $PGDATA -mf
waiting for server to shut down.... done
server stopped

ps -ef | grep postgres
postgres 3507 3092 0 09:14 pts/0    00:00:08 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml

cd $PGDATA
pwd
/data/pg13
rm -rf *

/usr/local/bin/patronictl -c /engine/patroni.yml list
+ Cluster: postgres (7321565912410233191) ---------+----+-----------+
| Member | Host                | Role    | State   | TL | Lag in MB |
+--------+---------------------+---------+---------+----+-----------+
| pgsvr1 | 192.168.56.201:5413 | Replica | stopped |    |   unknown |
| pgsvr2 | 192.168.56.202:5413 | Leader | running | 8 |           |
+--------+---------------------+---------+---------+----+-----------+
Maintenance mode: on

/usr/local/bin/patronictl -c /engine/patroni.yml resume postgres
Success: cluster management is resumed

/usr/local/bin/patronictl -c /engine/patroni.yml list
+ Cluster: postgres (7321565912410233191) -----------+----+-----------+
| Member | Host                | Role    | State     | TL | Lag in MB |
+--------+---------------------+---------+-----------+----+-----------+
| pgsvr1 | 192.168.56.201:5413 | Replica | streaming | 8 |         0 |
| pgsvr2 | 192.168.56.202:5413 | Leader | running   | 8 |           |
+--------+---------------------+---------+-----------+----+-----------+

cd $PGDATA
ls
backup_label.old current_logfiles patroni.dynamic.json pg_hba.conf         pg_ident.conf.backup pg_notify    pg_snapshots pg_subtrans PG_VERSION postgresql.auto.conf         postgresql.conf         postmaster.pid
backup_manifest   global            pg_commit_ts          pg_hba.conf.backup pg_logical            pg_replslot pg_stat       pg_tblspc    pg_wal      postgresql.base.conf         postgresql.conf.backup standby.signal
base              log               pg_dynshmem           pg_ident.conf       pg_multixact          pg_serial    pg_stat_tmp   pg_twophase pg_xact     postgresql.base.conf.backup postmaster.opts

ps -ef | grep postgres
postgres 3507 3092 0 09:14 pts/0    00:00:10 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml
postgres 5944     1 0 09:50 ?        00:00:00 postgres -D /data/pg13 --config-file=/data/pg13/postgresql.conf --listen_addresses=192.168.56.201 --port=5413 --cluster_name=postgres --wal_level=replica --hot_standby=on --max_connections=500 --max_wal_senders=8 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on
postgres 5946 5944 0 09:50 ?        00:00:00 postgres: postgres: logger
postgres 5947 5944 0 09:50 ?        00:00:00 postgres: postgres: startup recovering 000000080000000000000006
postgres 5948 5944 0 09:50 ?        00:00:00 postgres: postgres: checkpointer
postgres 5949 5944 0 09:50 ?        00:00:00 postgres: postgres: background writer
postgres 5950 5944 0 09:50 ?        00:00:00 postgres: postgres: stats collector
postgres 5951 5944 0 09:50 ?        00:00:01 postgres: postgres: walreceiver streaming 0/6000148
postgres 5960 5944 0 09:50 ?        00:00:00 postgres: postgres: postgres postgres 192.168.56.201(56398) idle
postgres 5965 5944 0 09:51 ?        00:00:00 postgres: postgres: postgres postgres 192.168.56.201(56404) idle

4.5 Patroni 테스트

4.5.1 replica 서버의 patroni 강제 종료

4.5.2 Leader 서버의 patroni 강제 종료

4.5.3 Replica PostgreSQL 서버 Down

patroni가 자동으로 PostgreSQL 기동

4.5.4 Leader PostgreSQL 서버 Down

해당 시점의 pgsrv2 의 patroni log를 보면 pgsvr1번 서버의 패트로니로 부터 기동할 수 없다는 메세지가 온 즉시 promote를 진행한다 (약 1초 소요)

201서버의 patroni 재 기동

[postgres@pgserver1 pg13]v13 $ ps -ef | grep patroni
postgres 7247 3092 0 10:59 pts/0    00:00:17 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml

[postgres@pgserver1 pg13]v13 $ kill -9 7247
[1]+ Killed                  /usr/local/bin/patroni /engine/patroni.yml > /engine/patroni.log 2>&1

[postgres@pgserver1 pg13]v13 $ /usr/local/bin/patroni /engine/patroni.yml > /engine/patroni.log 2>&1 &
[1] 9879

/usr/local/bin/patronictl -c /engine/patroni.yml list
+ Cluster: postgres (7321565912410233191) -----------+----+-----------+
| Member | Host                | Role    | State     | TL | Lag in MB |
+--------+---------------------+---------+-----------+----+-----------+
| pgsvr1 | 192.168.56.201:5413 | Replica | streaming | 10 |         0 |
| pgsvr2 | 192.168.56.202:5413 | Leader | running   | 10 |           |
+--------+---------------------+---------+-----------+----+-----------+

다시 patroni의 로그를 확인해 보면 정상적으로 pg_rewind로 데이터를 동기화 했음을 확인 할 수 있다

다시 테스트 !!

이번에는 HammerDB로 tpch 스키마를 생성 시키면서 Leader PostgreSQL을 강제 종료

PostgreSQL 서버가 죽었을 경우, Patroni가 빠르게 재 기동(13:18:06 --> 13:18:09)을 시키는데 정상적으로 재 기동이 될 경우 기존의
role 체인지는 없으나, 기동을 시키는데 정상 기동이 되지 않을 경우 role을 변경 한다는 것을 확인 할 수 있다

4.5.5 etcd가 종료 될 경우

patroni , postgreSQL 모두 정상 동작을 하고 있으나 etcd가 종료 된 경우. 현재의 구성은 etcd를 한대로 구성한 상태

/usr/local/bin/patronictl -c /engine/patroni.yml list
+ Cluster: postgres (7321565912410233191) -----------+----+-----------+
| Member | Host                | Role    | State     | TL | Lag in MB |
+--------+---------------------+---------+-----------+----+-----------+
| pgsvr1 | 192.168.56.201:5413 | Replica | streaming | 10 |         0 |
| pgsvr2 | 192.168.56.202:5413 | Leader | running   | 10 |           |
+--------+---------------------+---------+-----------+----+-----------+

[root@pgserver3 ~]# ps -ef | grep etcd
root      1483 1468 2 Jan09 pts/0    00:35:35 /engine/etcd-v3.5.11/etcd --config-file /engine/etcd-v3.5.11/etcd.yml
[root@pgserver3 ~]# kill -9 1483
[1]+ Killed                  /engine/etcd-v3.5.11/etcd --config-file /engine/etcd-v3.5.11/etcd.yml > /engine/etcd-v3.5.11/etcd.log 2>&1

/usr/local/bin/patronictl -c /engine/patroni.yml list
2024-01-10 13:38:37,570 - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f73e1536630>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v2/machines
2024-01-10 13:38:37,571 - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f73e15362b0>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v2/machines
2024-01-10 13:38:37,572 - ERROR - Failed to get list of machines from http://192.168.56.203:2379/v2: MaxRetryError("HTTPConnectionPool(host='192.168.56.203', port=2379): Max retries exceeded with url: /v2/machines (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f73e1536780>: Failed to establish a new connection: [Errno 111] Connection refused',))",)
2024-01-10 13:38:42,579 - WARNING - Retrying (Retry(total=1, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f73e1536cf8>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v2/machines
2024-01-10 13:38:42,580 - WARNING - Retrying (Retry(total=0, connect=None, read=None, redirect=0, status=None)) after connection broken by 'NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f73e1536c50>: Failed to establish a new connection: [Errno 111] Connection refused',)': /v2/machines
2024-01-10 13:38:42,581 - ERROR - Failed to get list of machines from http://192.168.56.203:2379/v2: MaxRetryError("HTTPConnectionPool(host='192.168.56.203', port=2379): Max retries exceeded with url: /v2/machines (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f73e1536fd0>: Failed to establish a new connection: [Errno 111] Connection refused',))",)

etcd와의 통신이 끊어지게 되므로 patroni가 정상 동작하지 못한다

PostgreSQL은 정상 동작하는지 확인 (Leader에 접속)

[postgres@pgserver2 log]v13 $ psql -h 192.168.56.202 -p 5413 -U postgres -W
postgres=# select * from pg_stat_replication ;
pid | usesysid | usename | application_name | client_addr   | client_hostname | client_port |         backend_start         | bac
kend_xmin |   state   | sent_lsn | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | syn
c_state |          reply_time
------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+----
----------+-----------+------------+------------+------------+------------+-----------+-----------+------------+---------------+----
--------+-------------------------------
7476 |    16384 | repl    | pgsvr1           | 192.168.56.201 |                 |       34860 | 2024-01-10 13:37:56.066935+09 |
          | streaming | 0/A478EA78 | 0/A478EA78 | 0/A478EA78 | 0/A478EA78 |           |           |            |             0 | asy
nc      | 2024-01-10 13:39:06.282104+09
(1 row)

PostgreSQL은 정상 동작하는지 확인 (replica에 접속)

[postgres@pgserver1 pg13]v13 $ psql -h 192.168.56.201 -p 5413 -U postgres -W
postgres=# select * from pg_stat_wal_receiver ;
pid | status   | receive_start_lsn | receive_start_tli | written_lsn | flushed_lsn | received_tli |      last_msg_send_time
|     last_msg_receipt_time     | latest_end_lsn |        latest_end_time        | slot_name | sender_host   | sender_port |

     conninfo

-------+-----------+-------------------+-------------------+-------------+-------------+--------------+-----------------------------
--+-------------------------------+----------------+-------------------------------+-----------+----------------+-------------+-----
------------------------------------------------------------------------------------------------------------------------------------
------------------------------------------------------------------------------------------------------------------------------------
-----------------------
11182 | streaming | 0/A4000000        |                10 | 0/A478EA78 | 0/A478EA78 |           10 | 2024-01-10 13:39:26.315423+0
9 | 2024-01-10 13:39:26.322137+09 | 0/A478EA78     | 2024-01-10 13:37:56.070323+09 | pgsvr1    | 192.168.56.202 |        5413 | user
=repl passfile=/tmp/pgpass0 channel_binding=prefer dbname=replication host=192.168.56.202 port=5413 application_name=pgsvr1 fallback
_application_name=postgres sslmode=prefer sslcompression=0 ssl_min_protocol_version=TLSv1.2 gssencmode=prefer krbsrvname=postgres ta
rget_session_attrs=any
(1 row)

PostgreSQL은 정상 동작하고 있는 상태

이때 Leader PostgreSQL을 강제 종료 시키면 ? pgsvr2서버의 PostgreSQL을 kill 시켜 보자

[postgres@pgserver2 log]v13 $ ps -ef | grep postgres
postgres 6107 3013 0 11:21 pts/0    00:00:15 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml
postgres 7461     1 0 13:37 ?        00:00:00 postgres -D /data/pg13 --config-file=/data/pg13/postgresql.conf --listen_addresses=192.168.56.202 --port=5413 --cluster_name=postgres --wal_level=replica --hot_standby=on --max_connections=500 --max_wal_senders=8 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on
postgres 7463 7461 0 13:37 ?        00:00:00 postgres: postgres: logger
postgres 7464 7461 0 13:37 ?        00:00:00 postgres: postgres: startup recovering 0000000A00000000000000A4
postgres 7465 7461 0 13:37 ?        00:00:00 postgres: postgres: checkpointer
postgres 7466 7461 0 13:37 ?        00:00:00 postgres: postgres: background writer
postgres 7467 7461 0 13:37 ?        00:00:00 postgres: postgres: stats collector
postgres 7472 7461 0 13:37 ?        00:00:00 postgres: postgres: postgres postgres 192.168.56.202(54254) idle
postgres 7474 7461 0 13:37 ?        00:00:00 postgres: postgres: postgres postgres 192.168.56.202(54256) idle
postgres 7476 7461 0 13:37 ?        00:00:00 postgres: postgres: walsender repl 192.168.56.201(34860) streaming 0/A478EA78

[postgres@pgserver2 log]v13 $ kill -9 7461

[postgres@pgserver2 log]v13 $ ps -ef | grep postgres
postgres 6107 3013 0 11:21 pts/0    00:00:16 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml

[postgres@pgserver2 log]v13 $ ps -ef | grep postgres
postgres 6107 3013 0 11:21 pts/0    00:00:16 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml

patroni는 기동 되어 있지만, 종료된 PostgreSQL을 재 기동 시키지는 못한다

이때 pgsvr1서버의 상태는 ? 계속 Replica 상태 이다

[postgres@pgserver1 pg13]v13 $ psql -h 192.168.56.201 -p 5413 -U postgres -W
postgres=# select pg_is_in_recovery() ;
pg_is_in_recovery
-------------------
t
(1 row)

etcd를 재 기동 시켜 보자!! (현재 Leader PostgreSQL은 Down된 상태이고, Patroni는 etcd와 통신이 안 되서 정상적인 동작을 하지 못하는 상태)

[root@pgserver3 ~]# /engine/etcd-v3.5.11/etcd --config-file /engine/etcd-v3.5.11/etcd.yml > /engine/etcd-v3.5.11/etcd.log 2>&1 &
[1] 2081

/usr/local/bin/patronictl -c /engine/patroni.yml list
+ Cluster: postgres (7321565912410233191) -----------+----+-----------+
| Member | Host                | Role    | State     | TL | Lag in MB |
+--------+---------------------+---------+-----------+----+-----------+
| pgsvr1 | 192.168.56.201:5413 | Replica | streaming | 11 |         0 |
| pgsvr2 | 192.168.56.202:5413 | Leader | running   | 11 |           |
+--------+---------------------+---------+-----------+----+-----------+

[postgres@pgserver2 log]v13 $ ps -ef | grep postgres
postgres 6107 3013 0 11:21 pts/0    00:00:18 /usr/bin/python3 /usr/local/bin/patroni /engine/patroni.yml
postgres 7967     1 0 13:52 ?        00:00:00 postgres -D /data/pg13 --config-file=/data/pg13/postgresql.conf --listen_addresses=192.168.56.202 --port=5413 --cluster_name=postgres --wal_level=replica --hot_standby=on --max_connections=500 --max_wal_senders=8 --max_prepared_transactions=0 --max_locks_per_transaction=64 --track_commit_timestamp=off --max_replication_slots=10 --max_worker_processes=8 --wal_log_hints=on
postgres 7968 7967 0 13:52 ?        00:00:00 postgres: postgres: logger
postgres 7974 7967 0 13:52 ?        00:00:00 postgres: postgres: checkpointer
postgres 7975 7967 0 13:52 ?        00:00:00 postgres: postgres: background writer
postgres 7976 7967 0 13:52 ?        00:00:00 postgres: postgres: stats collector
postgres 7979 7967 0 13:52 ?        00:00:00 postgres: postgres: postgres postgres 192.168.56.202(55374) idle
postgres 7984 7967 0 13:52 ?        00:00:00 postgres: postgres: walwriter
postgres 7985 7967 0 13:52 ?        00:00:00 postgres: postgres: autovacuum launcher
postgres 7986 7967 0 13:52 ?        00:00:00 postgres: postgres: logical replication launcher
postgres 7990 7967 0 13:52 ?        00:00:00 postgres: postgres: postgres postgres 192.168.56.202(55378) idle
postgres 7991 7967 0 13:52 ?        00:00:00 postgres: postgres: walsender repl 192.168.56.201(40830) streaming 0/A478EB90

Leader 서버는 변화되지 않았다. 그런데 TL이 변경된 것을 확인 할 수 있다. TL이 변화가 되었으면 나름의 promote로직이 돌았을 것으로 생각되어서 patroni log 파일을 확인해 본다

테스트 결론 :

Leader가 이상이 있을 경우 새로운 Replica를 Leader로 변경하는 대시 기존 Leader를 그대로 유지 하려고 한다

다만 기존 Leader에 문제가 심해서 기동이 안 되거나 할 경우 Replica중 하나를 leader로 변경하여 지속적인 서비스를 가능하게 한다

5. HAproxy

Application에서는 Primary가 어느 서버든지 상관하지 않고, 지속적인 서비스를 원할 경우 VIP를 설정해서 장애 발생시 VIP를 넘겨서 사용하거나, router 서버를 이용하여 Primary에 상관하지 않고

지속적인 서비스를 할 수 있다

HAproxy가 router 역할을 수행하면서 patroni와 통신하면서 leader 서버로 Application의 요청을 보내 주는 역할을 한다

yum install -y haproxy

which haproxy
/usr/sbin/haproxy

haproxy -v
HA-Proxy version 1.5.18 2016/05/10
Copyright 2000-2016 Willy Tarreau <willy@haproxy.org>

echo "global
    maxconn 200

defaults
    log global
    mode tcp
    retries 2
    timeout client 30m
    timeout connect 4s
    timeout server 30m
    timeout check 5s

listen stats
    mode http
    bind *:7000
    stats enable
    stats uri /

listen postgres
    bind *:5413
    option httpchk
    http-check expect status 200
    default-server inter 3s fall 3 rise 2 on-marked-down shutdown-sessions
    server postgresql_db-node-1_5413 192.168.56.201:5413 maxconn 200 check port 8008
    server postgresql_db-node-2_5413 192.168.56.202:5413 maxconn 200 check port 8008" > /etc/haproxy/haproxy.cfg

listen stats : HAproxy 서버에 대한 상태 체크 가능 http://ip:7000
listen postgres : Application에 대한 접속을 받아서 해당 시점의 Leader노드의 PostgreSQL로 연결
      server postgresql_[db-node-1_5413] : 서버 이름 지정
                     192.168.56.201:5413 : PostgeSQL 서버 아이피:포트
                             maxconn 200 : 해당 PostgreSQL 서버의 max_connection 값과 일치 시킴
                         check port 8008 : 해당 서버의 patroni restapi port

systemctl restart haproxy
systemctl status haproxy

netstat -tunelp | grep LISTEN
tcp        0      0 192.168.56.203:2379     0.0.0.0:*               LISTEN      0          24473      2320/etcd
tcp        0      0 127.0.0.1:2379          0.0.0.0:*               LISTEN      0          24472      2320/etcd
tcp        0      0 192.168.56.203:2380     0.0.0.0:*               LISTEN      0          24471      2320/etcd
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      0          15932      967/sshd
tcp        0      0 0.0.0.0:7000            0.0.0.0:*               LISTEN      0          25950      2483/haproxy
tcp        0      0 127.0.0.1:7001          0.0.0.0:*               LISTEN      0          24470      2320/etcd
tcp        0      0 127.0.0.1:25            0.0.0.0:*               LISTEN      0          19650      1145/master
tcp        0      0 0.0.0.0:5413            0.0.0.0:*               LISTEN      0          25951      2483/haproxy
tcp6       0      0 :::22                   :::*                    LISTEN      0          15934      967/sshd
tcp6       0      0 ::1:25                  :::*                    LISTEN      0          19651      1145/master

HAproxy 정보 확인 (http://haproxy_server_ip:7000)

Client는 haproxy_server_ip 5413포트를 이용하여 접속

6. 종료 / 기동

종료

HAproxy 종료 --> Replica 서버의 patroni --> Replica 서버의 PostgreSQL --> Leader 서버의 Patroni --> Leader 서버의 PostgreSQL --> etcd 클러스터의 Leader가 아닌 서버 --> etcd 클러스터의 Leader

기동

etcd 기동 (기동 시 알아서 Leader선출) --> leader 서버의 patroni --> replica 서버의 patroni --> HAproxy

덧, 기존에는 oneNote 붙여 넣기가 oneNote랑 똑같이 잘 붙여 넣어 져서 가독성이 좋았었는데

이제는 제대로 붙여 넣어 지지가 않아서 포스팅 하기가 귀찮아지네요 ...

vacuum & long transaction

★깜장별★ — Fri, 1 Jul 2022 12:23:35 +0900

auto vacuum이나 수동으로 vacuum 명령을 수행하여도 실제 vacuum이 수행되지 않는 경우가 발생할 수 있다.

DML이 빈번히 발생하고 있는 테이블이라서 auto vacuum 이나 vacuum이 수행 되어서 dead tuple을 정리해 줘야 하는데 그렇지 못한 경우가 발생

원인 :

DML이 빈번히 발생하는 테이블에 대한 조회 쿼리가 오래 수행되고 있는 중에 vacuum이나 auto vacuum이 수행 되면 해당 명령은 정상적으로 수행 되었다고 나옴

dead_tuple도 삭제 되었다고 나오지만, 실제 블럭을 확인해 보면 정리가 되진 않음 --> 테이블 사이즈는 계속 증가

시나리오 :

세션 1에서 update 수행 --> dead tuple 발생

세션 2에서 해당 테이블을 select하는 long query 수행

세션 1에서 vacuum 수행 --> long 쿼리와 상관없는 테이블은 정상적으로 vacuum이 수행 됨

세션 1	세션 2
drop table if exists t1 ; drop table if exists t2 ; create table t1 ( id int , name varchar(10)) ; create table t2 ( id int , name varchar(10)) ; insert into t1 values ( 1 , 'name1') ; insert into t2 values ( 1 , 'name1') ; update t1 set name = 'changed' where id = 1 ; update t2 set name = 'changed' where id = 1 ; select c.relname, c.reltuples , n_tup_ins , n_tup_upd , n_live_tup , n_dead_tup , last_vacuum from pg_class c , pg_stat_all_tables s where c.relname = s.relname and c.relname in ('t1','t2') ; +---------+-----------+-----------+-----------+------------+------------+-------------+ \| relname \| reltuples \| n_tup_ins \| n_tup_upd \| n_live_tup \| n_dead_tup \| last_vacuum \| +---------+-----------+-----------+-----------+------------+------------+-------------+ \| t1 \| 0 \| 0 \| 0 \| 0 \| 0 \| NULL \| \| t2 \| 0 \| 0 \| 0 \| 0 \| 0 \| NULL \| +---------+-----------+-----------+-----------+------------+------------+-------------+ t1 +-------+--------+--------+-------+--------+ \| tuple \| t_xmin \| t_xmax \| t_cid \| t_ctid \| +-------+--------+--------+-------+--------+ \| 1 \| 1270 \| 1272 \| 0 \| (0,2) \| \| 2 \| 1272 \| 0 \| 0 \| (0,2) \| +-------+--------+--------+-------+--------+ t2 +-------+--------+--------+-------+--------+ \| tuple \| t_xmin \| t_xmax \| t_cid \| t_ctid \| +-------+--------+--------+-------+--------+ \| 1 \| 1271 \| 1273 \| 0 \| (0,2) \| \| 2 \| 1273 \| 0 \| 0 \| (0,2) \| +-------+--------+--------+-------+--------+
	select * , pg_sleep(120) from t2 ;
vacuum t1 ; vacuum t2 ; select c.relname, c.reltuples , n_tup_ins , n_tup_upd , n_live_tup , n_dead_tup , last_vacuum from pg_class c , pg_stat_all_tables s where c.relname = s.relname and c.relname in ('t1','t2') ; +---------+-----------+-----------+-----------+------------+------------+-------------------------------+ \| relname \| reltuples \| n_tup_ins \| n_tup_upd \| n_live_tup \| n_dead_tup \| last_vacuum \| +---------+-----------+-----------+-----------+------------+------------+-------------------------------+ \| t1 \| 1 \| 1 \| 1 \| 1 \| 0 \| 2022-07-01 11:56:24.216476+09 \| \| t2 \| 0 \| 1 \| 1 \| 0 \| 0 \| 2022-07-01 11:56:25.023881+09 \| +---------+-----------+-----------+-----------+------------+------------+-------------------------------+ t1 +-------+--------+--------+-------+--------+ \| tuple \| t_xmin \| t_xmax \| t_cid \| t_ctid \| +-------+--------+--------+-------+--------+ \| 1 \| NULL \| NULL \| NULL \| NULL \| \| 2 \| 1272 \| 0 \| 0 \| (0,2) \| +-------+--------+--------+-------+--------+ t2 +-------+--------+--------+-------+--------+ \| tuple \| t_xmin \| t_xmax \| t_cid \| t_ctid \| +-------+--------+--------+-------+--------+ \| 1 \| 1271 \| 1273 \| 0 \| (0,2) \| \| 2 \| 1273 \| 0 \| 0 \| (0,2) \| +-------+--------+--------+-------+--------+
세션2의 쿼리가 끝난 후 조회 t2 +-------+--------+--------+-------+--------+ \| tuple \| t_xmin \| t_xmax \| t_cid \| t_ctid \| +-------+--------+--------+-------+--------+ \| 1 \| 1271 \| 1273 \| 0 \| (0,2) \| \| 2 \| 1273 \| 0 \| 0 \| (0,2) \| +-------+--------+--------+-------+--------+ vacuum t2 ; +---------+-----------+-----------+-----------+------------+------------+-------------------------------+ \| relname \| reltuples \| n_tup_ins \| n_tup_upd \| n_live_tup \| n_dead_tup \| last_vacuum \| +---------+-----------+-----------+-----------+------------+------------+-------------------------------+ \| t1 \| 1 \| 1 \| 1 \| 1 \| 0 \| 2022-07-01 11:56:24.216476+09 \| \| t2 \| 1 \| 1 \| 1 \| 1 \| 0 \| 2022-07-01 12:00:27.061505+09 \| +---------+-----------+-----------+-----------+------------+------------+-------------------------------+ t2 +-------+--------+--------+-------+--------+ \| tuple \| t_xmin \| t_xmax \| t_cid \| t_ctid \| +-------+--------+--------+-------+--------+ \| 1 \| NULL \| NULL \| NULL \| NULL \| \| 2 \| 1273 \| 0 \| 0 \| (0,2) \| +-------+--------+--------+-------+--------+

View의 특이 사항

★깜장별★ — Sun, 19 Jun 2022 11:00:27 +0900

PostgreSQL의 View는 약간 다르게 동작하는 부분이 있어서 테스트 진행

View 특징

1. View에서 참조하는 테이블의 이름이 변경되면, View안의 테이블이름도 변경된다

2. View가 존재 하는 원본 테이블을 삭제 하려고 하거나, View에서 참조하는 칼럼을 변경하려고 하면 에러가 난다.

1. 원본 테이블 명 변경 시 View 내용 변경

9.3 ~ 14까지 다 테스트를 진행해 보니 실제로 참조 테이블에서 이름을 변경하게 되면 View안에서 해당 테이블 이름도 자동으로 변경 된다.

Why ?

PostgreSQL은 다른 DBMS와 다르게 View 생성 SQL문을 저장해 놓고 있지 않다. 즉, view에 대한 SQL문이 저장되어서 쿼리 수행 시 Rewrite가 되는 것이 아니라,

parse tree 형태로 저장하기 때문이다. 해당 parse tree 형태에서는 테이블 이름을 저장하고 있지 않고, 테이블의 OID값을 사용하기 때문에 테이블을 rename 하더라도

기존의 OID가 변경되지 않기 때문에 자동으로 변경 된 이름의 테이블 명을 그대로 참조하게 된다.

create table emp (
empno    numeric(4,0) not null constraint emp_pk primary key
, ename    varchar(10)
, job      varchar(9)
, mgr      numeric(4,0)
, hiredate timestamp
, sal      numeric(7,2)
, comm     numeric(7,2)
, deptno   numeric(2,0)
);

insert into emp
(empno, ename, job, mgr, hiredate, sal, comm, deptno)
values
(7369, 'SMITH' , 'CLERK'     , 7902, '17-DEC-80 00:00:00', 800.00,    NULL,     20),
(7499, 'ALLEN' , 'SALESMAN' , 7698, '20-FEB-81 00:00:00', 1600.00, 300.00,     30),
(7521, 'WARD'   , 'SALESMAN' , 7698, '22-FEB-81 00:00:00', 1250.00, 500.00,     30),
(7566, 'JONES' , 'MANAGER'   , 7839, '02-APR-81 00:00:00', 2975.00,    NULL,     20),
(7654, 'MARTIN' , 'SALESMAN' , 7698, '28-SEP-81 00:00:00', 1250.00, 1400.00,     30),
(7698, 'BLAKE' , 'MANAGER'   , 7839, '01-MAY-81 00:00:00', 2850.00,    NULL,     30),
(7782, 'CLARK' , 'MANAGER'   , 7839, '09-JUN-81 00:00:00', 2450.00,    NULL,     10),
(7788, 'SCOTT' , 'ANALYST'   , 7566, '19-APR-87 00:00:00', 3000.00,    NULL,     20),
(7839, 'KING'   , 'PRESIDENT' , NULL, '17-NOV-81 00:00:00', 5000.00,    NULL,     10),
(7844, 'TURNER' , 'SALESMAN' , 7698, '08-SEP-81 00:00:00', 1500.00,    0.00,     30),
(7876, 'ADAMS' , 'CLERK'     , 7788, '23-MAY-87 00:00:00', 1100.00,    NULL,     20),
(7900, 'JAMES' , 'CLERK'     , 7698, '03-DEC-81 00:00:00', 950.00,    NULL,     30),
(7902, 'FORD'   , 'ANALYST'   , 7566, '03-DEC-81 00:00:00', 3000.00,    NULL,     20),
(7934, 'MILLER' , 'CLERK'     , 7782, '23-JAN-82 00:00:00', 1300.00,    NULL,     10);

create view v_emp
as
select * from emp ;

select pg_get_viewdef('v_emp') ;
+----------------------+
|    pg_get_viewdef    |
+----------------------+
| SELECT emp."EMPNO",+|
|     emp."ENAME",    +|
|     emp."JOB",      +|
|     emp."MGR",      +|
|     emp."HIREDATE", +|
|     emp."SAL",      +|
|     emp."COMM",     +|
|     emp."DEPTNO"    +|
|    FROM emp;         |
+----------------------+

select oid , relname from pg_class where relname in ('emp' , 'v_emp') ;
+-------+---------+
| oid | relname |
+-------+---------+
| 16481 | v_emp   |
| 16462 | emp     |
+-------+---------+

select ev_action from pg_rewrite where ev_class = (select oid from pg_class where relname = 'v_emp') ;

({QUERY :commandType 1 :querySource 0 :canSetTag true :utilityStmt <> :resultRelation 0 :hasAggs false :hasWindowFuncs false :hasTargetSRFs false :hasSubLinks false :hasDistinctOn false :hasRecursive false :hasModifyingCTE false :hasForUpdate false :hasRowSe
curity false :isReturn false :cteList <> :rtable ({RTE :alias {ALIAS :aliasname old :colnames <>} :eref {ALIAS :aliasname old :colnames ("EMPNO" "ENAME" "JOB" "MGR" "HIREDATE" "SAL" "COMM" "DEPTNO")} :rtekind 0 :relid 16481 :relkind v :rellockmode 1 :tablesample <> :lat
eral false :inh false :inFromCl false :requiredPerms 0 :checkAsUser 0 :selectedCols (b) :insertedCols (b) :updatedCols (b) :extraUpdatedCols (b) :securityQuals <>} {RTE :alias {ALIAS :aliasname new :colnames <>} :eref {ALIAS :aliasname new :colnames ("EMPNO" "ENAME" "JO
B" "MGR" "HIREDATE" "SAL" "COMM" "DEPTNO")} :rtekind 0 :relid 16481 :relkind v :rellockmode 1 :tablesample <> :lateral false :inh false :inFromCl false :requiredPerms 0 :checkAsUser 0 :selectedCols (b) :insertedCols (b) :updatedCols (b) :extraUpdatedCols (b) :securityQu
als <>} {RTE :alias <> :eref {ALIAS :aliasname emp :colnames ("EMPNO" "ENAME" "JOB" "MGR" "HIREDATE" "SAL" "COMM" "DEPTNO")} :rtekind 0 :relid 16462 :relkind r :rellockmode 1 :tablesample <> :lateral false :inh true :inFromCl true :requiredPerms 2 :checkAsUser 0 :select
edCols (b 8 9 10 11 12 13 14 15) :insertedCols (b) :updatedCols (b) :extraUpdatedCols (b) :securityQuals <>}) :jointree {FROMEXPR :fromlist ({RANGETBLREF :rtindex 3}) :quals <>} :targetList ({TARGETENTRY :expr {VAR :varno 3 :varattno 1 :vartype 701 :vartypmod -1 :varcol
lid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 1 :location 28} :resno 1 :resname EMPNO :ressortgroupref 0 :resorigtbl 16462 :resorigcol 1 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 2 :vartype 25 :vartypmod -1 :varcollid 100 :varlevelsup 0 :varnosyn 3 :var
attnosyn 2 :location 28} :resno 2 :resname ENAME :ressortgroupref 0 :resorigtbl 16462 :resorigcol 2 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 3 :vartype 25 :vartypmod -1 :varcollid 100 :varlevelsup 0 :varnosyn 3 :varattnosyn 3 :location 28} :resno 3 :re
sname JOB :ressortgroupref 0 :resorigtbl 16462 :resorigcol 3 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 4 :vartype 701 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 4 :location 28} :resno 4 :resname MGR :ressortgroupref 0 :resorigtbl
16462 :resorigcol 4 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 5 :vartype 1184 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 5 :location 28} :resno 5 :resname HIREDATE :ressortgroupref 0 :resorigtbl 16462 :resorigcol 5 :resjunk fals
e} {TARGETENTRY :expr {VAR :varno 3 :varattno 6 :vartype 701 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 6 :location 28} :resno 6 :resname SAL :ressortgroupref 0 :resorigtbl 16462 :resorigcol 6 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :var
attno 7 :vartype 701 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 7 :location 28} :resno 7 :resname COMM :ressortgroupref 0 :resorigtbl 16462 :resorigcol 7 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 8 :vartype 701 :vartypmod -1 :var
collid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 8 :location 28} :resno 8 :resname DEPTNO :ressortgroupref 0 :resorigtbl 16462 :resorigcol 8 :resjunk false}) :override 0 :onConflict <> :returningList <> :groupClause <> :groupDistinct false :groupingSets <> :havingQual <
> :windowClause <> :distinctClause <> :sortClause <> :limitOffset <> :limitCount <> :limitOption 0 :rowMarks <> :setOperations <> :constraintDeps <> :withCheckOptions <> :stmt_location 0 :stmt_len 39})

테이블 이름 변경

alter table emp rename to emp_bak ;

select oid , relname from pg_class where relname in ('emp_bak' , 'v_emp') ;
+-------+---------+
| oid | relname |
+-------+---------+
| 16481 | v_emp   |
| 16462 | emp_bak |
+-------+---------+

select pg_get_viewdef('v_emp') ;
+--------------------------+
|      pg_get_viewdef      |
+--------------------------+
| SELECT emp_bak."EMPNO",+|
|     emp_bak."ENAME",    +|
|     emp_bak."JOB",      +|
|     emp_bak."MGR",      +|
|     emp_bak."HIREDATE", +|
|     emp_bak."SAL",      +|
|     emp_bak."COMM",     +|
|     emp_bak."DEPTNO"    +|
|    FROM emp_bak;         |
+--------------------------+

select ev_action from pg_rewrite where ev_class = (select oid from pg_class where relname = 'v_emp') ;

({QUERY :commandType 1 :querySource 0 :canSetTag true :utilityStmt <> :resultRelation 0 :hasAggs false :hasWindowFuncs false :hasTargetSRFs false :hasSubLinks false :hasDistinctOn false :hasRecursive false :hasModifyingCTE false :hasForUpdate false :hasRowSe
curity false :isReturn false :cteList <> :rtable ({RTE :alias {ALIAS :aliasname old :colnames <>} :eref {ALIAS :aliasname old :colnames ("EMPNO" "ENAME" "JOB" "MGR" "HIREDATE" "SAL" "COMM" "DEPTNO")} :rtekind 0 :relid 16481 :relkind v :rellockmode 1 :tablesample <> :lat
eral false :inh false :inFromCl false :requiredPerms 0 :checkAsUser 0 :selectedCols (b) :insertedCols (b) :updatedCols (b) :extraUpdatedCols (b) :securityQuals <>} {RTE :alias {ALIAS :aliasname new :colnames <>} :eref {ALIAS :aliasname new :colnames ("EMPNO" "ENAME" "JO
B" "MGR" "HIREDATE" "SAL" "COMM" "DEPTNO")} :rtekind 0 :relid 16481 :relkind v :rellockmode 1 :tablesample <> :lateral false :inh false :inFromCl false :requiredPerms 0 :checkAsUser 0 :selectedCols (b) :insertedCols (b) :updatedCols (b) :extraUpdatedCols (b) :securityQu
als <>} {RTE :alias <> :eref {ALIAS :aliasname emp :colnames ("EMPNO" "ENAME" "JOB" "MGR" "HIREDATE" "SAL" "COMM" "DEPTNO")} :rtekind 0 :relid 16462 :relkind r :rellockmode 1 :tablesample <> :lateral false :inh true :inFromCl true :requiredPerms 2 :checkAsUser 0 :select
edCols (b 8 9 10 11 12 13 14 15) :insertedCols (b) :updatedCols (b) :extraUpdatedCols (b) :securityQuals <>}) :jointree {FROMEXPR :fromlist ({RANGETBLREF :rtindex 3}) :quals <>} :targetList ({TARGETENTRY :expr {VAR :varno 3 :varattno 1 :vartype 701 :vartypmod -1 :varcol
lid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 1 :location 28} :resno 1 :resname EMPNO :ressortgroupref 0 :resorigtbl 16462 :resorigcol 1 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 2 :vartype 25 :vartypmod -1 :varcollid 100 :varlevelsup 0 :varnosyn 3 :var
attnosyn 2 :location 28} :resno 2 :resname ENAME :ressortgroupref 0 :resorigtbl 16462 :resorigcol 2 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 3 :vartype 25 :vartypmod -1 :varcollid 100 :varlevelsup 0 :varnosyn 3 :varattnosyn 3 :location 28} :resno 3 :re
sname JOB :ressortgroupref 0 :resorigtbl 16462 :resorigcol 3 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 4 :vartype 701 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 4 :location 28} :resno 4 :resname MGR :ressortgroupref 0 :resorigtbl
16462 :resorigcol 4 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 5 :vartype 1184 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 5 :location 28} :resno 5 :resname HIREDATE :ressortgroupref 0 :resorigtbl 16462 :resorigcol 5 :resjunk fals
e} {TARGETENTRY :expr {VAR :varno 3 :varattno 6 :vartype 701 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 6 :location 28} :resno 6 :resname SAL :ressortgroupref 0 :resorigtbl 16462 :resorigcol 6 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :var
attno 7 :vartype 701 :vartypmod -1 :varcollid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 7 :location 28} :resno 7 :resname COMM :ressortgroupref 0 :resorigtbl 16462 :resorigcol 7 :resjunk false} {TARGETENTRY :expr {VAR :varno 3 :varattno 8 :vartype 701 :vartypmod -1 :var
collid 0 :varlevelsup 0 :varnosyn 3 :varattnosyn 8 :location 28} :resno 8 :resname DEPTNO :ressortgroupref 0 :resorigtbl 16462 :resorigcol 8 :resjunk false}) :override 0 :onConflict <> :returningList <> :groupClause <> :groupDistinct false :groupingSets <> :havingQual <
> :windowClause <> :distinctClause <> :sortClause <> :limitOffset <> :limitCount <> :limitOption 0 :rowMarks <> :setOperations <> :constraintDeps <> :withCheckOptions <> :stmt_location 0 :stmt_len 39})

rename전과 비교해 보면 ev_action 부분에 변경된 부분이 없다.

2. view가 참조하는 원본 테이블 변경

View는 원본 테이블에 대한 Dependency가 존재 하게 된다.

그래서 View에서 참조하는 테이블 또는 칼럼이 변경이 되지 않게 하고 있어서, 해당 뷰를 삭제하지 않고는 뷰에서 참조하는 테이블이나 칼럼을 변경 할 수 없게 된다.

drop view v_emp ;

create view v_emp
as select empno , ename , sal from emp ;

select pg_get_viewdef('v_emp') ;
+--------------------+
|   pg_get_viewdef   |
+--------------------+
| SELECT emp.empno,+|
|     emp.ename,    +|
|     emp.sal       +|
|    FROM emp;       |
+--------------------+

                            Table "public.emp"
+----------+-----------------------------+-----------+----------+---------+
| Column |            Type             | Collation | Nullable | Default |
+----------+-----------------------------+-----------+----------+---------+
| empno    | numeric(4,0)                |           | not null |         |
| ename    | character varying(10)       |           |          |         |
| job      | character varying(9)        |           |          |         |
| mgr      | numeric(4,0)                |           |          |         |
| hiredate | timestamp without time zone |           |          |         |
| sal      | numeric(7,2)                |           |          |         |
| comm     | numeric(10,2)               |           |          |         |
| deptno   | numeric(2,0)                |           |          |         |
+----------+-----------------------------+-----------+----------+---------+
Indexes:
    "emp_pk" PRIMARY KEY, btree (empno)

칼럼의 사이즈 변경

alter table emp alter column sal type numeric(10,2) ;
ERROR: cannot alter type of a column used by a view or rule
DETAIL: rule _RETURN on view v_emp depends on column "sal"

칼럼 삭제

alter table emp drop column sal ;
ERROR: cannot drop column sal of table emp because other objects depend on it
DETAIL: view v_emp depends on column sal of table emp
HINT: Use DROP ... CASCADE to drop the dependent objects too

참조하지 않는 칼럼 사이즈 변경

alter table emp alter column comm type numeric(10,2) ;
ALTER TABLE

참조하는 테이블 삭제

drop table emp ;
ERROR: cannot drop table emp because other objects depend on it
DETAIL: view v_emp depends on table emp
HINT: Use DROP ... CASCADE to drop the dependent objects too.

참조 테이블에 새로운 칼럼 추가

alter table emp add column new_col varchar(10) ;
ALTER TABLE

View에서 참조하는 테이블 및 칼럼 확인

select v.oid::regclass as view
       , a.attrelid::regclass as table
       , a.attname as column
from pg_attribute as a
join pg_depend as d on d.refobjsubid = a.attnum and d.refobjid = a.attrelid
join pg_rewrite as r on r.oid = d.objid
join pg_class as v on v.oid = r.ev_class
where v.relkind = 'v'    -- only interested in views
and   d.classid = 'pg_rewrite'::regclass
and   d.refclassid = 'pg_class'::regclass
and   d.deptype = 'n'    -- normal dependency
and   v.relname = 'v_emp' ;

+-------+-------+--------+
| view | table | column |
+-------+-------+--------+
| v_emp | emp   | empno |
| v_emp | emp   | ename |
| v_emp | emp   | sal    |
+-------+-------+--------+

PL/pgSQL

★깜장별★ — Tue, 7 Jun 2022 15:45:27 +0900

PL/SQL 14버전 전까지는 OUT 이 없음. IN, INOUT만 존재 함.

DBMS_OUTPUT.PUT_LINE('I got here:'||:new.col||' is the new value'); --> RAISE NOTICE 'I got here:% is the new value', NEW.col;

프로시저는 V11부터 사용가능

Anonymous

\timing on

create table t(
               id serial primary key,
               n numeric not null );

insert into t(n)
select 1 from generate_series(1,100000);

do $$
begin
    for i in 1..150 loop
        for j in 1..10000 loop
            insert into part_test_tri values ( i , j , to_date('20211231','YYYYMMDD')+i );
        end loop;
    end loop;
end $$;

do $$
declare
i record;
begin
for i in (select id from t) loop
execute 'update t set n = n + 1 where id = '||i.id;
end loop;
end;
$$ language plpgsql;

For Loop with SQL

CREATE OR REPLACE FUNCTION emp_members()
RETURNS NUMERIC AS $$
DECLARE
    r RECORD;
    total NUMERIC := 0;
BEGIN
    FOR r IN SELECT * FROM emp
    LOOP
        total := total + 1;
        RAISE NOTICE 'empno : % ename : %   salary : %' , r.empno ,r.ename, r.sal ;
    END LOOP;
    RETURN total;
END;
$$ LANGUAGE plpgsql;

select emp_members() ;

NOTICE: empno : 7369 ename : SMITH   salary : 800.00
NOTICE: empno : 7499 ename : ALLEN   salary : 1600.00
NOTICE: empno : 7521 ename : WARD   salary : 1250.00
NOTICE: empno : 7566 ename : JONES   salary : 2975.00
NOTICE: empno : 7654 ename : MARTIN   salary : 1250.00
NOTICE: empno : 7698 ename : BLAKE   salary : 2850.00
NOTICE: empno : 7782 ename : CLARK   salary : 2450.00
NOTICE: empno : 7788 ename : SCOTT   salary : 3000.00
NOTICE: empno : 7839 ename : KING   salary : 5000.00
NOTICE: empno : 7844 ename : TURNER   salary : 1500.00
NOTICE: empno : 7876 ename : ADAMS   salary : 1100.00
NOTICE: empno : 7900 ename : JAMES   salary : 950.00
NOTICE: empno : 7902 ename : FORD   salary : 3000.00
NOTICE: empno : 7934 ename : MILLER   salary : 1300.00
+-------------+
| emp_members |
+-------------+
|          14 |
+-------------+

CREATE OR REPLACE PROCEDURE p_emp_members()
AS $$
DECLARE
    r RECORD;
BEGIN
    FOR r IN SELECT * FROM emp
    LOOP
        RAISE NOTICE 'empno : % ename : %   salary : %' , r.empno ,r.ename, r.sal ;
    END LOOP;
END;
$$ LANGUAGE plpgsql;

call p_emp_members() ;

NOTICE: empno : 7369 ename : SMITH   salary : 800.00
NOTICE: empno : 7499 ename : ALLEN   salary : 1600.00
NOTICE: empno : 7521 ename : WARD   salary : 1250.00
NOTICE: empno : 7566 ename : JONES   salary : 2975.00
NOTICE: empno : 7654 ename : MARTIN   salary : 1250.00
NOTICE: empno : 7698 ename : BLAKE   salary : 2850.00
NOTICE: empno : 7782 ename : CLARK   salary : 2450.00
NOTICE: empno : 7788 ename : SCOTT   salary : 3000.00
NOTICE: empno : 7839 ename : KING   salary : 5000.00
NOTICE: empno : 7844 ename : TURNER   salary : 1500.00
NOTICE: empno : 7876 ename : ADAMS   salary : 1100.00
NOTICE: empno : 7900 ename : JAMES   salary : 950.00
NOTICE: empno : 7902 ename : FORD   salary : 3000.00
NOTICE: empno : 7934 ename : MILLER   salary : 1300.00

create table normal ( id int ) ;

DO
$$
DECLARE
   v_cnt int ;
   v_err varchar(255) ;
   v_user_err varchar(255) := 'User Error' ;
BEGIN
    delete from normal where id = 1 ; -- 영향 받는 데이터 0 건
    GET DIAGNOSTICS v_cnt = ROW_COUNT; -- sql%rowcount
    RAISE NOTICE '%',v_cnt ;           -- dbms_output.put_line
    select 1/v_cnt ;                   -- 1/0 으로 하려니 에러 발생
exception
when others then
   v_err := sqlerrm ;
   -- raise exception using message = v_err ;    -- raise_application_error (sqlerrm 출력)
   raise exception using message = v_user_err ; -- user Error 출력
END;
$$ LANGUAGE plpgsql;

NOTICE: 0
ERROR: User Error
CONTEXT: PL/pgSQL function inline_code_block line 15 at RAISE (14861)

DO
$$
DECLARE
   v_cnt int ;
   v_err varchar(255) ;
   v_user_err varchar(255) := 'User Error' ;
BEGIN
    delete from normal where id = 1 ; -- 영향 받는 데이터 0 건
    GET DIAGNOSTICS v_cnt = ROW_COUNT; -- sql%rowcount
    RAISE NOTICE '%',v_cnt ;           -- dbms_output.put_line
    select 1/v_cnt ;                   -- 1/0 으로 하려니 에러 발생
exception
when others then
   v_err := sqlerrm ;
   raise exception using message = v_err ;    -- raise_application_error (sqlerrm 출력)
   --raise exception using message = v_user_err ; -- user Error 출력
END;
$$ LANGUAGE plpgsql;

NOTICE: 0
ERROR: division by zero (11385)
CONTEXT: PL/pgSQL function inline_code_block line 14 at RAISE (14861)

Procedure에서 Function call

CREATE OR REPLACE PROCEDURE p_test()
LANGUAGE plpgsql
AS $$
DECLARE
    v_return varchar(20) ;
BEGIN
    --v_return := execute f_test();
    select f_test() into v_return ;
RAISE NOTICE '%',v_return ;
END;
$$;

CREATE OR REPLACE FUNCTION f_test()
RETURNS varchar AS $$
        BEGIN
                RETURN 'executed f_test';
        END;
$$ LANGUAGE plpgsql;

postgres=# call p_test() ;
NOTICE: executed f_test

트랜잭션 안에서 Now()값이 변화하지 않음

★깜장별★ — Tue, 7 Jun 2022 11:27:34 +0900

PostgreSQL은 트랜잭션 안에서 Now() 값이 변화하지 않는다.

하나의 트랜잭션 안에서 동일한 값을 사용해야 하면 상관은 없지만, 그렇지 않을 경우는 now() 함수를 사용하면 안 된다.

하나의 트랜잭션 안에서 다른 시간 값을 사용하려면, clock_timestamp()값을 사용해야 한다.

now() = current_timestamp = transaction_timestamp() : 트랜잭션의 시작 시간을 리턴

clock_timestamp() : 현재 시간 리턴

테이블에서 실제 Insert 시간이 필요할 경우는 now()를 사용하지 말고 clock_timestamp()를 사용해야 실제 insert 시간을 넣을 수 있다.

begin을 이용해서 동일한 시간대의 값을 여러 테이블에 저장할 경우는 데이터 일관성을 맞추기 위해서 now()

[local]:5413 postgres@postgres # select now(), current_timestamp , clock_timestamp() ;
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| now | current_timestamp | clock_timestamp |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| 2022-06-07 11:08:54.631999+09 | 2022-06-07 11:08:54.631999+09 | 2022-06-07 11:08:54.632143+09 |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+

[local]:5413 postgres@postgres # begin ;

[local]:5413 postgres@postgres # select now(), current_timestamp , clock_timestamp() ;
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| now | current_timestamp | clock_timestamp |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| 2022-06-07 11:09:11.256782+09 | 2022-06-07 11:09:11.256782+09 | 2022-06-07 11:09:13.608794+09 |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+

[local]:5413 postgres@postgres # select now(), current_timestamp , clock_timestamp() ;
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| now | current_timestamp | clock_timestamp |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| 2022-06-07 11:09:11.256782+09 | 2022-06-07 11:09:11.256782+09 | 2022-06-07 11:09:15.537125+09 |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+

[local]:5413 postgres@postgres # select now(), current_timestamp , clock_timestamp() ;
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| now | current_timestamp | clock_timestamp |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+
| 2022-06-07 11:09:11.256782+09 | 2022-06-07 11:09:11.256782+09 | 2022-06-07 11:09:17.224676+09 |
+-------------------------------------------+--------------------------------------------+-------------------------------------------+

[local]:5413 postgres@postgres # rollback ;

IP 주소/네트워크/호스트/Subnet

★깜장별★ — Tue, 17 May 2022 13:10:29 +0900

IP Address

주소는 32비트 숫자로 표현이 된다.

┌--------------32-------------┐

┌--8--┐┌---8--┐ ┌--8--┐ ┌--8--┐

xxxxxxxx . xxxxxxxx . xxxxxxxx . xxxxxxxx

└옥텟┘

옥텟 : 아이피 주소의 첫 번째 숫자 값으로 A,B,C Class를 확인 할 수 있다.

호스트 : 네트워크 안에서 개별 장치(컴퓨터, 프린터 등이 가지는 호스트 주소)

A 클라스

32비트 중 제일 처음의 비트가 0으로 고정 됨

앞 8비트가 네트워크, 나머지 24비트가 호스트

┌네트워크┐┌-----------호스트-----------┐

0xxxxxxx . xxxxxxxx . xxxxxxxx . xxxxxxxx

└----------┘ 8자리 중 7자리만 사용가능 하므로 2^7-1 = 127 (127은 내부 예약으로 사용 불가)

아이피 대역 : 1~126.0.0.0 ~ 1~126.255.255.255까지 사용가능

예) 10.0.10.1

A 클라스 / 10.0.0.0 이 네트워크 주소 / 0.0.10.1 이 호스트 주소

B 클라스

32비트 중 제일 처음의 비트가 10으로 고정 됨

앞 16비트가 네트워크, 나머지 16비트가 호스트

┌----네트워크----┐┌-----호스트------┐

10xxxxxx . xxxxxxxx . xxxxxxxx . xxxxxxxx

└--------┘ 8자리 중 6자리만 사용가능 하므로 2^6 = 64

A클라스가 127까지 사용하고 있으므로 127+64 = 191

아이피 대역 : 128~191.0.0.0 ~ 128~191.255.255.255

예) 172.10.9.5

B 클라스 / 172.10.0.0 이 네트워크 주소 / 0.0.9.5 가 호스트 주소

C 클라스

32비트 중 제일 처음의 비트가 110으로 고정 됨

앞 24비트가 네트워크, 나머지 8비트가 호스트

┌---------네트워크-----------┐┌호스트┐

110xxxxx . xxxxxxxx . xxxxxxxx . xxxxxxxx

└--------┘ 8자리 중 5자리만 사용가능 하므로 2^5 = 32

B클라스가 191까지 사용하고 있으므로 191+32 = 223

아이피 대역 : 192~223.0.0.0 ~ 192~223.255.255.255

예) 198.162.10.9

C 클라스 / 198.162.10.0 이 네트워크 주소 / 0.0.0.9 가 호스트 주소

Subnet

사용 목적 :

각 클라스로 나눠진 네트워크를 서비스의 규모에 맞게 분할(세분화)하여 낭비되는 IP주소 자원을 최소화 하려고 사용한다.

브로드 캐스팅을 사용할 경우, 호스트가 많은 경우는 동일 메세지를 여러 호스트에 보내게 되므로 성능 저하가 올 수 있다.

이때 네트워크의 규모를 줄여서 브로드 캐스팅으로 인한 부하를 줄이는 목적으로도 사용된다.

클라스 단위로 네트워크를 분리하여 사용할 경우,

A 클라스를 예를 들면 32비트 중 앞에 8비트를 제외하면, 호스트가 24비트를 차지하게 되므로 2^24= 16777216 개의 호스트를 가질 수 있다.

subnet 설정 방법

A클라스

10.0.10.0

SubnetMask 255.0.0.0로 설정 또는 10.0.10.0/8

호스트는 2^24 = 16777216개 사용 가능

A클라스에 24자리를 고정시켜서 아이피 관리

10.0.10.0

subnetMask 255.255.255.0 또는 10.0.10.0/24

호스트는 2^8 = 256개 사용 가능

A클라스에 25자리를 고정시켜서 아이피 관리

10.0.10.0

subnetMask 255.255.255.128(<--256-128) 또는 10.0.10.0/25

호스트는 2^7 = 128개 사용 가능.

A클라스에 26자리를 고정 시켜서 아이피 관리

10.0.10.0

subnetMask 255.255.255.192(<--256-64) 또는 10.0.10.0/26

호스트는 2^6 = 64개 사용 가능

Subnet, Subnet Mask, Host 개수

Subnet Mask 표

Subnet	Subnet Mask (Dec)	Host 수	비고
/8	255.0.0.0	16,777,216	A Class Default Subnet Mask
/9	255.128.0.0	8,388,608	A Class Subnet Mask
/10	255.192.0.0	4,194,304
/11	255.224.0.0	2,097,152
/12	255.240.0.0	1,048,576
/13	255.248.0.0	524,288
/14	255.252.0.0	262,144
/15	255.254.0.0	131,072
/16	255.255.0.0	65,536	B Class Default Subnet Mask
/17	255.255.128.0	32,768	B Class Subnet Mask
/18	255.255.192.0	16,384
/19	255.255.224.0	8,192
/20	255.255.240.0	4,096
/21	255.255.248.0	2,048
/22	255.255.252.0	1,024
/23	255.255.254.0	512
/24	255.255.255.0	256	C Class Default Subnet Mask
/25	255.255.255.128	128	C Class Subnet Mask
/26	255.255.255.192	64
/27	255.255.255.224	32
/28	255.255.255.240	16
/29	255.255.255.248	8
/30	255.255.255.252	4
/31	255.255.255.254	2
/32	255.255.255.255	1

갑자기 궁금해서 찾아보고 정리해 본겁니다.

전문적인 네트워크 지식이 있는 것은 아니니 개념 잡는 정도로만 참고해 주세요.

fillfactor

★깜장별★ — Sun, 15 May 2022 20:24:30 +0900

0. OverView

PostgreSQL은 기본적으로 다른 DBMS와 다르게 MVCC를 구현하기 위한 Undo 스페이스가 따로 존재 하지 않고,

데이터가 저장되어있는 블록에 이전 이미지를 저장해 놓고 MVCC를 구현한다.

그래서 Update가 발생하게 되면, 기본 tuple은 플래그를 변경하고, 새로운 Update된 데이터가 새로이 Insert가 된다.

기존 데이터는 Dead tuple로 되고, vacuum이 수행되게 되면 해당 Dead tuple이 있던 부분은 새로이 사용가능한 영역으로 바뀌게 된다. (FSM에서 관리)

그러면 추후 Insert 시에 해당 공간을 사용한다.

그러나 insert, Update가 빈번히 일어나는 테이블일 경우는 vacuum이 돌기 전까지는 dead tuple영역을 사용 할 수가 없게 되므로

테이블 사이즈는 증가 하게 된다. 물론 vacuum이 돌아도 늘어난 테이블 사이즈는 줄어들지 않는다.

Reog를 수행하기 위해서는 Full vacuum을 수행해 줘야 한다.

Full vacuum은 새로운 테이블을 내부적으로 다시 만들고, 데이터를 Insert하는 방식이라고 간단하게 생각하면 된다.

그래서 Insert, Update가 빈번히 발생하는 테이블에서 fillfactor의 중요성을 확인하기 위한 테스트를 진행 함.

테스트는 노트북 VM 환경

Cent7 : 4core / 8G

PostgreSQL : 13.6

Shared buffer : 128 MB

Block size : 8K

autovacuum_naptime : 1min

autovacuum_analyze_scale_factor : 0.1

autovacuum_analyze_threshold : 50

wal_level : replica

max_wal_size : 3G

테스트

기본 fillfactor가 100일때 하나의 블록(8K)에 120개의 tuple이 저장되게 테스트를 진행.

fillfactor를 100, 80, 50, 40, 30으로 변경해 가면서 테스트를 진행 함.

테스트 중에 Autovacuum이 수행 되면 안되므로, 테스트 테이블은 Autovacuum을 OFF 시키고 진행.

하나의 데이터는 insert 한번, Update 세 번 발생하는 로직이다.

주문(order insert) --> 주문확인(check update) --> 배송(delivery update) -> 배송완료(delivery_complite update)

테스트 항목

테이블 사이즈, 인덱스 사이즈, Wal Log 생성 빈도, Hot(%) 등을 테스트

1. 테스트 스크립트

데이터베이스 생성	create database test
테이블/인덱스	--ff100 create table test_ff100 ( item_code char(10) , order_id char(10) , order_flag char(1) , order_date timestamp , check_flag char(1) , check_date timestamp , delivery_flag char(1) , delivery_date timestamp , delivery_complite_flag char(1) , delivery_complite_date timestamp ) ; alter table test_ff100 set (autovacuum_enabled = false) ; create index test_ff100_idx on test_ff100(item_code) ; --ff80 create table test_ff80 ( item_code char(10) , order_id char(10) , order_flag char(1) , order_date timestamp , check_flag char(1) , check_date timestamp , delivery_flag char(1) , delivery_date timestamp , delivery_complite_flag char(1) , delivery_complite_date timestamp ) with (fillfactor = 80); alter table test_ff80 set (autovacuum_enabled = false) ; create index test_ff80_idx on test_ff80(item_code) ; --ff50 create table test_ff50 ( item_code char(10) , order_id char(10) , order_flag char(1) , order_date timestamp , check_flag char(1) , check_date timestamp , delivery_flag char(1) , delivery_date timestamp , delivery_complite_flag char(1) , delivery_complite_date timestamp ) with (fillfactor = 50); alter table test_ff50 set (autovacuum_enabled = false) ; create index test_ff50_idx on test_ff50(item_code) ; --ff40 create table test_ff40 ( item_code char(10) , order_id char(10) , order_flag char(1) , order_date timestamp , check_flag char(1) , check_date timestamp , delivery_flag char(1) , delivery_date timestamp , delivery_complite_flag char(1) , delivery_complite_date timestamp ) with (fillfactor = 40); alter table test_ff40 set (autovacuum_enabled = false) ; create index test_ff40_idx on test_ff40(item_code) ; --ff30 create table test_ff30 ( item_code char(10) , order_id char(10) , order_flag char(1) , order_date timestamp , check_flag char(1) , check_date timestamp , delivery_flag char(1) , delivery_date timestamp , delivery_complite_flag char(1) , delivery_complite_date timestamp ) with (fillfactor = 30); alter table test_ff30 set (autovacuum_enabled = false) ; create index test_ff30_idx on test_ff30(item_code) ;
데이터 생성/변경	테이블 명은 테스트 진행 마다 변경하면서 진행 -- 생성 do $order$ declare i int; begin for i in 1..240000 loop insert into test_ff100 values ( concat('item',lpad(i::bpchar,6,'0')) , 'admin' , 'Y' , now() , NULL , NULL , NULL , NULL , NULL , NULL ) ; end loop; end $order$ ; -- 변경 do $check$ declare i int; begin for i in 1..240000 loop update test_ff100 set check_flag = 'Y' , check_date = now() where item_code = concat('item',lpad(i::bpchar,6,'0'))::bpchar ; end loop; end $check$ ; do $delivery$ declare i int; begin for i in 1..240000 loop update test_ff100 set delivery_flag = 'Y' , delivery_date = now() where item_code = concat('item',lpad(i::bpchar,6,'0'))::bpchar ; end loop; end $delivery$ ; do $deliverycomp$ declare i int; begin for i in 1..240000 loop update test_ff100 set delivery_complite_flag = 'Y' , delivery_complite_date = now() where item_code = concat('item',lpad(i::bpchar,6,'0'))::bpchar ; end loop; end $deliverycomp$ ;
점검 스크립트	analyze test_ff100 ; select c.relname As tablename , c.relpages As n_page, pg_size_pretty(pg_table_size(c.oid)) As t_size, pg_size_pretty(pg_indexes_size(c.oid)) As i_size, pg_stat_get_live_tuples(c.oid) AS n_live_tup, pg_stat_get_dead_tuples(c.oid) AS n_dead_tup, pg_stat_get_tuples_inserted(c.oid) AS n_tup_ins, pg_stat_get_tuples_updated(c.oid) AS n_tup_upd, pg_stat_get_tuples_hot_updated(c.oid) AS n_tup_hot_upd, pg_stat_get_numscans(c.oid) AS seq_scan, (select (sum(pg_stat_get_numscans(i.indexrelid)))::bigint from pg_index i where c.oid = i.indrelid ) AS idx_scan, pg_stat_get_analyze_count(c.oid) AS analyze_count, c.reloptions from pg_class c where relname = 'test_ff100' ; select pg_ls_waldir() ; \! du -sh /data/pg13.6/pg_wal

2. 테스트 결과

2.1 요약

auto vacuum을 끄고 수행한 결과 wal count는 크게 차이가 존재 하지 않는다.

전체 live tuple은 240,000개로 변화가 없고, 테이블 사이즈만 가지고 비교를 하게 되면 fillfactor 50으로 했을 경우가 51M로 가장 적으면서, HOT(%)도 80% 이상을 유지해 준다.

하지만 fillfactor를 30으로 했을 경우 update가 3번 발생해도, tuple이동 없이 기존과 동일한 블록에서 update가 이루어진 것을 확인 할 수 있다 (HOT 100%)

동일 tuple에 update가 3번 발생하여도 50이 30과 비슷한 성능을 유지해 주는 것을 확인할 수 있다.

update 속도도 30이 가장 잘 나오고 있지만, 50도 나름 선방하고 있는 것을 확인할 수 있다.

로직 별 수행 시간 비교

fillfactor별 블럭 개수 비교

3번의 update를 모두 동일 블럭에서 처리 하는 fillfactor 30은 블럭 개수 변화가 없고, 블럭 개수 변화가 없기 때문에 인덱스 사이즈도 초기와 동일한 사이즈를 유지 하고 있다.

fillfactor별 테이블/인덱스 사이즈 비교

2.2 추가 테스트 (autovacuum 켜고 테스트)

wal 파일 개수 변화가 없어서, 기존에 테스트 시에 사용하던 set (autovacuum_enabled = false) 옵션을 빼고 테스트 진행

데이터는 기존에 240,000 --> 2,400,000로 10배 증가.

fillfactor 100, 50, 30 만 테스트 진행

기본 환경 설정으로 테스트 진행

autovacuum : on

autovacuum_naptime : 1min

autovacuum_analyze_scale_factor : 0.1

autovacuum_analyze_threshold : 50

기본 fillfactor를 이용하여 사용을 하게 되면 테이블 사이즈는 가장 작으나, tuple이동이 많아서 해당 정보를 기록하는 wal file이 많이 생성되는 것을 확인할 수 있다.

fillfactor 30은 초기 데이터가 들어가 있는 순서대로 블록의 데이터가 이동 없이 저장되어 있어서, 인덱스 사이즈가 변화하지 않은 것을 확인할 수 있다.

fillfactor 50은 wal file이 많이 생성되지 않고 update 속도가 fillfactor 30과 비슷한 속도를 내 주므로, 역시 중간적인 위치에 있다.

위의 테이블에 데이터가 있는 상태에서 한번 더 동일 로직을 수행시켜 보자...다시 insert, update, update, update

autovacuum이 잘 동작을 해서인지 f100일때 테이블 사이즈가 다른 옵션일 때 보다 작다.

대신 f30일 경우는 insert된 순서대로 데이터가 저장되어 있어서 인덱스를 이용한 Random I/O가 줄어 들 거라는 정도가 장점이 될 것 같다.

테스트 데이터 셋이 작아서 그런 건지, 테스트가 너무 순차적으로 진행되어서 인가 ?

f100 스크립트로 수행하고, f50 스크립트로 수행하고, f30 스크립트로 수행하고 ~~

Fillfactor를 변경함으로써 얻을 수 있는 드라마틱한 결과가 나오지 않았다.

pg_cron

★깜장별★ — Fri, 13 May 2022 11:18:47 +0900

pg_cron : https://access.crunchydata.com/documentation/pg_cron/1.3.1/

위의 문서를 보면 pg 9.5 이상부터 사용할 수 있다고 나오는데, 최신버전으로 컴파일 해 보면 V10부터 사용이 가능하다.

테스트 환경은 V9.3 ~ V14까지 source를 이용하여 하나의 서버에 설치 된 상태이다.

pg_cron을 rpm을 이용하여 설치 할 경우 source코드를 컴파일 해서 설치 하였으므로, PostgreSQL Server가 설치 되어 있지 않다고 하면서 설치가 진행이 되질 않는다.

그래서 source 컴파일을 하는 방식으로 진행을 함

postgres 유저에게 sudo 권한을 부여한 후에 postgres유저로 Engine Home을 변경해 가면서 설치 진행

여러 버전을 컴파일 해야 하므로, pg_cron 원본을 하나 따로 만들어 놓고, Engine(버전) 변경 시 마다 pg_cron 디렉터리를 새로 생성하면서 컴파일 진행

1. Download

su - postgres
cd /engine
git clone https://github.com/citusdata/pg_cron.git

-- 다운 받은 원본은 pg_cron_org으로 변경
mv pg_cron pg_cron_org

2. pg_cron 설치

V95	su - postgres cd /engine rm -rf pg_cron cp -R pg_cron_org pg_cron cd pg_cron -- 환경변수 확인 env $PATH /engine/pg9.5/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/engine/embluk/bin -- 컴파일 make && sudo PATH=$PATH make install gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg9.5/include -I. -I./ -I/engine/pg9.5/include/postgresql/server -I/engine/pg9.5/include/postgresql/internal -D_GNU_SOURCE -c -o src/pg_cron.o src/pg_cron.c src/pg_cron.c: In function ‘ManageCronTask’: src/pg_cron.c:1689:5: error: passing argument 1 of ‘shm_mq_detach’ from incompatible pointer type [-Werror] shm_mq_detach(responseq); ^ In file included from src/pg_cron.c:26:0: /engine/pg9.5/include/postgresql/server/storage/shm_mq.h:66:13: note: expected ‘struct shm_mq ’ but argument is of type ‘struct shm_mq_handle ’ extern void shm_mq_detach(shm_mq ); ^ src/pg_cron.c: At top level: cc1: error: unrecognized command line option "-Wno-implicit-fallthrough" [-Werror] cc1: all warnings being treated as errors make: ** [src/pg_cron.o] Error 1
V96	su - postgres cd /engine rm -rf pg_cron cp -R pg_cron_org pg_cron cd pg_cron -- 환경변수 확인 env $PATH /engine/pg9.6/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/engine/embluk/bin -- 컴파일 make && sudo PATH=$PATH make install gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg9.6/include -I. -I./ -I/engine/pg9.6/include/postgresql/server -I/engine/pg9.6/include/postgresql/internal -D_GNU_SOURCE -c -o src/pg_cron.o src/pg_cron.c src/pg_cron.c: In function ‘ManageCronTask’: src/pg_cron.c:1689:5: error: passing argument 1 of ‘shm_mq_detach’ from incompatible pointer type [-Werror] shm_mq_detach(responseq); ^ In file included from src/pg_cron.c:26:0: /engine/pg9.6/include/postgresql/server/storage/shm_mq.h:66:13: note: expected ‘struct shm_mq ’ but argument is of type ‘struct shm_mq_handle ’ extern void shm_mq_detach(shm_mq ); ^ src/pg_cron.c: At top level: cc1: error: unrecognized command line option "-Wno-implicit-fallthrough" [-Werror] cc1: all warnings being treated as errors make: ** [src/pg_cron.o] Error 1
V10	su - postgres cd /engine rm -rf pg_cron cp -R pg_cron_org pg_cron cd pg_cron -- 환경변수 확인 env $PATH /engine/pg10.20/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/engine/embluk/bin -- 컴파일 make && sudo PATH=$PATH make install gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg10.20/include -I. -I./ -I/engine/pg10.20/include/postgresql/server -I/engine/pg10.20/include/postgresql/internal -D_GNU_SOURCE -c -o src/pg_cron.o src/pg_cron.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg10.20/include -I. -I./ -I/engine/pg10.20/include/postgresql/server -I/engine/pg10.20/include/postgresql/internal -D_GNU_SOURCE -c -o src/job_metadata.o src/job_metadata.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg10.20/include -I. -I./ -I/engine/pg10.20/include/postgresql/server -I/engine/pg10.20/include/postgresql/internal -D_GNU_SOURCE -c -o src/misc.o src/misc.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg10.20/include -I. -I./ -I/engine/pg10.20/include/postgresql/server -I/engine/pg10.20/include/postgresql/internal -D_GNU_SOURCE -c -o src/task_states.o src/task_states.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg10.20/include -I. -I./ -I/engine/pg10.20/include/postgresql/server -I/engine/pg10.20/include/postgresql/internal -D_GNU_SOURCE -c -o src/entry.o src/entry.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -shared -o pg_cron.so src/pg_cron.o src/job_metadata.o src/misc.o src/task_states.o src/entry.o -L/engine/pg10.20/lib -Wl,--as-needed -Wl,-rpath,'/engine/pg10.20/lib',--enable-new-dtags -L/engine/pg10.20/lib -lpq cat pg_cron.sql > pg_cron--1.0.sql /bin/mkdir -p '/engine/pg10.20/lib/postgresql' /bin/mkdir -p '/engine/pg10.20/share/postgresql/extension' /bin/mkdir -p '/engine/pg10.20/share/postgresql/extension' /bin/install -c -m 755 pg_cron.so '/engine/pg10.20/lib/postgresql/pg_cron.so' /bin/install -c -m 644 .//pg_cron.control '/engine/pg10.20/share/postgresql/extension/' /bin/install -c -m 644 .//pg_cron--1.3--1.4.sql .//pg_cron--1.2--1.3.sql .//pg_cron--1.0--1.1.sql .//pg_cron--1.4--1.4-1.sql .//pg_cron--1.1--1.2.sql pg_cron--1.0.sql '/engine/pg10.20/share/postgresql/extension/' # add to postgresql.conf: shared_preload_libraries = 'pg_cron' cron.database_name = 'postgres' -- run as superuser: CREATE EXTENSION pg_cron;
V11	su - postgres cd /engine rm -rf pg_cron cp -R pg_cron_org pg_cron cd pg_cron -- 환경변수 확인 env $PATH /engine/pg11.15/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin:/engine/embluk/bin -- 컴파일 make && sudo PATH=$PATH make install gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg11.15/include -I. -I./ -I/engine/pg11.15/include/postgresql/server -I/engine/pg11.15/include/postgresql/internal -D_GNU_SOURCE -c -o src/pg_cron.o src/pg_cron.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg11.15/include -I. -I./ -I/engine/pg11.15/include/postgresql/server -I/engine/pg11.15/include/postgresql/internal -D_GNU_SOURCE -c -o src/job_metadata.o src/job_metadata.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg11.15/include -I. -I./ -I/engine/pg11.15/include/postgresql/server -I/engine/pg11.15/include/postgresql/internal -D_GNU_SOURCE -c -o src/misc.o src/misc.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg11.15/include -I. -I./ -I/engine/pg11.15/include/postgresql/server -I/engine/pg11.15/include/postgresql/internal -D_GNU_SOURCE -c -o src/task_states.o src/task_states.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -std=c99 -Wall -Wextra -Werror -Wno-unused-parameter -Wno-implicit-fallthrough -Iinclude -I/engine/pg11.15/include -I. -I./ -I/engine/pg11.15/include/postgresql/server -I/engine/pg11.15/include/postgresql/internal -D_GNU_SOURCE -c -o src/entry.o src/entry.c gcc -Wall -Wmissing-prototypes -Wpointer-arith -Wdeclaration-after-statement -Wendif-labels -Wmissing-format-attribute -Wformat-security -fno-strict-aliasing -fwrapv -fexcess-precision=standard -O2 -fPIC -shared -o pg_cron.so src/pg_cron.o src/job_metadata.o src/misc.o src/task_states.o src/entry.o -L/engine/pg11.15/lib -Wl,--as-needed -Wl,-rpath,'/engine/pg11.15/lib',--enable-new-dtags -L/engine/pg11.15/lib -lpq cat pg_cron.sql > pg_cron--1.0.sql /bin/mkdir -p '/engine/pg11.15/lib/postgresql' /bin/mkdir -p '/engine/pg11.15/share/postgresql/extension' /bin/mkdir -p '/engine/pg11.15/share/postgresql/extension' /bin/install -c -m 755 pg_cron.so '/engine/pg11.15/lib/postgresql/pg_cron.so' /bin/install -c -m 644 .//pg_cron.control '/engine/pg11.15/share/postgresql/extension/' /bin/install -c -m 644 .//pg_cron--1.3--1.4.sql .//pg_cron--1.2--1.3.sql .//pg_cron--1.0--1.1.sql .//pg_cron--1.4--1.4-1.sql .//pg_cron--1.1--1.2.sql pg_cron--1.0.sql '/engine/pg11.15/share/postgresql/extension/' # add to postgresql.conf: shared_preload_libraries = 'pg_cron' cron.database_name = 'postgres' -- run as superuser: CREATE EXTENSION pg_cron;

3. pg_cron 사용하기

postgresql.conf 파일에 설정을 하고 기동을 하게 되면, pg_cron launcher라는 프로세스가 기동되는 것을 확인 할 수 있다.

[postgres@postgres pg11.15]$ ps -u postgres uf
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
postgres 17189 0.0 0.0 115680 2224 pts/1    S    May12   0:00 -bash
postgres 22367 0.0 0.0 155448 1844 pts/1    R+   11:19   0:00 \_ ps -u postgres uf
postgres 16937 0.0 0.0 115680 2236 pts/0    S+   May12   0:00 -bash
postgres 22124 0.0 0.6 406832 50876 pts/1    S    09:53   0:00 /engine/pg11.15/bin/postgres -D /data/pg11.15
postgres 22125 0.0 0.0 133668   940 ?        Ss   09:53   0:00 \_ postgres: logger
postgres 22127 0.0 0.0 406984 3236 ?        Ss   09:53   0:00 \_ postgres: checkpointer
postgres 22128 0.0 0.0 406964 2200 ?        Ss   09:53   0:00 \_ postgres: background writer
postgres 22129 0.0 0.0 406964 5096 ?        Ss   09:53   0:00 \_ postgres: walwriter
postgres 22130 0.0 0.0 407384 2124 ?        Ss   09:53   0:00 \_ postgres: autovacuum launcher
postgres 22131 0.0 0.0 136064 1236 ?        Ss   09:53   0:00 \_ postgres: stats collector
postgres 22132 0.0 0.0 408672 7700 ?        Ss   09:53   0:00 \_ postgres: pg_cron launcher
postgres 22133 0.0 0.0 407280 1664 ?        Ss   09:53   0:00 \_ postgres: logical replication launcher

사용 설정은 O/S의 cron과 설정 방법이 같다

┌───────────── min (0 - 59)
│ ┌────────────── hour (0 - 23)
│ │ ┌─────────────── day of month (1 - 31)
│ │ │ ┌──────────────── month (1 - 12)
│ │ │ │ ┌───────────────── day of week (0 - 6) (0 to 6 are Sunday to
│ │ │ │ │ Saturday, or use names; 7 is also Sunday)
│ │ │ │ │
│ │ │ │ │
* * * * *

create extension을 수행하게 되면, cron 스키마가 생성이 되고, 등록 된 Job과 Job 수행결과를 확인 할 수 있는 테이블 과 Job 등록 및 삭제를 할 수 있는 프로시저를 가지게 된다.

select current_database() , current_user ;
+------------------+--------------+
| current_database | current_user |
+------------------+--------------+
| postgres         | postgres     |
+------------------+--------------+

-- 파라미터
SHOW cron.log_run;
+--------------+
| cron.log_run |
+--------------+
| on           |
+--------------+

SELECT name, setting, short_desc FROM pg_settings WHERE name LIKE 'cron.%' ORDER BY name;
+-----------------------------+-----------+----------------------------------------------------+
|            name             | setting |                     short_desc                     |
+-----------------------------+-----------+----------------------------------------------------+
| cron.database_name          | postgres | Database in which pg_cron metadata is kept.        |
| cron.enable_superuser_jobs | on        | Allow jobs to be scheduled as superuser            |
| cron.host                   | localhost | Hostname to connect to postgres.                   |
| cron.log_min_messages       | warning   | log_min_messages for the launcher bgworker.        |
| cron.log_run                | on        | Log all jobs runs into the job_run_details table   |
| cron.log_statement          | on        | Log all cron statements prior to execution.        |
| cron.max_running_jobs       | 32        | Maximum number of jobs that can run concurrently. |
| cron.use_background_workers | off       | Use background workers instead of client sessions. |
+-----------------------------+-----------+----------------------------------------------------+

-- 스키마
\dn
   List of schemas
+--------+----------+
| Name | Owner   |
+--------+----------+
| cron   | postgres |
| public | postgres |
+--------+----------+

select * from pg_namespace ;
+--------------------+----------+-------------------------------------+
|      nspname       | nspowner |               nspacl                |
+--------------------+----------+-------------------------------------+
| pg_toast           |       10 | NULL                                |
| pg_temp_1          |       10 | NULL                                |
| pg_toast_temp_1    |       10 | NULL                                |
| pg_catalog         |       10 | {postgres=UC/postgres,=U/postgres} |
| public             |       10 | {postgres=UC/postgres,=UC/postgres} |
| information_schema |       10 | {postgres=UC/postgres,=U/postgres} |
| cron               |       10 | NULL                                |
+--------------------+----------+-------------------------------------+

-- 테이블
select relname
        , n.nspname as schema
        , pg_catalog.pg_get_userbyid(c.relowner) as "Owner"
from    pg_class c , pg_namespace n
where   c.relnamespace = n.oid
and     n.nspname = 'cron'
and     c.relkind = 'r' ;

+-----------------+--------+----------+
|     relname     | schema | Owner   |
+-----------------+--------+----------+
| job_run_details | cron   | postgres |
| job             | cron   | postgres |
+-----------------+--------+----------+

SELECT * FROM cron.job;
+-------+----------+---------+----------+----------+----------+----------+--------+---------+
| jobid | schedule | command | nodename | nodeport | database | username | active | jobname |
+-------+----------+---------+----------+----------+----------+----------+--------+---------+
+-------+----------+---------+----------+----------+----------+----------+--------+---------+

select * from cron.job_run_details ;
+-------+-------+---------+----------+----------+---------+--------+----------------+------------+----------+
| jobid | runid | job_pid | database | username | command | status | return_message | start_time | end_time |
+-------+-------+---------+----------+----------+---------+--------+----------------+------------+----------+
+-------+-------+---------+----------+----------+---------+--------+----------------+------------+----------+

-- 프로시저
select n.nspname as schema
        , (select rolname from pg_authid a where a.oid = p.proowner) as owner
        , p.proname as procedure_name
        , p.proargnames as args
from    pg_proc p , pg_namespace n
where   p.pronamespace = n.oid
and     n.nspname not like 'pg_%'
and     n.nspname not in ('information_schema' , 'public') ;

+--------+----------+----------------------+------------------------------------------------------+
| schema | owner   |    procedure_name    |                         args                         |
+--------+----------+----------------------+------------------------------------------------------+
| cron   | postgres | schedule             | {schedule,command}                                   |
| cron   | postgres | unschedule           | {job_id}                                             |
| cron   | postgres | job_cache_invalidate | NULL                                                 |
| cron   | postgres | unschedule           | {job_name}                                           |
| cron   | postgres | schedule             | {job_name,schedule,command}                          |
| cron   | postgres | alter_job            | {job_id,schedule,command,database,username,active}   |
| cron   | postgres | schedule_in_database | {job_name,schedule,command,database,username,active} |
+--------+----------+----------------------+------------------------------------------------------+

나중에 Job을 삭제 하기 편하기 위해서 job name을 이용하여 잡을 생성하고 삭제를 진행 하도록 하자

기본으로 cron 스키마가 postgres에 설치 되었기 때문에 postgres DB에서 기본적으로 실행이 된다.

다른 DB에서 수행해야 할 명령을 Job으로 생성하려면, schedule_in_database 프로시저를 이용해서 database 이름과 username을 입력해 주면 동작 된다.

3.1 postgres DB에서 사용하기

-- 테스트 테이블 생성

create table test_cron ( id serial , insertdate timestamp ) ;

-- 현재 시간 확인

select now() ;

+-------------------------------+

| now |

+-------------------------------+

| 2022-05-13 10:46:21.162693+09 |

+-------------------------------+

-- Job 생성

select cron.schedule('test_cron_insert' , '50 * * * *' , $$insert into test_cron(insertdate) values (now()) $$) ;

+----------+

| schedule |

+----------+

| 5 |

+----------+

-- Job 동작 확인

SELECT * FROM cron.job_run_details WHERE jobid = 5 ;

+-------+-------+---------+----------+----------+---------------------------------------------------+-----------+----------------+-------------------------------+-------------------------------+

+-------+-------+---------+----------+----------+---------------------------------------------------+-----------+----------------+-------------------------------+-------------------------------+

| 5 | 4 | 22283 | postgres | postgres | insert into test_cron(insertdate) values (now()) | succeeded | INSERT 0 1 | 2022-05-13 10:50:00.260626+09 | 2022-05-13 10:50:00.264431+09 |

+-------+-------+---------+----------+----------+---------------------------------------------------+-----------+----------------+-------------------------------+-------------------------------+

select * from test_cron ;

+----+---------------------------+

| id | insertdate |

+----+---------------------------+

| 1 | 2022-05-13 10:50:00.26082 |

+----+---------------------------+

3.2 다른 데이터베이스에서 사용하기

-- 데이터 베이스 변경

\c test

select current_database() , current_user ;

+------------------+--------------+

| current_database | current_user |

+------------------+--------------+

| test | postgres |

+------------------+--------------+

-- 테스트 테이블 생성

create table test_cron_test ( id serial , insertdate timestamp ) ;

-- Job 생성 (postgres DB에서 생성)

/c postgres

select current_database() , current_user ;

+------------------+--------------+

| current_database | current_user |

+------------------+--------------+

| postgres | postgres |

+------------------+--------------+

select cron.schedule_in_database('test_cron_insert1' , '57 * * * *' , $$insert into test_cron_test(insertdate) values (now()) $$ , 'test', 'postgres') ;

+----------------------+

| schedule_in_database |

+----------------------+

| 6 |

+----------------------+

SELECT * FROM cron.job_run_details WHERE jobid = 6 ;

+-------+-------+---------+----------+----------+--------------------------------------------------------+-----------+----------------+-------------------------------+-------------------------------+

+-------+-------+---------+----------+----------+--------------------------------------------------------+-----------+----------------+-------------------------------+-------------------------------+

| 6 | 5 | 22302 | test | postgres | insert into test_cron_test(insertdate) values (now()) | succeeded | INSERT 0 1 | 2022-05-13 10:57:00.049459+09 | 2022-05-13 10:57:00.052424+09 |

+-------+-------+---------+----------+----------+--------------------------------------------------------+-----------+----------------+-------------------------------+-------------------------------+

-- Job 동작 확인

\c test

select * from test_cron_test ;

+----+----------------------------+

| id | insertdate |

+----+----------------------------+

| 1 | 2022-05-13 10:57:00.049657 |

+----+----------------------------+

3.3 Job 삭제

\c postgres

select cron.unschedule('test_cron_insert') ;

+------------+

| unschedule |

+------------+

| t |

+------------+

select cron.unschedule('test_cron_insert1') ;

+------------+

| unschedule |

+------------+

| t |

+------------+

DELETE FROM cron.job_run_details ;

Partition Table

★깜장별★ — Thu, 12 May 2022 09:26:54 +0900

PostgreSQL Table partition

0. Overview

파티션 테이블 생성 방법 2가지

Inheritance : Trigger나 rule을 이용하여 데이터를 child 테이블에 넣는 방식. 개별 child 테이블에 인덱스를 각각 생성해 줘야 함.

각각의 child 테이블에 직접 vacuum, analyze 명령을 수행해야 한다.

Declarative : 10 부터 지원. Trigger나 rule을 이용하지 않아도 됨.

V10은 Parent 테이블에 인덱스 생성 시 Child 테이블에 인덱스가 생성되지 않아서 기존처럼 개별적으로 child 테이블에 인덱스를 생성해 줘야 함

V11부터는 parent 테이블에 인덱스 생성 시 Child 테이블에 인덱스도 상속되어서 개별 child 테이블에 자동 생성 됨.

~ 10 까지는 Range, List 파티션만 지원 (SET constraint_exclusion = on; 파티션 푸르닝에 영향)

11 ~ Hash 파티션 지원 (SET enable_partition_pruning = on; 파티션 푸르닝에 영향)

V11에서 파티션 테이블을 Inheritance(Trigger), Inheritance(rule) , Declarative로 생성한 후

Insert테스트를 진행해본 결과

Declarative : 45초

Inheritance(rule) : 3분 36초

Inheritance(Trigger) : 1분 6초

1. Partition 점검 PL/pgSQL

점검 PL/pgSQL 참고

2. Inheritance 방식으로 파티션 테이블 생성

트리거나 rule을 이용하여 파티션에 데이터를 적재 함.

ALTER TABLE child 테이블 NO INHERIT Parent 테이블; 명령을 이용하여 detach

2.1 트리거 사용 파티션

테이블 생성

Parent Table

create table part_test_tri
( ci int ,
cj int ,
logdate date ) ;

Child Table

create table part_test_tri_202201
( check( logdate >= date '2022-01-01' and logdate < date '2022-02-01')) inherits (part_test_tri) ;

create table part_test_tri_202202
( check( logdate >= date '2022-02-01' and logdate < date '2022-03-01')) inherits (part_test_tri) ;

create table part_test_tri_202203
( check( logdate >= date '2022-03-01' and logdate < date '2022-04-01')) inherits (part_test_tri) ;

create table part_test_tri_202204
( check( logdate >= date '2022-04-01' and logdate < date '2022-05-01')) inherits (part_test_tri) ;

create table part_test_tri_202205
( check( logdate >= date '2022-05-01' and logdate < date '2022-06-01')) inherits (part_test_tri) ;

insert function 생성

create or replace function f_part_test_tri_insert()
returns trigger
as $$
begin
    if (NEW.logdate >= date '2022-05-01' and NEW.logdate < date '2022-06-01') then
        insert into part_test_tri_202205 values (NEW.*) ;
    elsif ( NEW.logdate >= date '2022-04-01' and NEW.logdate < date '2022-05-01') then
        insert into part_test_tri_202204 values (NEW.*) ;
    elsif ( NEW.logdate >= date '2022-03-01' and NEW.logdate < date '2022-04-01') then
        insert into part_test_tri_202203 values (NEW.*) ;
    elsif ( NEW.logdate >= date '2022-02-01' and NEW.logdate < date '2022-03-01') then
        insert into part_test_tri_202202 values (NEW.*) ;
    elsif ( NEW.logdate >= date '2022-01-01' and NEW.logdate < date '2022-02-01') then
        insert into part_test_tri_202201 values (NEW.*) ;
    else
        raise exception 'Out of range' ;
    end if ;
    return null ;
end ;
$$
language plpgsql ;

insert trigger 생성

create trigger t_part_test_tri_insert
before insert on part_test_tri
for each row execute procedure f_part_test_tri_insert() ;

테이블 생성 확인

데이터 Insert

do $$
begin
    for i in 1..150 loop
        for j in 1..10000 loop
            insert into part_test_tri values ( i , j , to_date('20211231','YYYYMMDD')+i );
        end loop;
    end loop;
end $$;

인덱스 추가

create index part_test_tri_202201_idx on part_test_tri_202201(ci) ;
create index part_test_tri_202202_idx on part_test_tri_202202(ci) ;
create index part_test_tri_202203_idx on part_test_tri_202203(ci) ;
create index part_test_tri_202204_idx on part_test_tri_202204(ci) ;
create index part_test_tri_202205_idx on part_test_tri_202205(ci) ;

2.2 Rule 사용 파티션

테이블 생성

Parent Table

create table part_test_rule
( ci int ,
cj int ,
logdate date ) ;

Child Table

create table part_test_rule_202201
( check( logdate >= date '2022-01-01' and logdate < date '2022-02-01')) inherits (part_test_rule) ;

create table part_test_rule_202202
( check( logdate >= date '2022-02-01' and logdate < date '2022-03-01')) inherits (part_test_rule) ;

create table part_test_rule_202203
( check( logdate >= date '2022-03-01' and logdate < date '2022-04-01')) inherits (part_test_rule) ;

create table part_test_rule_202204
( check( logdate >= date '2022-04-01' and logdate < date '2022-05-01')) inherits (part_test_rule) ;

create table part_test_rule_202205
( check( logdate >= date '2022-05-01' and logdate < date '2022-06-01')) inherits (part_test_rule) ;

룰 생성

CREATE RULE r_part_test_rule_202201
AS
ON INSERT TO part_test_rule WHERE ( logdate >= date '2022-01-01' and logdate < date '2022-02-01' )
DO INSTEAD insert into part_test_rule_202201 values (NEW.*);

CREATE RULE r_part_test_rule_202202
AS
ON INSERT TO part_test_rule WHERE ( logdate >= date '2022-02-01' and logdate < date '2022-03-01' )
DO INSTEAD insert into part_test_rule_202202 values (NEW.*);

CREATE RULE r_part_test_rule_202203
AS
ON INSERT TO part_test_rule WHERE ( logdate >= date '2022-03-01' and logdate < date '2022-04-01' )
DO INSTEAD insert into part_test_rule_202203 values (NEW.*);

CREATE RULE r_part_test_rule_202204
AS
ON INSERT TO part_test_rule WHERE ( logdate >= date '2022-04-01' and logdate < date '2022-05-01' )
DO INSTEAD insert into part_test_rule_202204 values (NEW.*);

CREATE RULE r_part_test_rule_202205
AS
ON INSERT TO part_test_rule WHERE ( logdate >= date '2022-05-01' and logdate < date '2022-06-01' )
DO INSTEAD insert into part_test_rule_202205 values (NEW.*);

테이블 생성 확인

\d+ part_test_rule
                                Table "public.part_test_rule"
+---------+---------+-----------+----------+---------+---------+--------------+-------------+
| Column | Type   | Collation | Nullable | Default | Storage | Stats target | Description |
+---------+---------+-----------+----------+---------+---------+--------------+-------------+
| ci      | integer |           |          |         | plain   |              |             |
| cj      | integer |           |          |         | plain   |              |             |
| logdate | date    |           |          |         | plain   |              |             |
+---------+---------+-----------+----------+---------+---------+--------------+-------------+
Rules:
    r_part_test_rule_202201 AS
    ON INSERT TO part_test_rule
   WHERE new.logdate >= '2022-01-01'::date AND new.logdate < '2022-02-01'::date DO INSTEAD INSERT INTO part_test_rule_202201 (ci, cj, logdate)
VALUES (new.ci, new.cj, new.logdate)
    r_part_test_rule_202202 AS
    ON INSERT TO part_test_rule
   WHERE new.logdate >= '2022-02-01'::date AND new.logdate < '2022-03-01'::date DO INSTEAD INSERT INTO part_test_rule_202202 (ci, cj, logdate)
VALUES (new.ci, new.cj, new.logdate)
    r_part_test_rule_202203 AS
    ON INSERT TO part_test_rule
   WHERE new.logdate >= '2022-03-01'::date AND new.logdate < '2022-04-01'::date DO INSTEAD INSERT INTO part_test_rule_202203 (ci, cj, logdate)
VALUES (new.ci, new.cj, new.logdate)
    r_part_test_rule_202204 AS
    ON INSERT TO part_test_rule
   WHERE new.logdate >= '2022-04-01'::date AND new.logdate < '2022-05-01'::date DO INSTEAD INSERT INTO part_test_rule_202204 (ci, cj, logdate)
VALUES (new.ci, new.cj, new.logdate)
    r_part_test_rule_202205 AS
    ON INSERT TO part_test_rule
   WHERE new.logdate >= '2022-05-01'::date AND new.logdate < '2022-06-01'::date DO INSTEAD INSERT INTO part_test_rule_202205 (ci, cj, logdate)
VALUES (new.ci, new.cj, new.logdate)
Child tables: part_test_rule_202201,
              part_test_rule_202202,
              part_test_rule_202203,
              part_test_rule_202204,
              part_test_rule_202205

데이터 insert

do $$
begin
    for i in 1..150 loop
        for j in 1..10000 loop
            insert into part_test_rule values ( i , j , to_date('20211231','YYYYMMDD')+i );
        end loop;
    end loop;
end $$;

인덱스 생성

create index part_test_rule_202201_idx on part_test_rule_202201(ci) ;
create index part_test_rule_202202_idx on part_test_rule_202202(ci) ;
create index part_test_rule_202203_idx on part_test_rule_202203(ci) ;
create index part_test_rule_202204_idx on part_test_rule_202204(ci) ;
create index part_test_rule_202205_idx on part_test_rule_202205(ci) ;

2.3 Detach, Attach

테이블 및 인덱스 확인

select relname , relpages, reltuples
from    pg_class
where   relname like 'part_test_tri%' ;
+--------------------------+----------+-----------+
|         relname          | relpages | reltuples |
+--------------------------+----------+-----------+
| part_test_tri_202201_idx |      852 |    310000 |
| part_test_tri_202202_idx |      771 |    280000 |
| part_test_tri_202203_idx |      852 |    310000 |
| part_test_tri_202204_idx |      825 |    300000 |
| part_test_tri_202205_idx |      825 |    300000 |
| part_test_tri_202201     |     1676 |    310000 |
| part_test_tri_202202     |     1514 |    280000 |
| part_test_tri_202203     |     1676 |    310000 |
| part_test_tri_202204     |     1622 |    300000 |
| part_test_tri_202205     |     1622 |    300000 |
| part_test_tri            |        0 |         0 |
+--------------------------+----------+-----------+

select count(*) from part_test_tri ;
+---------+
| count |
+---------+
| 1500000 |
+---------+

select relname , relpages, reltuples
from    pg_class
where   relname like 'part_test_rule%' ;
+---------------------------+----------+-----------+
|          relname          | relpages | reltuples |
+---------------------------+----------+-----------+
| part_test_rule_202202_idx |      771 |    280000 |
| part_test_rule_202203_idx |      852 |    310000 |
| part_test_rule_202201_idx |      852 |    310000 |
| part_test_rule_202204_idx |      825 |    300000 |
| part_test_rule_202205_idx |      825 |    300000 |
| part_test_rule_202201     |     1676 |    310000 |
| part_test_rule_202202     |     1514 |    280000 |
| part_test_rule_202203     |     1676 |    310000 |
| part_test_rule_202204     |     1622 |    300000 |
| part_test_rule_202205     |     1622 |    300000 |
| part_test_rule            |        0 |         0 |
+---------------------------+----------+-----------+

select count(*) from part_test_rule ;
+---------+
| count |
+---------+
| 1500000 |
+---------+

2.3.1 Detach

2.3.2 Attach

ALTER TABLE part_test_tri_202205 INHERIT part_test_tri ;

select count(*) from part_test_tri ;
+---------+
| count |
+---------+
| 1500000 |
+---------+

ALTER TABLE part_test_rule_202205 INHERIT part_test_rule ;

select count(*) from part_test_rule ;
+---------+
| count |
+---------+
| 1500000 |
+---------+

2.3.3 Parent Table 삭제

Child 테이블까지 모두 삭제 할 경우, Parent 테이블을 삭제 할 때 Cascade를 사용해야 한다.

개별 Child 테이블은 Attach 후 drop table 명령으로 삭제

3. Declarative 방식으로 파티션 테이블 생성(V10부터 지원)

트리거나 Rule을 사용하지 않아도 됨.

ALTER TABLE 메인테이블 DETACH PARTITION 파티션 테이블; -- detach

V11 부터는 Parent Table에 인덱스를 생성하게 되면, 자동으로 Child 테이블에도 인덱스가 생성되게 된다.

그래서 기존 V10까지는 pg_inherits테이블에 파티션 테이블에 대한 정보만 저장되었으나, V11부터는 인덱스에 대한 정보도 추가 되게 된다.

테이블 생성

Parent Table

create table part_test
( ci int ,
cj int ,
logdate date )
partition by range (logdate) ;

Child Table

create table part_test_202201
partition of part_test for values from ('2022-01-01') to ('2022-02-01') ;

create table part_test_202202
partition of part_test for values from ('2022-02-01') to ('2022-03-01') ;

create table part_test_202203
partition of part_test for values from ('2022-03-01') to ('2022-04-01');

create table part_test_202204
partition of part_test for values from ('2022-04-01') to ('2022-05-01') ;

create table part_test_202205
partition of part_test for values from ('2022-05-01') to ('2022-06-01') ;

테이블 생성 정보 확인

\d+ part_test

                                  Table "public.part_test"
+---------+---------+-----------+----------+---------+---------+--------------+-------------+
| Column | Type   | Collation | Nullable | Default | Storage | Stats target | Description |
+---------+---------+-----------+----------+---------+---------+--------------+-------------+
| ci      | integer |           |          |         | plain   |              |             |
| cj      | integer |           |          |         | plain   |              |             |
| logdate | date    |           |          |         | plain   |              |             |
+---------+---------+-----------+----------+---------+---------+--------------+-------------+
Partition key: RANGE (logdate)
Partitions: part_test_202201 FOR VALUES FROM ('2022-01-01') TO ('2022-02-01'),
            part_test_202202 FOR VALUES FROM ('2022-02-01') TO ('2022-03-01'),
            part_test_202203 FOR VALUES FROM ('2022-03-01') TO ('2022-04-01'),
            part_test_202204 FOR VALUES FROM ('2022-04-01') TO ('2022-05-01'),
            part_test_202205 FOR VALUES FROM ('2022-05-01') TO ('2022-06-01')

데이터 Insert

do $$
begin
    for i in 1..150 loop
        for j in 1..10000 loop
            insert into part_test values ( i , j , to_date('20211231','YYYYMMDD')+i );
        end loop;
    end loop;
end $$;

인덱스 생성

V10	create index part_test_202201_idx on part_test_202201(ci) ; create index part_test_202202_idx on part_test_202202(ci) ; create index part_test_202203_idx on part_test_202203(ci) ; create index part_test_202204_idx on part_test_202204(ci) ; create index part_test_202205_idx on part_test_202205(ci) ;
V11 ~	create index part_test_idx on part_test(ci) ;

select * from pg_inherits ;

+----------+-----------+----------+

| inhrelid | inhparent | inhseqno |

+----------+-----------+----------+

| 16395 | 16392 | 1 |

| 16398 | 16392 | 1 |

| 16401 | 16392 | 1 |

| 16404 | 16392 | 1 |

| 16407 | 16392 | 1 |

+----------+-----------+----------+

4. Partition Pruning

테스트에 사용할 테이블들 정보 확인

위에서 사용한 PL/pgSQL을 이용하여 정보 확인 가능

테스트에는 V11 서버를 사용함.

테스트 결과 요약 :

Declarative를 이용할 경우, date 또는 to_date 모두 파티션 푸르닝은 되나, 속도 차이가 있으므로 date를 사용하는 것이 좋아 보인다.

Inheritance를 이용할 경우, date를 이용하여 파티션을 나누고, 조건에 date를 이용해야지만 파티션 푸르닝이 된다. 전체 child 스캔이 발생해도 date를 사용하는 것이 성능적으로 좋다.

NOTICE: part_test_tri (inherits with trigger)
NOTICE:     |
NOTICE:     \-- part_test_tri_202201 : T/I size ( 13 MB / 6816 kB ) pages ( 1676 ) rows ( 310000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202201_idx : size ( 6816 kB ) idx_scan ( 0 ) CREATE INDEX part_test_tri_202201_idx ON public.part_test_tri_202201 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202202 : T/I size ( 12 MB / 6168 kB ) pages ( 1514 ) rows ( 280000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202202_idx : size ( 6168 kB ) idx_scan ( 0 ) CREATE INDEX part_test_tri_202202_idx ON public.part_test_tri_202202 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202203 : T/I size ( 13 MB / 6816 kB ) pages ( 1676 ) rows ( 310000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202203_idx : size ( 6816 kB ) idx_scan ( 0 ) CREATE INDEX part_test_tri_202203_idx ON public.part_test_tri_202203 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202204 : T/I size ( 13 MB / 6600 kB ) pages ( 1622 ) rows ( 300000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202204_idx : size ( 6600 kB ) idx_scan ( 0 ) CREATE INDEX part_test_tri_202204_idx ON public.part_test_tri_202204 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202205 : T/I size ( 13 MB / 6600 kB ) pages ( 1622 ) rows ( 300000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202205_idx : size ( 6600 kB ) idx_scan ( 0 ) CREATE INDEX part_test_tri_202205_idx ON public.part_test_tri_202205 USING btree (ci)
NOTICE:
NOTICE: part_test_rule (inherits with rule)
NOTICE:     |
NOTICE:     \-- part_test_rule_202201 : T/I size ( 13 MB / 6816 kB ) pages ( 1676 ) rows ( 310000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_rule_202201_idx : size ( 6816 kB ) idx_scan ( 0 ) CREATE INDEX part_test_rule_202201_idx ON public.part_test_rule_202201 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_rule_202202 : T/I size ( 12 MB / 6168 kB ) pages ( 1514 ) rows ( 280000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_rule_202202_idx : size ( 6168 kB ) idx_scan ( 0 ) CREATE INDEX part_test_rule_202202_idx ON public.part_test_rule_202202 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_rule_202203 : T/I size ( 13 MB / 6816 kB ) pages ( 1676 ) rows ( 310000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_rule_202203_idx : size ( 6816 kB ) idx_scan ( 0 ) CREATE INDEX part_test_rule_202203_idx ON public.part_test_rule_202203 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_rule_202204 : T/I size ( 13 MB / 6600 kB ) pages ( 1622 ) rows ( 300000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_rule_202204_idx : size ( 6600 kB ) idx_scan ( 0 ) CREATE INDEX part_test_rule_202204_idx ON public.part_test_rule_202204 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_rule_202205 : T/I size ( 13 MB / 6600 kB ) pages ( 1622 ) rows ( 300000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_rule_202205_idx : size ( 6600 kB ) idx_scan ( 0 ) CREATE INDEX part_test_rule_202205_idx ON public.part_test_rule_202205 USING btree (ci)
NOTICE:
NOTICE: part_test (Declarative)
NOTICE:     |
NOTICE:     \-- part_test_202201 : T/I size ( 13 MB / 13 MB ) pages ( 1676 ) rows ( 310000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202201_ci_idx : size ( 6816 kB ) idx_scan ( 0 ) CREATE INDEX part_test_202201_ci_idx ON public.part_test_202201 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_202202 : T/I size ( 12 MB / 12 MB ) pages ( 1514 ) rows ( 280000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202202_ci_idx : size ( 6168 kB ) idx_scan ( 0 ) CREATE INDEX part_test_202202_ci_idx ON public.part_test_202202 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_202203 : T/I size ( 13 MB / 13 MB ) pages ( 1676 ) rows ( 310000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202203_ci_idx : size ( 6816 kB ) idx_scan ( 0 ) CREATE INDEX part_test_202203_ci_idx ON public.part_test_202203 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_202204 : T/I size ( 13 MB / 13 MB ) pages ( 1622 ) rows ( 300000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202204_ci_idx : size ( 6600 kB ) idx_scan ( 0 ) CREATE INDEX part_test_202204_ci_idx ON public.part_test_202204 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_202205 : T/I size ( 13 MB / 13 MB ) pages ( 1622 ) rows ( 300000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202205_ci_idx : size ( 6600 kB ) idx_scan ( 0 ) CREATE INDEX part_test_202205_ci_idx ON public.part_test_202205 USING btree (ci)

4.1 Declarative

3월달의 특정 기간의 데이터를 조회 하는데 전체 child 테이블을 액세스 하는지, 원하는 대로 특정 child 테이블을 조회 하는지 테스트 진행

조회 기간은 3월 5일부터 3월 20일까지의 데이터를 조회 하는데, to_date와 date를 이용하여 조회할 진행 함

파라미터 ON

set enable_partition_pruning = on ;

플랜 확인

파라미터 OFF

set enable_partition_pruning = off ;

플랜 확인

explain
select count(*) from part_test where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            QUERY PLAN                                                                             |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27258.44..27258.45 rows=1 width=8)                                                                                                      |
|   -> Gather (cost=27258.22..27258.43 rows=2 width=8)                                                                                                            |
|         Workers Planned: 2                                                                                                                                        |
|         -> Partial Aggregate (cost=26258.22..26258.23 rows=1 width=8)                                                                                           |
|               -> Parallel Append (cost=0.00..26091.17 rows=66822 width=0)                                                                                       |
|                     -> Parallel Seq Scan on part_test_202201 (cost=0.00..5323.06 rows=1 width=0)                                                                |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_202203 (cost=0.00..5323.06 rows=94331 width=0)                                                            |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_202204 (cost=0.00..5151.41 rows=1 width=0)                                                                |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_202205 (cost=0.00..5151.41 rows=1 width=0)                                                                |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_202202 (cost=0.00..4808.12 rows=1 width=0)                                                                |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+

explain
select count(*) from part_test where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                        QUERY PLAN                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27258.44..27258.45 rows=1 width=8)                                                                                              |
|   -> Gather (cost=27258.22..27258.43 rows=2 width=8)                                                                                                    |
|         Workers Planned: 2                                                                                                                                |
|         -> Partial Aggregate (cost=26258.22..26258.23 rows=1 width=8)                                                                                   |
|               -> Parallel Append (cost=0.00..26091.17 rows=66822 width=0)                                                                               |
|                     -> Parallel Seq Scan on part_test_202201 (cost=0.00..5323.06 rows=1 width=0)                                                        |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_202203 (cost=0.00..5323.06 rows=94331 width=0)                                                    |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_202204 (cost=0.00..5151.41 rows=1 width=0)                                                        |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_202205 (cost=0.00..5151.41 rows=1 width=0)                                                        |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_202202 (cost=0.00..4808.12 rows=1 width=0)                                                        |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+

explain
select count(*) from part_test where logdate between date '2022-03-05' and date '2022-03-20' ;
+---------------------------------------------------------------------------------------------------------+
|                                               QUERY PLAN                                                |
+---------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=22846.67..22846.68 rows=1 width=8)                                            |
|   -> Gather (cost=22846.46..22846.67 rows=2 width=8)                                                  |
|         Workers Planned: 2                                                                              |
|         -> Partial Aggregate (cost=21846.46..21846.47 rows=1 width=8)                                 |
|               -> Parallel Append (cost=0.00..21679.40 rows=66822 width=0)                             |
|                     -> Parallel Seq Scan on part_test_202201 (cost=0.00..4411.29 rows=1 width=0)      |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date)) |
|                     -> Parallel Seq Scan on part_test_202203 (cost=0.00..4411.29 rows=94331 width=0) |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date)) |
|                     -> Parallel Seq Scan on part_test_202204 (cost=0.00..4269.06 rows=1 width=0)      |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date)) |
|                     -> Parallel Seq Scan on part_test_202205 (cost=0.00..4269.06 rows=1 width=0)      |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date)) |
|                     -> Parallel Seq Scan on part_test_202202 (cost=0.00..3984.59 rows=1 width=0)      |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date)) |
+---------------------------------------------------------------------------------------------------------+

실제 쿼리를 수행해 보자 (쿼리는 아래의 순서대로 수행)

select count(*) from part_test where logdate between date '2022-03-05' and date '2022-03-20' ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 355.319 ms

select count(*) from part_test where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 1927.607 ms (00:01.928)

select count(*) from part_test where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2217.092 ms (00:02.217)

역시 전체 Child 테이블을 조회 하는데도 date를 사용하는 속도가 현저히 빠르다.

4.2 Inheritance(rule/trigger)

파라미터 ON

set enable_partition_pruning = on ;

플랜 확인(rule)

explain
select count(*) from part_test_rule where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            QUERY PLAN                                                                             |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27256.53..27256.54 rows=1 width=8)                                                                                                      |
|   -> Gather (cost=27256.32..27256.53 rows=2 width=8)                                                                                                            |
|         Workers Planned: 2                                                                                                                                        |
|         -> Partial Aggregate (cost=26256.32..26256.33 rows=1 width=8)                                                                                           |
|               -> Parallel Append (cost=0.00..26089.90 rows=66568 width=0)                                                                                       |
|                     -> Parallel Seq Scan on part_test_rule_202201 (cost=0.00..5323.06 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202203 (cost=0.00..5323.06 rows=93972 width=0)                                                       |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202204 (cost=0.00..5151.41 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202205 (cost=0.00..5151.41 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202202 (cost=0.00..4808.12 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule (cost=0.00..0.00 rows=1 width=0)                                                                     |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+

explain
select count(*) from part_test_rule where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                        QUERY PLAN                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27256.53..27256.54 rows=1 width=8)                                                                                              |
|   -> Gather (cost=27256.32..27256.53 rows=2 width=8)                                                                                                    |
|         Workers Planned: 2                                                                                                                                |
|         -> Partial Aggregate (cost=26256.32..26256.33 rows=1 width=8)                                                                                   |
|               -> Parallel Append (cost=0.00..26089.90 rows=66568 width=0)                                                                               |
|                     -> Parallel Seq Scan on part_test_rule_202201 (cost=0.00..5323.06 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202203 (cost=0.00..5323.06 rows=93972 width=0)                                               |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202204 (cost=0.00..5151.41 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202205 (cost=0.00..5151.41 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule_202202 (cost=0.00..4808.12 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule (cost=0.00..0.00 rows=1 width=0)                                                             |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+

explain
select count(*) from part_test_rule where logdate between date '2022-03-05' and date '2022-03-20' ;
+-------------------------------------------------------------------------------------------------------------+
|                                                 QUERY PLAN                                                  |
+-------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=5910.74..5910.75 rows=1 width=8)                                                  |
|   -> Gather (cost=5910.53..5910.74 rows=2 width=8)                                                        |
|         Workers Planned: 2                                                                                  |
|         -> Partial Aggregate (cost=4910.53..4910.54 rows=1 width=8)                                       |
|               -> Parallel Append (cost=0.00..4744.11 rows=66565 width=0)                                  |
|                     -> Seq Scan on part_test_rule (cost=0.00..0.00 rows=1 width=0)                        |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
|                     -> Parallel Seq Scan on part_test_rule_202203 (cost=0.00..4411.29 rows=93972 width=0) |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
+-------------------------------------------------------------------------------------------------------------+

위의 결과를 확인해 보면 date를 사용하는 경우만 파티션 푸르닝이 되고 나머지는 전체 파티션에 대한 스캔이 일어나는 것을 확인 할 수 있다.
이유는 rule이나 trigger는 파티션 데이터를 넣는 부분(insert function or rule)에 logdate >= date '2022-02-01' and logdate < date '2022-03-01' 처럼
데이터 조건을 체크하는 부분이 있는데, 이 부분과 동일하게 where절을 사용하지 않을 경우 푸르닝이 되지 않는다.

실제 쿼리를 수행해 보자

select count(*) from part_test_rule where logdate between date '2022-03-05' and date '2022-03-20' ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 138.249 ms

select count(*) from part_test_rule where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2068.087 ms (00:02.068)

select count(*) from part_test_rule where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2198.934 ms (00:02.199)

플랜확인 (trigger)

explain
select count(*) from part_test_tri where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            QUERY PLAN                                                                             |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27256.53..27256.54 rows=1 width=8)                                                                                                      |
|   -> Gather (cost=27256.32..27256.53 rows=2 width=8)                                                                                                            |
|         Workers Planned: 2                                                                                                                                        |
|         -> Partial Aggregate (cost=26256.32..26256.33 rows=1 width=8)                                                                                           |
|               -> Parallel Append (cost=0.00..26089.90 rows=66568 width=0)                                                                                       |
|                     -> Parallel Seq Scan on part_test_tri_202201 (cost=0.00..5323.06 rows=1 width=0)                                                            |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202203 (cost=0.00..5323.06 rows=93972 width=0)                                                        |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202204 (cost=0.00..5151.41 rows=1 width=0)                                                            |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202205 (cost=0.00..5151.41 rows=1 width=0)                                                            |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202202 (cost=0.00..4808.12 rows=1 width=0)                                                            |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri (cost=0.00..0.00 rows=1 width=0)                                                                      |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_tri where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2143.009 ms (00:02.143)

explain
select count(*) from part_test_tri where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                        QUERY PLAN                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27256.53..27256.54 rows=1 width=8)                                                                                              |
|   -> Gather (cost=27256.32..27256.53 rows=2 width=8)                                                                                                    |
|         Workers Planned: 2                                                                                                                                |
|         -> Partial Aggregate (cost=26256.32..26256.33 rows=1 width=8)                                                                                   |
|               -> Parallel Append (cost=0.00..26089.90 rows=66568 width=0)                                                                               |
|                     -> Parallel Seq Scan on part_test_tri_202201 (cost=0.00..5323.06 rows=1 width=0)                                                    |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202203 (cost=0.00..5323.06 rows=93972 width=0)                                                |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202204 (cost=0.00..5151.41 rows=1 width=0)                                                    |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202205 (cost=0.00..5151.41 rows=1 width=0)                                                    |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri_202202 (cost=0.00..4808.12 rows=1 width=0)                                                    |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri (cost=0.00..0.00 rows=1 width=0)                                                              |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_tri where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2014.080 ms (00:02.014)

explain
select count(*) from part_test_tri where logdate between date '2022-03-05' and date '2022-03-20' ;
+------------------------------------------------------------------------------------------------------------+
|                                                 QUERY PLAN                                                 |
+------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=5910.74..5910.75 rows=1 width=8)                                                 |
|   -> Gather (cost=5910.53..5910.74 rows=2 width=8)                                                       |
|         Workers Planned: 2                                                                                 |
|         -> Partial Aggregate (cost=4910.53..4910.54 rows=1 width=8)                                      |
|               -> Parallel Append (cost=0.00..4744.11 rows=66565 width=0)                                 |
|                     -> Seq Scan on part_test_tri (cost=0.00..0.00 rows=1 width=0)                        |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))    |
|                     -> Parallel Seq Scan on part_test_tri_202203 (cost=0.00..4411.29 rows=93972 width=0) |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))    |
+------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_tri where logdate between date '2022-03-05' and date '2022-03-20' ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 87.649 ms

역시나 date를 사용해야지만 파티션 푸르닝이 된다.

4.3 Inheritance(rule/trigger) -- date를 사용하지 않고 to_date를 사용하게 생성 후 비교 테스트

date를 사용하지 않고 to_date를 사용하는 방식으로 테이블(part_test_rule2)을 생성 후 푸르닝 테스트를 진행해 보자

조건에 date가 아니고 to_date를 주었을 경우 파티션 푸르닝이 되는지 확인 용

결과 요약 :

to_date를 이용하여 데이터를 분리 시키도록 하였으나, 조회 조건에 to_date를 사용할 경우 파티션 푸르닝이 되질 않음. (조건에 date를 준 것이 속도도 빠르다)

tigger나 rule을 이용할 경우 date로 데이터를 분리 시키고, 조회 조건에 date를 사용하는 것이 성능에 효과적이다.

to_date를 사용하는 방식으로 part_test_rule2 파티션 테이블 생성

create table part_test_rule2
( ci int ,
cj int ,
logdate date ) ;

-- child table
create table part_test_rule2_202201
( check( logdate >= to_date ('2022-01-01','YYYY-MM-DD') and logdate < to_date('2022-02-01','YYYY-MM-DD'))) inherits (part_test_rule2) ;
create table part_test_rule2_202202
( check( logdate >= to_date ( '2022-02-01','YYYY-MM-DD') and logdate < to_date ( '2022-03-01','YYYY-MM-DD'))) inherits (part_test_rule2) ;
create table part_test_rule2_202203
( check( logdate >= to_date ( '2022-03-01','YYYY-MM-DD') and logdate < to_date ( '2022-04-01','YYYY-MM-DD'))) inherits (part_test_rule2) ;
create table part_test_rule2_202204
( check( logdate >= to_date ( '2022-04-01','YYYY-MM-DD') and logdate < to_date ( '2022-05-01','YYYY-MM-DD'))) inherits (part_test_rule2) ;
create table part_test_rule2_202205
( check( logdate >= to_date ( '2022-05-01','YYYY-MM-DD') and logdate < to_date ( '2022-06-01','YYYY-MM-DD'))) inherits (part_test_rule2) ;

CREATE RULE r_part_test_rule2_202201
AS
ON INSERT TO part_test_rule2 WHERE ( logdate >= to_date ('2022-01-01','YYYY-MM-DD') and logdate < to_date ('2022-02-01','YYYY-MM-DD') )
DO INSTEAD insert into part_test_rule2_202201 values (NEW.*);

CREATE RULE r_part_test_rule2_202202
AS
ON INSERT TO part_test_rule2 WHERE ( logdate >= to_date ('2022-02-01','YYYY-MM-DD') and logdate < to_date ('2022-03-01','YYYY-MM-DD') )
DO INSTEAD insert into part_test_rule2_202202 values (NEW.*);

CREATE RULE r_part_test_rule2_202203
AS
ON INSERT TO part_test_rule2 WHERE ( logdate >= to_date ('2022-03-01','YYYY-MM-DD') and logdate < to_date ('2022-04-01','YYYY-MM-DD') )
DO INSTEAD insert into part_test_rule2_202203 values (NEW.*);

CREATE RULE r_part_test_rule2_202204
AS
ON INSERT TO part_test_rule2 WHERE ( logdate >= to_date ('2022-04-01','YYYY-MM-DD') and logdate < to_date ('2022-05-01','YYYY-MM-DD') )
DO INSTEAD insert into part_test_rule2_202204 values (NEW.*);

CREATE RULE r_part_test_rule2_202205
AS
ON INSERT TO part_test_rule WHERE ( logdate >= to_date( '2022-05-01','YYYY-MM-DD') and logdate < to_date ('2022-06-01','YYYY-MM-DD') )
DO INSTEAD insert into part_test_rule2_202205 values (NEW.*);

do $$
begin
    for i in 1..150 loop
        for j in 1..10000 loop
            insert into part_test_rule2 values ( i , j , to_date('20211231','YYYYMMDD')+i );
        end loop;
    end loop;
end $$;

플랜을 확인해 보자

explain
select count(*) from part_test_rule2 where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            QUERY PLAN                                                                             |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27290.56..27290.57 rows=1 width=8)                                                                                                      |
|   -> Gather (cost=27290.34..27290.55 rows=2 width=8)                                                                                                            |
|         Workers Planned: 2                                                                                                                                        |
|         -> Partial Aggregate (cost=26290.34..26290.35 rows=1 width=8)                                                                                           |
|               -> Parallel Append (cost=0.00..26123.91 rows=66571 width=0)                                                                                       |
|                     -> Parallel Seq Scan on part_test_rule2_202201 (cost=0.00..5323.06 rows=1 width=0)                                                          |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202203 (cost=0.00..5323.06 rows=93972 width=0)                                                      |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2 (cost=0.00..5151.41 rows=1 width=0)                                                                 |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202204 (cost=0.00..5151.41 rows=1 width=0)                                                          |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202202 (cost=0.00..4808.12 rows=1 width=0)                                                          |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202205 (cost=0.00..34.00 rows=6 width=0)                                                            |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_rule2 where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2193.900 ms (00:02.194)

explain
select count(*) from part_test_rule2 where logdate >= to_date ('2022-03-05','YYYY-MM-DD') and logdate < to_date ('2022-03-20','YYYY-MM-DD') ;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            QUERY PLAN                                                                            |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27259.63..27259.64 rows=1 width=8)                                                                                                     |
|   -> Gather (cost=27259.41..27259.62 rows=2 width=8)                                                                                                           |
|         Workers Planned: 2                                                                                                                                       |
|         -> Partial Aggregate (cost=26259.41..26259.42 rows=1 width=8)                                                                                          |
|               -> Parallel Append (cost=0.00..26103.29 rows=62447 width=0)                                                                                      |
|                     -> Parallel Seq Scan on part_test_rule2_202201 (cost=0.00..5323.06 rows=1 width=0)                                                         |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202203 (cost=0.00..5323.06 rows=88149 width=0)                                                     |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2 (cost=0.00..5151.41 rows=1 width=0)                                                                |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202204 (cost=0.00..5151.41 rows=1 width=0)                                                         |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202202 (cost=0.00..4808.12 rows=1 width=0)                                                         |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202205 (cost=0.00..34.00 rows=6 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_rule2 where logdate >= to_date ('2022-03-05','YYYY-MM-DD') and logdate < to_date ('2022-03-20','YYYY-MM-DD') ;
+--------+
| count |
+--------+
| 150000 |
+--------+
(1 row)
Time: 2244.484 ms (00:02.244)

explain
select count(*) from part_test_rule2 where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                        QUERY PLAN                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27290.56..27290.57 rows=1 width=8)                                                                                              |
|   -> Gather (cost=27290.34..27290.55 rows=2 width=8)                                                                                                    |
|         Workers Planned: 2                                                                                                                                |
|         -> Partial Aggregate (cost=26290.34..26290.35 rows=1 width=8)                                                                                   |
|               -> Parallel Append (cost=0.00..26123.91 rows=66571 width=0)                                                                               |
|                     -> Parallel Seq Scan on part_test_rule2_202201 (cost=0.00..5323.06 rows=1 width=0)                                                  |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202203 (cost=0.00..5323.06 rows=93972 width=0)                                              |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2 (cost=0.00..5151.41 rows=1 width=0)                                                         |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202204 (cost=0.00..5151.41 rows=1 width=0)                                                  |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202202 (cost=0.00..4808.12 rows=1 width=0)                                                  |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_rule2_202205 (cost=0.00..34.00 rows=6 width=0)                                                    |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_rule2 where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2034.739 ms (00:02.035)

explain
select count(*) from part_test_rule2 where logdate between date '2022-03-05' and date '2022-03-20' ;
+--------------------------------------------------------------------------------------------------------------+
|                                                  QUERY PLAN                                                  |
+--------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=22872.79..22872.80 rows=1 width=8)                                                 |
|   -> Gather (cost=22872.58..22872.79 rows=2 width=8)                                                       |
|         Workers Planned: 2                                                                                   |
|         -> Partial Aggregate (cost=21872.58..21872.59 rows=1 width=8)                                      |
|               -> Parallel Append (cost=0.00..21706.15 rows=66571 width=0)                                  |
|                     -> Parallel Seq Scan on part_test_rule2_202201 (cost=0.00..4411.29 rows=1 width=0)     |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))      |
|                     -> Parallel Seq Scan on part_test_rule2_202203 (cost=0.00..4411.29 rows=93972 width=0) |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))      |
|                     -> Parallel Seq Scan on part_test_rule2 (cost=0.00..4269.06 rows=1 width=0)            |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))      |
|                     -> Parallel Seq Scan on part_test_rule2_202204 (cost=0.00..4269.06 rows=1 width=0)     |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))      |
|                     -> Parallel Seq Scan on part_test_rule2_202202 (cost=0.00..3984.59 rows=1 width=0)     |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))      |
|                     -> Parallel Seq Scan on part_test_rule2_202205 (cost=0.00..28.00 rows=6 width=0)       |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))      |
+--------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_rule2 where logdate between date '2022-03-05' and date '2022-03-20' ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 411.314 ms

모두 Child 테이블에 대한 Full Scan이 수행되고, 파티션 푸르닝이 되질 않지만 date를 사용할 경우 속도 차이가 거의 5배 정도 난다.
rule 방식일 때만 파티션 푸르닝이 되질 않는 것인지 트리거 방식으로 만들어서 같이 테스트 진행

to_date를 사용하는 방식으로 part_test_tri2 파티션 테이블 생성

-- Parent table
create table part_test_tri2
( ci int ,
cj int ,
logdate date ) ;

-- child table
create table part_test_tri2_202201
( check( logdate >= to_date('2022-01-01','YYYY-MM-DD') and logdate < to_date('2022-02-01','YYYY-MM-DD'))) inherits (part_test_tri2) ;
create table part_test_tri2_202202
( check( logdate >= to_date('2022-02-01','YYYY-MM-DD') and logdate < to_date('2022-03-01','YYYY-MM-DD'))) inherits (part_test_tri2) ;
create table part_test_tri2_202203
( check( logdate >= to_date('2022-03-01','YYYY-MM-DD') and logdate < to_date('2022-04-01','YYYY-MM-DD'))) inherits (part_test_tri2) ;
create table part_test_tri2_202204
( check( logdate >= to_date('2022-04-01','YYYY-MM-DD') and logdate < to_date('2022-05-01','YYYY-MM-DD'))) inherits (part_test_tri2) ;
create table part_test_tri2_202205
( check( logdate >= to_date('2022-05-01','YYYY-MM-DD') and logdate < to_date('2022-06-01','YYYY-MM-DD'))) inherits (part_test_tri2) ;

-- insert function

create or replace function f_part_test_tri2_insert()
returns trigger
as $$
begin
    if (NEW.logdate >= to_date('2022-05-01','YYYY-MM-DD') and NEW.logdate < to_date('2022-06-01','YYYY-MM-DD')) then
        insert into part_test_tri2_202205 values (NEW.*) ;
    elsif ( NEW.logdate >= to_date('2022-04-01','YYYY-MM-DD') and NEW.logdate < to_date('2022-05-01','YYYY-MM-DD')) then
        insert into part_test_tri2_202204 values (NEW.*) ;
    elsif ( NEW.logdate >= to_date('2022-03-01','YYYY-MM-DD') and NEW.logdate < to_date('2022-04-01','YYYY-MM-DD')) then
        insert into part_test_tri2_202203 values (NEW.*) ;
    elsif ( NEW.logdate >= to_date('2022-02-01','YYYY-MM-DD') and NEW.logdate < to_date('2022-03-01','YYYY-MM-DD')) then
        insert into part_test_tri2_202202 values (NEW.*) ;
    elsif ( NEW.logdate >= to_date('2022-01-01','YYYY-MM-DD') and NEW.logdate < to_date('2022-02-01','YYYY-MM-DD')) then
        insert into part_test_tri2_202201 values (NEW.*) ;
    else
        raise exception 'Out of range' ;
    end if ;
    return null ;
end ;
$$
language plpgsql ;

-- insert trigger

create trigger t_part_test_tri2_insert
before insert on part_test_tri2
for each row execute procedure f_part_test_tri2_insert() ;

do $$
begin
    for i in 1..150 loop
        for j in 1..10000 loop
            insert into part_test_tri2 values ( i , j , to_date('20211231','YYYYMMDD')+i );
        end loop;
    end loop;
end $$;

플랜을 확인해 보자

explain
select count(*) from part_test_tri2 where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            QUERY PLAN                                                                             |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27256.53..27256.54 rows=1 width=8)                                                                                                      |
|   -> Gather (cost=27256.32..27256.53 rows=2 width=8)                                                                                                            |
|         Workers Planned: 2                                                                                                                                        |
|         -> Partial Aggregate (cost=26256.32..26256.33 rows=1 width=8)                                                                                           |
|               -> Parallel Append (cost=0.00..26089.90 rows=66568 width=0)                                                                                       |
|                     -> Parallel Seq Scan on part_test_tri2_202201 (cost=0.00..5323.06 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202203 (cost=0.00..5323.06 rows=93972 width=0)                                                       |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202204 (cost=0.00..5151.41 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202205 (cost=0.00..5151.41 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202202 (cost=0.00..4808.12 rows=1 width=0)                                                           |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2 (cost=0.00..0.00 rows=1 width=0)                                                                     |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate <= to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
+-------------------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_tri2 where logdate between to_date ('2022-03-05','YYYY-MM-DD') and to_date ('2022-03-20','YYYY-MM-DD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 2087.663 ms (00:02.088)

explain
select count(*) from part_test_tri2 where logdate >= to_date ('2022-03-05','YYYY-MM-DD') and logdate < to_date ('2022-03-20','YYYY-MM-DD') ;
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                            QUERY PLAN                                                                            |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27225.60..27225.61 rows=1 width=8)                                                                                                     |
|   -> Gather (cost=27225.39..27225.60 rows=2 width=8)                                                                                                           |
|         Workers Planned: 2                                                                                                                                       |
|         -> Partial Aggregate (cost=26225.39..26225.40 rows=1 width=8)                                                                                          |
|               -> Parallel Append (cost=0.00..26069.28 rows=62444 width=0)                                                                                      |
|                     -> Parallel Seq Scan on part_test_tri2_202201 (cost=0.00..5323.06 rows=1 width=0)                                                          |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202203 (cost=0.00..5323.06 rows=88149 width=0)                                                      |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202204 (cost=0.00..5151.41 rows=1 width=0)                                                          |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202205 (cost=0.00..5151.41 rows=1 width=0)                                                          |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202202 (cost=0.00..4808.12 rows=1 width=0)                                                          |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2 (cost=0.00..0.00 rows=1 width=0)                                                                    |
|                           Filter: ((logdate >= to_date('2022-03-05'::text, 'YYYY-MM-DD'::text)) AND (logdate < to_date('2022-03-20'::text, 'YYYY-MM-DD'::text))) |
+------------------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_tri2 where logdate >= to_date ('2022-03-05','YYYY-MM-DD') and logdate < to_date ('2022-03-20','YYYY-MM-DD') ;
+--------+
| count |
+--------+
| 150000 |
+--------+
(1 row)
Time: 2174.951 ms (00:02.175)

explain
select count(*) from part_test_tri2 where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
|                                                                        QUERY PLAN                                                                         |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=27256.53..27256.54 rows=1 width=8)                                                                                              |
|   -> Gather (cost=27256.32..27256.53 rows=2 width=8)                                                                                                    |
|         Workers Planned: 2                                                                                                                                |
|         -> Partial Aggregate (cost=26256.32..26256.33 rows=1 width=8)                                                                                   |
|               -> Parallel Append (cost=0.00..26089.90 rows=66568 width=0)                                                                               |
|                     -> Parallel Seq Scan on part_test_tri2_202201 (cost=0.00..5323.06 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202203 (cost=0.00..5323.06 rows=93972 width=0)                                               |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202204 (cost=0.00..5151.41 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202205 (cost=0.00..5151.41 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2_202202 (cost=0.00..4808.12 rows=1 width=0)                                                   |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
|                     -> Parallel Seq Scan on part_test_tri2 (cost=0.00..0.00 rows=1 width=0)                                                             |
|                           Filter: ((logdate >= to_date('20220305'::text, 'YYYYMMDD'::text)) AND (logdate <= to_date('20220320'::text, 'YYYYMMDD'::text))) |
+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_tri2 where logdate between to_date ('20220305','YYYYMMDD') and to_date ('20220320','YYYYMMDD') ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 1969.876 ms (00:01.970)

explain
select count(*) from part_test_tri2 where logdate between date '2022-03-05' and date '2022-03-20' ;
+-------------------------------------------------------------------------------------------------------------+
|                                                 QUERY PLAN                                                  |
+-------------------------------------------------------------------------------------------------------------+
| Finalize Aggregate (cost=22844.77..22844.78 rows=1 width=8)                                                |
|   -> Gather (cost=22844.55..22844.76 rows=2 width=8)                                                      |
|         Workers Planned: 2                                                                                  |
|         -> Partial Aggregate (cost=21844.55..21844.56 rows=1 width=8)                                     |
|               -> Parallel Append (cost=0.00..21678.13 rows=66568 width=0)                                 |
|                     -> Parallel Seq Scan on part_test_tri2_202201 (cost=0.00..4411.29 rows=1 width=0)     |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
|                     -> Parallel Seq Scan on part_test_tri2_202203 (cost=0.00..4411.29 rows=93972 width=0) |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
|                     -> Parallel Seq Scan on part_test_tri2_202204 (cost=0.00..4269.06 rows=1 width=0)     |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
|                     -> Parallel Seq Scan on part_test_tri2_202205 (cost=0.00..4269.06 rows=1 width=0)     |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
|                     -> Parallel Seq Scan on part_test_tri2_202202 (cost=0.00..3984.59 rows=1 width=0)     |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
|                     -> Parallel Seq Scan on part_test_tri2 (cost=0.00..0.00 rows=1 width=0)               |
|                           Filter: ((logdate >= '2022-03-05'::date) AND (logdate <= '2022-03-20'::date))     |
+-------------------------------------------------------------------------------------------------------------+
select count(*) from part_test_tri2 where logdate between date '2022-03-05' and date '2022-03-20' ;
+--------+
| count |
+--------+
| 160000 |
+--------+
(1 row)
Time: 369.237 ms

Partition 정보 확인 PL/pgSQL

★깜장별★ — Wed, 11 May 2022 15:11:12 +0900

파티션 테이블 정보를 가져오는 PL/pgSQL을 만들어 봤습니다.

정보를 하나하나 찾다 보니 귀찮아서 ...

티스토리가 예전에는 원노트 붙여넣기 하면 이쁘게 잘 붙었는데, 이제는 글꼴도 별로 맘에 안들고 --;

터미널 데이터 긁어서 붙여도 이상하게 나오네요 --;

-- PL/pgSQL

do
$$
declare
    v_main       record ;
    v_parent     record ;
    v_child      record ;
    v_index      record ;
    v_check      record ;

    v_p_table_oid  oid ;
    v_loop_main int := 1 ;
    v_loop_idx  int := 1 ;

    v_p_cnt         int ;
    v_c_cnt         int ;
begin
    for v_main in (select * from pg_inherits order by inhparent , inhrelid) loop

        -- get parent table id
        if v_loop_main = 1 then
            v_p_table_oid := v_main.inhparent ;
            v_loop_main := v_loop_main + 1 ;

            select relname
                  , concat(case relhasrules when 't' then '(inherits with rule)' end, case relhastriggers when 't' then '(inherits with trigger)' end, case when relhasrules = 'f' and relhastriggers = 'f' then '(Declarative)' end ) as addition
            into v_parent
            from pg_class where oid = v_p_table_oid
            and  relkind in ('p' , 'r') ;
            GET DIAGNOSTICS v_p_cnt = ROW_COUNT;

            if v_p_cnt >= 1 then
                RAISE NOTICE '%  %' , v_parent.relname , v_parent.addition ;
            end if;
        else
            if v_p_table_oid != v_main.inhparent then
                v_p_table_oid := v_main.inhparent ;

                select relname
                       , concat(case relhasrules when 't' then '(inherits with rule)' end, case relhastriggers when 't' then '(inherits with trigger)' end, case when relhasrules = 'f' and relhastriggers = 'f' then '(Declarative)' end) as addition
                into v_parent
                from pg_class where oid = v_p_table_oid
                and  relkind in ('p' , 'r');
                GET DIAGNOSTICS v_p_cnt = ROW_COUNT;

                if v_p_cnt >= 1 then
                    RAISE NOTICE '  ' ;
                    RAISE NOTICE '%  %' , v_parent.relname , v_parent.addition ;
                end if;
            end if;
        end if;

        -- get child table info

        select *
               ,(select count(*) from pg_indexes i where tablename = c.relname) as index_cnt
               ,pg_size_pretty (pg_table_size(c.oid)) as table_size
               ,pg_size_pretty (pg_indexes_size(c.oid)) as index_size
        into   v_child
        from   pg_class c
        where  relfilenode = v_main.inhrelid
        and    relkind = 'r' ;
        GET DIAGNOSTICS v_c_cnt = ROW_COUNT;

        if v_c_cnt >= 1 then
            RAISE NOTICE '   |';
            RAISE NOTICE '   \-- % : T/I size ( % / % )  pages ( % ) rows ( % ) indexes ( % )  ' , v_child.relname , v_child.table_size, v_child.index_size, v_child.relpages , v_child.reltuples , v_child.index_cnt;
        end if ;

        -- get index info

        if v_child.relhasindex = 't' then
            for v_index in (select  indexname , indexdef , pg_size_pretty (pg_table_size(c.oid)) as index_size , si.*
                            from    pg_indexes i , pg_class c , pg_stat_all_indexes si
                            where   i.tablename = v_child.relname
                            and     i.indexname = c.relname
                            and     i.indexname = si.indexrelname) loop

                if v_loop_idx = 1 then
                    RAISE NOTICE '            |' ;
                    v_loop_idx := v_loop_idx + 1 ;
                end if;

                RAISE NOTICE '            \-- % :  size ( % ) idx_scan ( % )  %  ' , v_index.indexname , v_index.index_size, v_index.idx_scan, v_index.indexdef ;
            end loop;
        end if;

        v_loop_idx := 1 ;

    end loop;
end $$ ;

-- 수행 결과

NOTICE:  part_test_tri  (inherits with trigger)
NOTICE:     |
NOTICE:     \-- part_test_tri_202201 : T/I size ( 13 MB / 6816 kB )  pages ( 1676 ) rows ( 310000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202201_idx :  size ( 6816 kB ) idx_scan ( 0 )  CREATE INDEX part_test_tri_202201_idx ON public.part_test_tri_202201 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202202 : T/I size ( 12 MB / 6168 kB )  pages ( 1514 ) rows ( 280000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202202_idx :  size ( 6168 kB ) idx_scan ( 0 )  CREATE INDEX part_test_tri_202202_idx ON public.part_test_tri_202202 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202203 : T/I size ( 13 MB / 6816 kB )  pages ( 1676 ) rows ( 310000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202203_idx :  size ( 6816 kB ) idx_scan ( 0 )  CREATE INDEX part_test_tri_202203_idx ON public.part_test_tri_202203 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202204 : T/I size ( 13 MB / 6600 kB )  pages ( 1622 ) rows ( 300000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202204_idx :  size ( 6600 kB ) idx_scan ( 0 )  CREATE INDEX part_test_tri_202204_idx ON public.part_test_tri_202204 USING btree (ci)
NOTICE:     |
NOTICE:     \-- part_test_tri_202205 : T/I size ( 13 MB / 6600 kB )  pages ( 1622 ) rows ( 300000 ) indexes ( 1 )
NOTICE:              |
NOTICE:              \-- part_test_tri_202205_idx :  size ( 6600 kB ) idx_scan ( 0 )  CREATE INDEX part_test_tri_202205_idx ON public.part_test_tri_202205 USING btree (ci)
NOTICE:
NOTICE:  part_test_rule  (inherits with rule)
NOTICE:     |
NOTICE:     \-- part_test_rule_202201 : T/I size ( 13 MB / 0 bytes )  pages ( 1676 ) rows ( 310000 ) indexes ( 0 )
NOTICE:     |
NOTICE:     \-- part_test_rule_202202 : T/I size ( 12 MB / 0 bytes )  pages ( 1514 ) rows ( 280000 ) indexes ( 0 )
NOTICE:     |
NOTICE:     \-- part_test_rule_202203 : T/I size ( 13 MB / 0 bytes )  pages ( 1676 ) rows ( 310000 ) indexes ( 0 )
NOTICE:     |
NOTICE:     \-- part_test_rule_202204 : T/I size ( 13 MB / 0 bytes )  pages ( 1622 ) rows ( 300000 ) indexes ( 0 )
NOTICE:     |
NOTICE:     \-- part_test_rule_202205 : T/I size ( 13 MB / 0 bytes )  pages ( 1622 ) rows ( 300000 ) indexes ( 0 )
NOTICE:
NOTICE:  part_test  (Declarative)
NOTICE:     |
NOTICE:     \-- part_test_202201 : T/I size ( 13 MB / 13 MB )  pages ( 1676 ) rows ( 310000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202201_ci_idx :  size ( 6816 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202201_ci_idx ON public.part_test_202201 USING btree (ci)
NOTICE:              \-- part_test_202201_cj_idx :  size ( 6816 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202201_cj_idx ON public.part_test_202201 USING btree (cj)
NOTICE:     |
NOTICE:     \-- part_test_202202 : T/I size ( 12 MB / 12 MB )  pages ( 1514 ) rows ( 280000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202202_ci_idx :  size ( 6168 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202202_ci_idx ON public.part_test_202202 USING btree (ci)
NOTICE:              \-- part_test_202202_cj_idx :  size ( 6168 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202202_cj_idx ON public.part_test_202202 USING btree (cj)
NOTICE:     |
NOTICE:     \-- part_test_202203 : T/I size ( 13 MB / 13 MB )  pages ( 1676 ) rows ( 310000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202203_ci_idx :  size ( 6816 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202203_ci_idx ON public.part_test_202203 USING btree (ci)
NOTICE:              \-- part_test_202203_cj_idx :  size ( 6816 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202203_cj_idx ON public.part_test_202203 USING btree (cj)
NOTICE:     |
NOTICE:     \-- part_test_202204 : T/I size ( 13 MB / 13 MB )  pages ( 1622 ) rows ( 300000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202204_ci_idx :  size ( 6600 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202204_ci_idx ON public.part_test_202204 USING btree (ci)
NOTICE:              \-- part_test_202204_cj_idx :  size ( 6600 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202204_cj_idx ON public.part_test_202204 USING btree (cj)
NOTICE:     |
NOTICE:     \-- part_test_202205 : T/I size ( 13 MB / 13 MB )  pages ( 1622 ) rows ( 300000 ) indexes ( 2 )
NOTICE:              |
NOTICE:              \-- part_test_202205_ci_idx :  size ( 6600 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202205_ci_idx ON public.part_test_202205 USING btree (ci)
NOTICE:              \-- part_test_202205_cj_idx :  size ( 6600 kB ) idx_scan ( 0 )  CREATE INDEX part_test_202205_cj_idx ON public.part_test_202205 USING btree (cj)

CTAS

★깜장별★ — Tue, 10 May 2022 10:17:43 +0900

-- Oracle

CREATE TABLE EMP (

EMPNO NUMBER(4) NOT NULL Primary key,

ENAME VARCHAR2(10),

JOB VARCHAR2(9),

MGR NUMBER(4) ,

HIREDATE DATE,

SAL NUMBER(7,2),

COMM NUMBER(7,2),

DEPTNO NUMBER(2) ,

);

-- PostgreSQL

CREATE TABLE emp (

empno decimal(4) NOT NULL PRIMARY KEY,

ename varchar(10),

job varchar(9),

mgr int ,

hiredate date,

sal decimal(7,2),

comm decimal(7,2),

deptno decimal(2) );

create index emp_deptno on emp(deptno) ;

INSERT INTO EMP VALUES (7839,'KING','PRESIDENT',NULL,'1981-11-17',5000,NULL,10);

INSERT INTO EMP VALUES (7698,'BLAKE','MANAGER',7839,'1981-05-01',2850,NULL,30);

INSERT INTO EMP VALUES (7782,'CLARK','MANAGER',7839,'1981-05-09',2450,NULL,10);

INSERT INTO EMP VALUES (7566,'JONES','MANAGER',7839,'1981-04-01',2975,NULL,20);

INSERT INTO EMP VALUES (7654,'MARTIN','SALESMAN',7698,'1981-09-10',1250,1400,30);

INSERT INTO EMP VALUES (7499,'ALLEN','SALESMAN',7698,'1981-02-11',1600,300,30);

INSERT INTO EMP VALUES (7844,'TURNER','SALESMAN',7698,'1981-08-21',1500,0,30);

INSERT INTO EMP VALUES (7900,'JAMES','CLERK',7698,'1981-12-11',950,NULL,30);

INSERT INTO EMP VALUES (7521,'WARD','SALESMAN',7698,'1981-02-23',1250,500,30);

INSERT INTO EMP VALUES (7902,'FORD','ANALYST',7566,'1981-12-11',3000,NULL,20);

INSERT INTO EMP VALUES (7369,'SMITH','CLERK',7902,'1980-12-09',800,NULL,20);

INSERT INTO EMP VALUES (7788,'SCOTT','ANALYST',7566,'1982-12-22',3000,NULL,20);

INSERT INTO EMP VALUES (7876,'ADAMS','CLERK',7788,'1983-01-15',1100,NULL,20);

INSERT INTO EMP VALUES (7934,'MILLER','CLERK',7782,'1982-01-11',1300,NULL,10);

\d+ emp

Table "public.emp"

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

Indexes:

"emp_pkey" PRIMARY KEY, btree (empno)

"emp_deptno" btree (deptno)

Access method: heap

CTAS #1

create table emp1 as select * from emp ;

-- Not Null, Primary key, Foreign Key, Index 정보 같이 생성 안됨

\d+ emp1

Table "public.emp1"

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

Access method: heap

CTAS #2

create table emp2 (like emp including all) ;

-- Not null, Primary Key, Foreign Key, Index 모두 동일한 구조로 만듦

insert into emp2 select * from emp ;

Table "public.emp2"

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

+----------+-----------------------+-----------+----------+---------+----------+-------------+--------------+-------------+

Indexes:

"emp2_pkey" PRIMARY KEY, btree (empno)

"emp2_deptno_idx" btree (deptno)

Access method: heap

Hot Backup & Recovery

★깜장별★ — Fri, 22 Apr 2022 18:38:09 +0900

HOT Backup & Recovery 테스트

V12부터 recovery.conf 파일이 없어졌으므로 9.6 버전과 12버전만 테스트를 진행 함

########################
  Backup & Archive 설정
########################

mkdir -p /archive/pg96
mkdir -p /archive/pg96/temp
mkdir -p /backup/pg96

mkdir -p /archive/pg12
mkdir -p /archive/pg12/temp
mkdir -p /backup/pg12

chown -R postgres:postgres /archive /backup

##############
Hot backup
##############

    1. 기존에 백업 받았던 파일들 삭제
        rm -rf /backup/pg96/*
    2. Begin backup 수행
        select pg_start_backup('hotbackup',true);
        $PGDATA 안에 backup_label 파일이 생성되면서 정보가 저장 됨
    3. O/S 상에서 File Copy
        cp -a $PGDATA /backup/pg96/
    4. End backup 수행
        select pg_stop_backup();
        $PGDATA 안에 backup_label 파일이 없어 짐


##########
9.6.24
##########

아카이브 설정

    echo "
    wal_level = replica
    archive_mode = on
    archive_command = 'dd conv=fdatasync bs=256k if=%p of=/archive/pg96/temp/%f && mv -f /archive/pg96/temp/%f /archive/pg96' " >> $PGDATA/postgres.conf

PG start

archive가 정상적으로 동작 되는지 테스트

    [local]:5496 postgres@postgres # select pg_current_xlog_location();
    +--------------------------+
    | pg_current_xlog_location |
    +--------------------------+
    | 0/1589CE0                |
    +--------------------------+

    [postgres@postgres pg96]$ ls $PGDATA/pg_xlog
    000000010000000000000001  archive_status

    [local]:5496 postgres@postgres # select pg_switch_xlog() ;
    +----------------+
    | pg_switch_xlog |
    +----------------+
    | 0/1589CF8      |
    +----------------+

    [postgres@postgres pg96]$ ls $PGDATA/pg_xlog
    000000010000000000000001  000000010000000000000002  archive_status

    [postgres@postgres pg96]$ ls /archive/pg96
    000000010000000000000001  temp

Hot backup 시작

    [local]:5496 postgres@postgres # select pg_start_backup('hotbackup',true);
    +-----------------+
    | pg_start_backup |
    +-----------------+
    | 0/3000028       |
    +-----------------+
    (1 row)

    [postgres@postgres pg96]$ cat $PGDATA/backup_label
    START WAL LOCATION: 0/3000028 (file 000000010000000000000003)
    CHECKPOINT LOCATION: 0/3000060
    BACKUP METHOD: pg_start_backup
    BACKUP FROM: master
    START TIME: 2022-04-22 16:35:46 KST
    LABEL: hotbackup

File Copy

    [postgres@postgres pg96]$ cp -a $PGDATA /backup/pg96/

    [postgres@postgres pg96]$ ls /backup/pg96
    pg9.6

    [postgres@postgres pg96]$ ls /backup/pg96/pg9.6
    backup_label  global   pg_clog       pg_dynshmem  pg_ident.conf  pg_logical    pg_notify    pg_serial     pg_stat      pg_subtrans  pg_twophase  pg_xlog               postgresql.conf  postmaster.pid
    base          logfile  pg_commit_ts  pg_hba.conf  pg_log         pg_multixact  pg_replslot  pg_snapshots  pg_stat_tmp  pg_tblspc    PG_VERSION   postgresql.auto.conf  postmaster.opts

End backup

    [local]:5496 postgres@postgres # select pg_stop_backup();
    NOTICE:  pg_stop_backup complete, all required WAL segments have been archived
    +----------------+
    | pg_stop_backup |
    +----------------+
    | 0/3000130      |
    +----------------+
    (1 row)

    [postgres@postgres pg96]$ cat $PGDATA/backup_label
    cat: /data/pg9.6/backup_label: No such file or directory

Test Data 추가

    [local]:5496 postgres@postgres # \l
                                    List of databases
    +-----------+----------+----------+-------------+-------------+-----------------------+
    |   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    | postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       |
    | template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    | template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    (3 rows)

    [local]:5496 postgres@postgres # create database test ;
    CREATE DATABASE
    Time: 543.756 ms
    [local]:5496 postgres@postgres # \c test
    You are now connected to database "test" as user "postgres".
    [local]:5496 postgres@test # create table test ( id serial , insert_date timestamp without time zone ) ;
    CREATE TABLE
    Time: 46.379 ms
    [local]:5496 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1
    Time: 44.350 ms
    [local]:5496 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1
    Time: 42.240 ms
    [local]:5496 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1
    Time: 12.298 ms
    [local]:5496 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1
    Time: 13.200 ms
    [local]:5496 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1
    Time: 12.544 ms
    [local]:5496 postgres@test # select * from test ;
    +----+----------------------------+
    | id |        insert_date         |
    +----+----------------------------+
    |  1 | 2022-04-22 16:45:40.541072 |
    |  2 | 2022-04-22 16:45:44.366287 |
    |  3 | 2022-04-22 16:45:45.156317 |
    |  4 | 2022-04-22 16:45:46.165845 |
    |  5 | 2022-04-22 16:45:46.999159 |
    +----+----------------------------+
    (5 rows)

장애 발생
    장애가 발생한 것으로 가정하기 위하여 PG Down

    archive가 어디까지 적용 되었는지 확인
    [postgres@postgres pg9.6]$ ls /archive/pg96
    000000010000000000000001  000000010000000000000002  000000010000000000000003  000000010000000000000003.00000028.backup  000000010000000000000004  temp

    xlog는 어디까지 생성되어 있는지 확인
    [postgres@postgres pg9.6]$ ls $PGDATA/pg_xlog
    000000010000000000000003.00000028.backup  000000010000000000000004  000000010000000000000005  000000010000000000000006  archive_status

    [postgres@postgres pg9.6]$ ls $PGDATA/pg_xlog/archive_status
    000000010000000000000003.00000028.backup.done  000000010000000000000004.done

Recovery
    월래 사용하던 $PGDATA /data/pg9.6 대신에 새로운 디렉터리를 생성 후 데이터 복구 테스트

    [root@postgres ~]# mkdir /recovery
    [root@postgres ~]# chown -R postgres:postgres /recovery

    [postgres@postgres pg96]$ cp -a /backup/pg96/pg9.6/ /recovery
    [postgres@postgres pg96]$ ls /recovery
    pg9.6

    postmaster.pid 삭제

    [postgres@postgres pg9.6]$ cd /recovery/pg9.6
    [postgres@postgres pg9.6]$ cat postmaster.pid
    1473
    /data/pg9.6
    1650611620
    5496
    /tmp
    *
    5496001         0
    [postgres@postgres pg9.6]$ rm -rf postmaster.pid

    불필요한 기존 로그 삭제

    [postgres@postgres pg9.6]$ rm -rf pg_log/*

    recovery.conf 파일 생성

    [postgres@postgres pg9.6]$ vi recovery.conf
    restore_command = 'cp /archive/pg96/%f %p'

    [postgres@postgres pg9.6]$ export PGDATA=/recovery/pg9.6
    [postgres@postgres pg9.6]$ echo $PGDATA
    /recovery/pg9.6

    PG 기동

    [postgres@postgres pg9.6]$ pg_ctl start -D $PGDATA -w
    waiting for server to start....2022-04-22 17:11:19 KST [1647]: [1-1] user=,db=,remote=LOG:  redirecting log output to logging collector process
    2022-04-22 17:11:19 KST [1647]: [2-1] user=,db=,remote=HINT:  Future log output will appear in directory "pg_log".
    . done
    server started

    Recovery가 완료 되면 recovery.conf 파일이 recovery.done으로 변경 됨

    [postgres@postgres pg9.6]$ ls $PGDATA/recovery*
    /recovery/pg9.6/recovery.done

    [postgres@postgres pg9.6]$ cat $PGDATA/postmaster.pid
    1647
    /recovery/pg9.6
    1650615079
    5496
    /tmp
    *
    5496001         1

    [postgres@postgres pg9.6]$ ps -u postgres uf
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    postgres  1592  0.0  0.0 115652  2188 pts/1    S    16:59   0:00 -bash
    postgres  1716  0.0  0.0 155448  1840 pts/1    R+   17:32   0:00  \_ ps -u postgres uf
    postgres  1424  0.0  0.0 115648  2156 pts/0    S+   16:07   0:00 -bash
    postgres  1647  0.0  0.1 275460 15448 pts/1    S    17:11   0:00 /engine/pg9.6/bin/postgres -D /recovery/pg9.6
    postgres  1648  0.0  0.0 126108   708 ?        Ss   17:11   0:00  \_ postgres: logger process
    postgres  1651  0.0  0.0 275580  2868 ?        Ss   17:11   0:00  \_ postgres: checkpointer process
    postgres  1652  0.0  0.0 275460  2128 ?        Ss   17:11   0:00  \_ postgres: writer process
    postgres  1659  0.0  0.0 275460  5028 ?        Ss   17:11   0:00  \_ postgres: wal writer process
    postgres  1660  0.0  0.0 275864  1984 ?        Ss   17:11   0:00  \_ postgres: autovacuum launcher process
    postgres  1661  0.0  0.0 128228   904 ?        Ss   17:11   0:00  \_ postgres: archiver process   last was 00000002.history
    postgres  1662  0.0  0.0 128228  1012 ?        Ss   17:11   0:00  \_ postgres: stats collector process

    데이터 확인

    [local]:5496 postgres@postgres # \l
                                    List of databases
    +-----------+----------+----------+-------------+-------------+-----------------------+
    |   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    | postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       |
    | template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    | template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    | test      | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    (4 rows)

    [local]:5496 postgres@postgres # \c test
    You are now connected to database "test" as user "postgres".

    [local]:5496 postgres@test # \d
                    List of relations
    +--------+-------------+----------+----------+
    | Schema |    Name     |   Type   |  Owner   |
    +--------+-------------+----------+----------+
    | public | test        | table    | postgres |
    | public | test_id_seq | sequence | postgres |
    +--------+-------------+----------+----------+
    (2 rows)

    [local]:5496 postgres@test # select * from test ;
    +----+----------------------------+
    | id |        insert_date         |
    +----+----------------------------+
    |  1 | 2022-04-22 16:45:40.541072 |
    |  2 | 2022-04-22 16:45:44.366287 |
    |  3 | 2022-04-22 16:45:45.156317 |
    |  4 | 2022-04-22 16:45:46.165845 |
    |  5 | 2022-04-22 16:45:46.999159 |
    +----+----------------------------+
    (5 rows)

기존 DataDirectory로 복구 파일 cpoy

    PG 종료
    [postgres@postgres pg9.6]$ pg_ctl -D $PGDATA stop -mf -w

    기존 datadirectory안의 파일 삭제

    [postgres@postgres pg9.6]$ rm -rf /data/pg9.6/*

    복구한 파일 복사

    [postgres@postgres pg9.6]$ cp -r /recovery/pg9.6/* /data/pg9.6/

    PG 기동

    [postgres@postgres pg9.6]$ export PGDATA=/data/pg9.6
    [postgres@postgres pg9.6]$ pg_ctl -D $PGDATA logfile start



#######
12.10
#######

아카이브 설정

    echo "
    wal_level = replica
    archive_mode = on
    archive_command = 'dd conv=fdatasync bs=256k if=%p of=/archive/pg12/temp/%f && mv -f /archive/pg12/temp/%f /archive/pg12' " >> $PGDATA/postgres.conf

PG 기동

archive가 정상적으로 동작 되는지 테스트

    [local]:5412 postgres@postgres # select pg_current_wal_lsn();
    +--------------------+
    | pg_current_wal_lsn |
    +--------------------+
    | 0/1686B50          |
    +--------------------+
    (1 row)

    [postgres@postgres pg12]$ ls $PGDATA/pg_wal
    000000010000000000000001  archive_status

    [local]:5412 postgres@postgres # select pg_switch_wal();
    +---------------+
    | pg_switch_wal |
    +---------------+
    | 0/1688528     |
    +---------------+
    (1 row)

    [postgres@postgres pg12]$ ls $PGDATA/pg_wal
    000000010000000000000001  000000010000000000000002  archive_status

    [postgres@postgres pg12]$ ls $PGDATA/pg_wal/archive_status
    000000010000000000000001.done

    [postgres@postgres pg12]$ ls /archive/pg12
    000000010000000000000001  temp

Hot backup 시작

    [local]:5412 postgres@postgres # select pg_start_backup('hotbackup',true);
    +-----------------+
    | pg_start_backup |
    +-----------------+
    | 0/3000028       |
    +-----------------+
    (1 row)

    [postgres@postgres pg12]$ cat $PGDATA/backup_label
    START WAL LOCATION: 0/3000028 (file 000000010000000000000003)
    CHECKPOINT LOCATION: 0/3000060
    BACKUP METHOD: pg_start_backup
    BACKUP FROM: master
    START TIME: 2022-04-22 17:53:13 KST
    LABEL: hotbackup
    START TIMELINE: 1

File Copy

    [postgres@postgres pg12]$ cp -a $PGDATA /backup/pg12/

    [postgres@postgres pg12]$ ls /backup/pg12
    pg12.10

    [postgres@postgres pg12]$ ls /backup/pg12/pg12.10/
    backup_label  current_logfiles  logfile       pg_dynshmem  pg_ident.conf  pg_logical    pg_notify    pg_serial     pg_stat      pg_subtrans  pg_twophase  pg_wal   postgresql.auto.conf  postmaster.opts
    base          global            pg_commit_ts  pg_hba.conf  pg_log         pg_multixact  pg_replslot  pg_snapshots  pg_stat_tmp  pg_tblspc    PG_VERSION   pg_xact  postgresql.conf       postmaster.pid

End backup

    [local]:5412 postgres@postgres # select pg_stop_backup();
    NOTICE:  all required WAL segments have been archived
    +----------------+
    | pg_stop_backup |
    +----------------+
    | 0/3000138      |
    +----------------+
    (1 row)


    [postgres@postgres pg12]$ cat $PGDATA/backup_label
    cat: /data/pg12.10/backup_label: No such file or directory


Test Data 추가

    [local]:5412 postgres@postgres # \l
                                    List of databases
    +-----------+----------+----------+-------------+-------------+-----------------------+
    |   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    | postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       |
    | template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    | template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    (3 rows)

    [local]:5412 postgres@postgres # create database test ;
    CREATE DATABASE

    [local]:5412 postgres@postgres # \c test
    You are now connected to database "test" as user "postgres".
    [local]:5412 postgres@test # create table test ( id serial , insert_date timestamp without time zone ) ;
    CREATE TABLE

    [local]:5412 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1

    [local]:5412 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1

    [local]:5412 postgres@test # insert into test (insert_date) values (current_timestamp) ;
    INSERT 0 1

    [local]:5412 postgres@test # select * from test ;
    +----+----------------------------+
    | id |        insert_date         |
    +----+----------------------------+
    |  1 | 2022-04-22 17:56:43.653391 |
    |  2 | 2022-04-22 17:56:44.822594 |
    |  3 | 2022-04-22 17:56:45.835276 |
    +----+----------------------------+
    (3 rows)


장애 발생
    장애가 발생한 것으로 가정하기 위하여 PG Down

    archive가 어디까지 적용 되었는지 확인
    [postgres@postgres pg12]$ ls /archive/pg12
    000000010000000000000001  000000010000000000000002  000000010000000000000003  000000010000000000000003.00000028.backup  000000010000000000000004  temp

    wal로그 파일 확인
    [postgres@postgres pg12]$ ls $PGDATA/pg_wal
    000000010000000000000003.00000028.backup  000000010000000000000005  000000010000000000000006  000000010000000000000007  archive_status

    [postgres@postgres pg12]$ ls $PGDATA/pg_wal/archive_status
    000000010000000000000003.00000028.backup.done

Recovery
    월래 사용하던 $PGDATA /data/pg12.10 대신에 새로운 디렉터리를 생성 후 데이터 복구 테스트

    [root@postgres ~]# mkdir /recovery
    [root@postgres ~]# chown -R postgres:postgres /recovery

    [postgres@postgres pg12]$ cp -a /backup/pg12/pg12.10/ /recovery
    [postgres@postgres pg12]$ ls /recovery/
    pg12.10

    postmaster.pid 삭제

    [postgres@postgres pg12]$ cd /recovery/pg12.10
    [postgres@postgres pg12.10]$ cat postmaster.pid
    1802
    /data/pg12.10
    1650617120
    5412
    /tmp
    *
    5412001         3
    ready
    [postgres@postgres pg12.10]$ rm -rf postmaster.pid

    불필요한 기존 로그 삭제

    [postgres@postgres pg12.10]$ rm -rf pg_log/*

    기존버전과 다르게 restore_command는 postgresql.conf 파일에 추가 하고, recovery.signal 파일을 생성해 준다.

    echo "
    restore_command = 'cp /archive/pg12/%f %p'" >> /recovery/pg12.10/postgresql.conf

    touch /recovery/pg12.10/recovery.signal

    PG 기동

    [postgres@postgres pg12.10]$ pg_ctl start -D $PGDATA -w
    waiting for server to start....2022-04-22 18:29:46 KST [1945]: [1-1] user=,db=,remote=LOG:  starting PostgreSQL 12.10 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
    2022-04-22 18:29:46 KST [1945]: [2-1] user=,db=,remote=LOG:  listening on IPv4 address "0.0.0.0", port 5412
    2022-04-22 18:29:46 KST [1945]: [3-1] user=,db=,remote=LOG:  listening on IPv6 address "::", port 5412
    2022-04-22 18:29:46 KST [1945]: [4-1] user=,db=,remote=LOG:  listening on Unix socket "/tmp/.s.PGSQL.5412"
    2022-04-22 18:29:46 KST [1945]: [5-1] user=,db=,remote=LOG:  redirecting log output to logging collector process
    2022-04-22 18:29:46 KST [1945]: [6-1] user=,db=,remote=HINT:  Future log output will appear in directory "pg_log".
    . done
    server started

    Recovery가 완료 되면 recovery.signal 파일 없어진다.

    [postgres@postgres pg12.10]$ ls /recovery/pg12.10/recovery.signal
    ls: cannot access /recovery/pg12.10/recovery.signal: No such file or directory

    [postgres@postgres pg12.10]$ cat postmaster.pid
    1945
    /recovery/pg12.10
    1650619786
    5412
    /tmp
    *
    5412001         6
    ready

    [postgres@postgres pg12.10]$ ps -u postgres uf
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    postgres  1813  0.0  0.0 115664  2164 pts/1    S    17:45   0:00 -bash
    postgres  1968  0.0  0.0 155448  1836 pts/1    R+   18:32   0:00  \_ ps -u postgres uf
    postgres  1777  0.0  0.0 115660  2164 pts/0    S+   17:45   0:00 -bash
    postgres  1945  0.0  0.6 402944 51164 ?        Ss   18:29   0:00 /engine/pg12.10/bin/postgres -D /recovery/pg12.10
    postgres  1946  0.0  0.0 129260   908 ?        Ss   18:29   0:00  \_ postgres: logger
    postgres  1950  0.0  0.0 403088  2944 ?        Ss   18:29   0:00  \_ postgres: checkpointer
    postgres  1951  0.0  0.0 403104  2160 ?        Ss   18:29   0:00  \_ postgres: background writer
    postgres  1953  0.0  0.0 131380  1008 ?        Ss   18:29   0:00  \_ postgres: stats collector
    postgres  1958  0.0  0.0 402944  5060 ?        Ss   18:29   0:00  \_ postgres: walwriter
    postgres  1959  0.0  0.0 403532  2064 ?        Ss   18:29   0:00  \_ postgres: autovacuum launcher
    postgres  1960  0.0  0.0 131380   988 ?        Ss   18:29   0:00  \_ postgres: archiver   last was 00000002.history
    postgres  1961  0.0  0.0 403528  1628 ?        Ss   18:29   0:00  \_ postgres: logical replication launcher

    데이터 확인

    [local]:5412 postgres@postgres # \l
                                    List of databases
    +-----------+----------+----------+-------------+-------------+-----------------------+
    |   Name    |  Owner   | Encoding |   Collate   |    Ctype    |   Access privileges   |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    | postgres  | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       |
    | template0 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    | template1 | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 | =c/postgres          +|
    |           |          |          |             |             | postgres=CTc/postgres |
    | test      | postgres | UTF8     | en_US.UTF-8 | en_US.UTF-8 |                       |
    +-----------+----------+----------+-------------+-------------+-----------------------+
    (4 rows)

    [local]:5412 postgres@postgres # \c test
    You are now connected to database "test" as user "postgres".

    [local]:5412 postgres@test # select * from test ;
    +----+----------------------------+
    | id |        insert_date         |
    +----+----------------------------+
    |  1 | 2022-04-22 17:56:43.653391 |
    |  2 | 2022-04-22 17:56:44.822594 |
    |  3 | 2022-04-22 17:56:45.835276 |
    +----+----------------------------+
    (3 rows)

기존 DataDirectory로 복구 파일 cpoy

    PG 종료
    [postgres@postgres pg12.10]$ pg_ctl -D $PGDATA stop -mf -w

    기존 datadirectory안의 파일 삭제

    [postgres@postgres pg12.10]$ rm -rf /data/pg12.10/*

    복구한 파일 복사

    [postgres@postgres pg12.10]$ cp -r /recovery/pg12.10/* /data/pg12.10/

    PG 기동

    [postgres@postgres pg12.10]$ export PGDATA=/data/pg12.10/
    [postgres@postgres pg12.10]$ pg_ctl -D $PGDATA logfile start

pgbench

★깜장별★ — Wed, 20 Apr 2022 16:43:57 +0900

PostgreSQL설치 시 기본으로 제공 되는 벤치 마크 툴.

source를 컴파일 해서 설치 하면 기본으로 설치 됨

초기 데이터 생성

테스트 데이터 베이스를 생성해 준 후 -i 옵션을 이용하여 테스트 데이터 셋 만듦

-i : 초기 데이터 생성

-s : 초기 데이터 사이즈가 작으므로 데이터 사이즈를 더 크게 하기 위한 옵션

-s 10은 기본 데이터 사이즈의 10배를 만들라는 뜻.

즉 기본 데이터가 100,000 row 이었으면, -s 10은 10 * 100,000 = 1,000,000 rows

pgbench -h localhost -p 5493 -U postgres -i -s 5 test

부하 테스트 방법

-c : DB에 접속하는 가상의 client의 수를 설정

-j : -c에서 설정한 client를 몇 개의 thread에 걸쳐 동작시킬 것인지 설정. (j는 CPU thread수에 맞춰서 수행 하면 된다.)

-c 는 항상 -j 값보다 크거나 같아야 한다. 그렇지 않으면 -j값은 무시되고 -c 값으로 대체 된다.

-c는 -j의 배수이어야 한다.

-t : transaction의 수

pgbench -h localhost -p 5493 -c 8 -j 4 -t 10000 test

테스트

1. 테스트 데이터 베이스 생성

echo "create database test;" | psql -p 5493 -d postgres

2. 테스트 데이터 생성

pgbench -h localhost -p 5493 -U postgres -i -s 5 test

psql -p 5493 -d postgres

postgres=# \l+

List of databases

-----------+----------+----------+-------------+-------------+-----------------------+---------+------------+--------------------------------------------

(4 rows)

postgres=# \c test

test=# \dt+

List of relations

--------+------------------+-------+----------+---------+-------------

(4 rows)

test=# select schemaname , relname , seq_scan, idx_scan , n_live_tup from pg_stat_user_tables ;

schemaname | relname | seq_scan | idx_scan | n_live_tup

------------+------------------+----------+----------+------------

public | pgbench_branches | 1 | 0 | 5

public | pgbench_history | 0 | | 0

public | pgbench_tellers | 1 | 0 | 50

public | pgbench_accounts | 1 | 0 | 500000

(4 rows)

3. 성능 테스트

pgbench -h localhost -p 5493 -c 8 -j 4 -t 10000 test

starting vacuum...end.

transaction type: TPC-B (sort of)

scaling factor: 5

query mode: simple

number of clients: 8

number of threads: 4

number of transactions per client: 10000

number of transactions actually processed: 80000/80000

tps = 1512.535602 (including connections establishing)

tps = 1512.750661 (excluding connections establishing)

Logical Replication

★깜장별★ — Tue, 19 Apr 2022 18:29:48 +0900

Logical Replication은 V10부터 가능하며, 기존의 Replication의 한계를 해결할 목적으로 도입 됨

기존 Replication의 한계

- 일부 데이터 베이스 또는 테이블만 복제 할 수 없다.

- Standby 서버에는 Write 작업이 불 가능하다.

- 서로 다른 버전의 master로부터 복제를 할 수 없다.

- 서로 다른 플랫폼 간의 복제를 할 수 없다. (Linux <--> Window)

동작 방식

변경 사항을 synchronization worker가 변경 사항을 감지하여, 구독자 에게 전달하는 방식을 사용.

실제는 구독 서버에서 Master의 변경 사항을 가져가는 방식이다.

Master서버는 wal_level = logical 를 설정하고, Replication 유저를 생성해 준다.

Master서버에서 publication 설정을 해 주고, 구독 서버에서는 Master에서 생성해 준 replication 유저를 이용하여 변경 데이터를 가져가서 반영을 한다.

PUBLICATION 설정 방법

해당 데이터베이스 안의 모든 테이블

CREATE PUBLICATION my_publication FOR ALL TABLES;

특정 테이블만 전달하고자 한다면

CREATE PUBLICATION my_publication FOR TABLE emp , dept ;

INSERT Operation만 전달하고자 한다면

CREATE PUBLICATION my_publication FOR TABLE sales_log WITH (publish = 'insert') ;

V10 : insert, update, delete

V11 ~ : insert, update, delete, truncate

SUBSCRIPTION 설정 방법

CREATE SUBSCRIPTION my_subscription CONNECTION 'host=MasterIP user=MasterUser password=pwd port=MasterPort dbname=MasterDB' PUBLICATION my_publication ;

리플리케이션 슬롯은 master서버에 생성 된다.

Logical Replication의 한계

schema나 DDL은 복제 하지 않는다. (standby에서 동일한 테이블 또는 스키마를 생성해 줘야 한다. pg_dump --schema-only)

DML만 지원. truncate는 지원하지 않음(V11부터는 지원)

시퀀스는 복제 하지 않는다.

복제 대상 테이블은 PK나 UNIQUE Key가 있어야 한다.

Large Objects는 복제를 지원하지 않는다.

테스트는 V10과 V11만 테스트를 진행해 본다.

V11은 Truncate 지원여부를 확인하기 위하여 진행 함.

테스트 (10.20)

Master(publication) : 192.168.56.116

Replication(subscription) : 192.168.56.117

Master(publication)

-- wal 데이터에 logical replication에 필요한 정보를 저장하기 위한 레벨 설정

echo "wal_level = logical" >> $PGDATA/postgresql.conf

-- 외부에서 접속하기 위한 설정

echo "host all all 0.0.0.0/0 md5" >> $PGDATA/pg_hba.conf

-- 테스트 데이터 베이스 생성

CREATE DATABASE test ;

-- Replication 유저 생성

CREATE USER repl WITH REPLICATION PASSWORD 'repl1122' ;

GRANT ALL PRIVILEGES ON DATABASE test TO repl;

-- 테스트 테이블 생성

\c test postgres

CREATE TABLE tb1 ( id int primary key , name varchar(10) ) ;

-- 테스트 데이터베이스 안의 public 테이블인 tb1에 대한 권한 부여

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO repl;

-- publication 설정

CREATE PUBLICATION pub_tb1 FOR TABLE tb1;

CREATE PUBLICATION

Replication(subscription)

-- 테스트 데이터 베이스 생성

CREATE DATABASE test ;

-- 테스트 테이블 생성

\c test postgres

CREATE TABLE tb1 ( id int primary key, name varchar(10) ) ;

-- subscription 설정 (master의 publication 이름과 Replication 유저 정보 입력)

CREATE SUBSCRIPTION sub_tb1 CONNECTION 'host=192.168.56.116 user=repl password=repl1122 port=5410 dbname=test' PUBLICATION pub_tb1;

NOTICE: created replication slot "sub_tb1" on publisher

CREATE SUBSCRIPTION

Master(publication)

-- INSERT 테스트

\c test postgres

INSERT INTO tb1 VALUES ( 1 , 'name1' ) ;

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

(1 row)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

(1 row)

Master(publication)

-- UPDATE 테스트

\c test postgres

UPDATE tb1 SET name = 'name2' ;

SELECT * FROM tb1 ;

id | name

----+-------

1 | name2

(1 row)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name2

(1 row)

Master(publication)

-- DELETE 테스트

\c test postgres

DELETE FROM tb1 ;

SELECT * FROM tb1 ;

id | name

----+------

(0 rows)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+------

(0 rows)

Master(publication)

-- Truncate 테스트를 위한 데이터 입력

\c test postgres

INSERT INTO tb1 VALUES ( 1 , 'name1' ) ;

INSERT INTO tb1 VALUES ( 2 , 'name2' ) ;

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

2 | name2

(2 rows)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

2 | name2

(2 rows)

Master(publication)

-- Truncate 작업 진행

\c test postgres

TRUNCATE TABLE tb1 ;

SELECT * FROM tb1 ;

id | name

----+------

(0 rows)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

2 | name2

(2 rows)

-- 데이터 삭제 되지 않은 상태 임

Master(publication)

-- 구독서버에 있는 동일 데이터를 Master에 Insert 해봄

\c test postgres

INSERT INTO tb1 VALUES ( 1 , 'name1' ) ;

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

(1 row)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

2 | name2

(2 rows)

-- 기존 데이터 때문에 에러 발생 (계속해서 발생 됨. 로그 파일 사이즈는 계속 커짐...)

Error 로그에는 기존에 있는 데이터라고 에러 메시지가 계속 출력 됨.

2022-04-19 17:58:17 KST [1960]: [1-1] user=,db=,remote=LOG: logical replication apply worker for subscription "sub_tb1" has started

2022-04-19 17:58:17 KST [1960]: [2-1] user=,db=,remote=ERROR: duplicate key value violates unique constraint "tb1_pkey"

2022-04-19 17:58:17 KST [1960]: [3-1] user=,db=,remote=DETAIL: Key (id)=(1) already exists.

2022-04-19 17:58:17 KST [1847]: [42-1] user=,db=,remote=LOG: worker process: logical replication worker for subscription 16390 (PID 1960) exited with exit code 1

2022-04-19 17:58:23 KST [1961]: [1-1] user=,db=,remote=LOG: logical replication apply worker for subscription "sub_tb1" has started

2022-04-19 17:58:23 KST [1961]: [2-1] user=,db=,remote=ERROR: duplicate key value violates unique constraint "tb1_pkey"

2022-04-19 17:58:23 KST [1961]: [3-1] user=,db=,remote=DETAIL: Key (id)=(1) already exists.

2022-04-19 17:58:23 KST [1847]: [43-1] user=,db=,remote=LOG: worker process: logical replication worker for subscription 16390 (PID 1961) exited with exit code 1

Master(publication)

-- 혹시 몰라서 Master에서 다른 명령어 수행해 봄

\c test postgres

DELETE FROM tb1 ;

SELECT * FROM tb1 ;

id | name

----+------

(0 rows)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

2 | name2

(2 rows)

-- 기존 에러로 인하여 에러 이후에 수행되는 Master의 변경 사항이 반영 되지 않음

2022-04-19 18:02:55 KST [2039]: [1-1] user=,db=,remote=LOG: logical replication apply worker for subscription "sub_tb1" has started

2022-04-19 18:02:55 KST [2039]: [2-1] user=,db=,remote=ERROR: duplicate key value violates unique constraint "tb1_pkey"

2022-04-19 18:02:55 KST [2039]: [3-1] user=,db=,remote=DETAIL: Key (id)=(1) already exists.

2022-04-19 18:02:55 KST [1847]: [97-1] user=,db=,remote=LOG: worker process: logical replication worker for subscription 16390 (PID 2039) exited with exit code 1

2022-04-19 18:03:00 KST [2040]: [1-1] user=,db=,remote=LOG: logical replication apply worker for subscription "sub_tb1" has started

2022-04-19 18:03:00 KST [2040]: [2-1] user=,db=,remote=ERROR: duplicate key value violates unique constraint "tb1_pkey"

2022-04-19 18:03:00 KST [2040]: [3-1] user=,db=,remote=DETAIL: Key (id)=(1) already exists.

2022-04-19 18:03:00 KST [1847]: [98-1] user=,db=,remote=LOG: worker process: logical replication worker for subscription 16390 (PID 2040) exited with exit code 1

-- 조치를 위하여 구독 서버에서 subscription 삭제를 해 줌

DROP SUBSCRIPTION sub_tb1 ;

-- 이제 에러 메시지 더 이상 발생 안 함.

TRUNCATE TABLE tb1 ;

테스트 (11.15)

Truncate 명령어가 적용 되는지 테스트 진행

Master(publication) : 192.168.56.116

Replication(subscription) : 192.168.56.117

Master(publication)

-- wal 데이터에 logical replication에 필요한 정보를 저장하기 위한 레벨 설정

echo "wal_level = logical" >> $PGDATA/postgresql.conf

-- 외부에서 접속하기 위한 설정

echo "host all all 0.0.0.0/0 md5" >> $PGDATA/pg_hba.conf

-- 테스트 데이터 베이스 생성

CREATE DATABASE test ;

-- Replication 유저 생성

CREATE USER repl WITH REPLICATION PASSWORD 'repl1122' ;

GRANT ALL PRIVILEGES ON DATABASE test TO repl;

-- 테스트 테이블 생성

\c test postgres

CREATE TABLE tb1 ( id int primary key , name varchar(10) ) ;

-- 테스트 데이터베이스 안의 public 테이블인 tb1에 대한 권한 부여

GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO repl;

-- publication 설정

CREATE PUBLICATION pub_tb1 FOR TABLE tb1;

CREATE PUBLICATION

Replication(subscription)

-- 테스트 데이터 베이스 생성

CREATE DATABASE test ;

-- 테스트 테이블 생성

\c test postgres

CREATE TABLE tb1 ( id int primary key, name varchar(10) ) ;

-- subscription 설정 (master의 publication 이름과 Replication 유저 정보 입력)

CREATE SUBSCRIPTION sub_tb1 CONNECTION 'host=192.168.56.116 user=repl password=repl1122 port=5411 dbname=test' PUBLICATION pub_tb1;

NOTICE: created replication slot "sub_tb1" on publisher

CREATE SUBSCRIPTION

Master(publication)

-- 테스트 데이터 입력

\c test postgres

INSERT INTO tb1 VALUES ( 1 , 'name1' ) ;

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

(1 row)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

(1 row)

-- 테스트 데이터 입력

\c test postgres

TRUNCATE TABLE tb1 ;

SELECT * FROM tb1 ;

id | name

----+------

(0 rows)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+------

(0 rows)

2022-04-19 18:11:04 KST [2130]: [2-1] user=,db=,remote=LOG: logical replication table synchronization worker for subscription "sub_tb1", table "tb1" has finished

-- 테스트 데이터 입력

\c test postgres

INSERT INTO tb1 VALUES ( 1 , 'name1' ) ;

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

(1 row)

Replication(subscription)

-- 데이터 확인

\c test postgres

SELECT * FROM tb1 ;

id | name

----+-------

1 | name1

(1 row)

PostgreSQL Replication rntjd

★깜장별★ — Tue, 19 Apr 2022 13:25:12 +0900

기존에 만들었던 PG VM을 Copy해서 Replication을 구성 함

###############
2가지 구성 종류
###############

log-shipping (physical replication - archive log file 전달)
    pg_xlog 디렉터리 안에 wal 파일 자체를 전달해서 사용하는 방식 (Oracle의 아카이브 파일을 전송 후 반영하는 것과 같은 ??)
    wal file이 생성되기 까지 master와 standby의 데이터 갭이 발생 할 수 있다.

streaming replication (logical replication - SQL command 전달)
    9.0부터 가능
    Master Server의 wal 내용을 standby로 전송하여 반영하는 방식
    wal sender, wal receiver 프로세스가 이 작업을 수행
    하나 이상의 standby 서버 가능

    master
        replication user 생성
            create user repl with replication password 'repl1122' ;
        replication user 접근 설정
            pg_hba.conf
            host     replication     repl            0.0.0.0/0                 md5
            또는
            host     replication     repl            192.168.56.117/32                 md5
        parameter 설정
            postgresql.conf
                wal_level = 버전에 맞게 설정 (9.5까지는 hot_standby , 9.6 부터는 replica)
                max_wal_senders = 3  (넉넉하게)
                wal_keep_segments = 100 (기본이 16M 이므로 ,약 1.6G- 100개 파일)  or max_replication_slots = 2
                -- replication slot을 사용할 경우 slot을 생성해 줘야 한다.
                -- select * from pg_create_physical_replication_slot('slot1') ;
                -- select * from pg_replication_slots ;

    standby
        DB stop
        rm -rf $PGDATA
        pg_basebackup -h 192.168.56.116 -p 5493 -D /data/pg9.3 -U repl -P -v -X stream

        vi $PGDATA/recovery.conf
            standby_mode='on'
            primary_conninfo='host=192.168.56.116 port=5493 user=repl password=repl1122'
            trigger_file='/data/pg9.3/failover_trigger'
            -- slot을 사용할 경우 아래 주석 제거
            #primary_slot_name='slot1'

######################
Replication 구성 서버
######################

master
    192.168.56.116
standby
    192.168.56.117

##############
관련 파라미터들
##############

master
    wal_level : wal 에 저장되는 데이터의 수준을 지정.
                9.3 :  minimal, archive, or hot_standby
                9.4 :  minimal, archive, hot_standby, or logical
                9.5 :  minimal, archive, hot_standby, or logical
                9.6 :  minimal, replica, or logical
                10.x :  minimal, replica, or logical
                11.x :  minimal, replica, or logical
                12.x :  minimal, replica, or logical
                13.x :  minimal, replica, or logical
                14.x :  minimal, replica, or logical
    max_wal_senders : wal 내용을 읽어서 전송할 프로세스 개수 설정. (0은 비활성화, 1이상으로 설정 시 streaming replication, 일반 유저 커넥션과 sender 커넥션을 포함하는 값이 max_connecitons)
                      standby 서버에서 연결이 잠시 끊어졌다가 다시 접속 될수도 있으므로, standby 개수보다 크게 준다.
    wal_keep_segments : pg_xlog 디렉터리에 저장해 놓을 과거 로그 파일의 개수 지정. standby에서 replication이 늦게 따라오고 있을 경우 아직 standby 서버에 적용되지 않은 wal 세그먼트가 삭제 될 수 있으니 잘 설정해 주어야 한다.
                        (하나의 세그먼트는 기본으로 16M , 기본값은 0)
                        9.4 부터는 자동으로 standby에서 적용 못 시킨 wal 파일을 보관할 수 있게 replication slot을 지원한다. 이럴 경우 standby 서버가 오래 끊어져 있을 경우 pg_xlog 가 계속 커 질 수 있다.
    wal_sender_timeout : 설정 값보다 오래된 비활성화 sender 세션을 종료 시킴 (ms). standby서버와 네트워크 이상이거나, standby 서버 장애를 감지 하는데 유용함. (기본값 60초)
    synchronous_standby_names : standby 서버 리스트를 컴마로 구별하여 지정해 준다. 커밋을 기다리는 트랜잭션은 리스트 된 standby 서버 중 첫번째 standby 서버가 데이터를 수신한 것을 확인한 후에 커밋을 진행 한다.
                                첫 번째 standby 서버가 연결이 끊어지면, 두 번째 standby 서버로 우선순위가 교체 된다.
                                이 값에 *을 설정하면 모든 standby 서버의 walreceiver의 이름을 뜻하게 된다.
                                이 값에 아무값도 없으면 synchronous 복제가 비 활성화 된다. synchronous 복제가 활성화 된 상태에서도 "synchronous_commit" 파라미터를 local 또는 off를 시켜서 사용할 수 있다.
                                여기에 설정해야 하는 이름은 standby 서버 walreceiver의 primary_conninfo값에 application_name으로 설정 한다.
    vacuum_defer_cleanup_age : vacuum 이나 hot update시 dead row를 정리 해야 하는데 이를 연기 시키는 트랜잭션 수를 지정 한다.(기본은 0)
                               이 값을 대신하는 것은 standby 서버에서 hot_standby_feedback 값을 설정 하는 것이다.

Standby
    hot_standby : standby 모드에서 커넥션을 받아서 쿼리를 수행가능하게 할지 여부를 결정(기본 OFF). server start시에만 변경 가능. archive recovery 또는 standby 모드에서만 적용 됨

#######
9.3
#######

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg9.3/bin/psql -p 5493 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=hot_standby
        max_wal_senders = 3
        wal_keep_segments = 100 " >> $PGDATA/postgresql.conf
    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf

    pg restart


standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5493 -D $PGDATA -U repl -P -v -X stream
    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf
    -- recovery.conf 파일 생성 ($PGDATA)
        echo "
        standby_mode='on'
        primary_conninfo='host=192.168.56.116 port=5494 user=repl password=repl1122'
        #primary_slot_name='repl_slot_01'
        trigger_file='/data/pg9.4/failover_trigger'" >> $PGDATA/recovery.conf

    stanby start...

master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)
    echo "select * from pg_stat_replication ;" | /engine/pg9.3/bin/psql -p 5493 -d postgres
    pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
    ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
    1591 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       39032 | 2022-04-08 12:56:36.515781+09 | streaming | 0/3000528     | 0/3000528      | 0/3000528      | 0/3000528       |             0 | async
    (1 row)


    -- process 확인 (wal sender)

    ps -u postgres uf
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    postgres  1326  0.0  0.0 115548  2144 pts/0    S    09:14   0:00 -bash
    postgres  1665  0.0  0.0 155448  1840 pts/0    R+   13:39   0:00  \_ ps -u postgres uf
    postgres  1571  0.0  0.3 265276 12664 pts/0    S    12:50   0:00 /engine/pg9.3/bin/postgres -D /data/pg9.3
    postgres  1573  0.0  0.0 265404  2216 ?        Ss   12:50   0:00  \_ postgres: checkpointer process
    postgres  1574  0.0  0.0 265276  1764 ?        Ss   12:50   0:00  \_ postgres: writer process
    postgres  1575  0.0  0.0 265276   752 ?        Ss   12:50   0:00  \_ postgres: wal writer process
    postgres  1576  0.0  0.0 266140  2092 ?        Ss   12:50   0:00  \_ postgres: autovacuum launcher process
    postgres  1577  0.0  0.0 120952   932 ?        Ss   12:50   0:00  \_ postgres: stats collector process
    postgres  1591  0.0  0.0 266088  2392 ?        Ss   12:56   0:00  \_ postgres: wal sender process repl 192.168.56.117(39032) streaming 0/3000528

standby
    -- process 확인 (wal receiver)

    ps -u postgres uf
    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    postgres  1327  0.0  0.0 115548  2160 pts/0    S    09:14   0:00 -bash
    postgres  1491  0.0  0.0 155448  1840 pts/0    R+   13:40   0:00  \_ ps -u postgres uf
    postgres  1457  0.0  0.3 265316 12656 pts/0    S    12:56   0:00 /engine/pg9.3/bin/postgres -D /data/pg9.3
    postgres  1458  0.0  0.0 265416  1544 ?        Ss   12:56   0:00  \_ postgres: startup process   recovering 000000010000000000000003
    postgres  1459  0.0  0.0 265416  2012 ?        Ss   12:56   0:00  \_ postgres: checkpointer process
    postgres  1460  0.0  0.0 265316  1756 ?        Ss   12:56   0:00  \_ postgres: writer process
    postgres  1461  0.0  0.0 120952   728 ?        Ss   12:56   0:00  \_ postgres: stats collector process
    postgres  1462  0.0  0.0 271884  1712 ?        Ss   12:56   0:02  \_ postgres: wal receiver process   streaming 0/3000528

    -- logfile 확인

    LOG:  database system was interrupted; last known up at 2022-04-08 12:50:26 KST
    LOG:  entering standby mode
    LOG:  redo starts at 0/2000028
    LOG:  consistent recovery state reached at 0/20000F0
    LOG:  database system is ready to accept read only connections
    LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg9.3/bin/psql -p 5493 -d postgres
        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
        1591 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       39032 | 2022-04-08 12:56:36.515781+09 | streaming | 0/3013F48     | 0/3013F48      | 0/3013F48      | 0/3013F48       |             0 | async
        (1 row)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1326  0.0  0.0 115548  2148 pts/0    S    09:14   0:00 -bash
        postgres  1713  0.0  0.0 155448  1836 pts/0    R+   14:00   0:00  \_ ps -u postgres uf
        postgres  1571  0.0  0.3 265276 12664 pts/0    S    12:50   0:00 /engine/pg9.3/bin/postgres -D /data/pg9.3
        postgres  1573  0.0  0.0 265404  2476 ?        Ss   12:50   0:00  \_ postgres: checkpointer process
        postgres  1574  0.0  0.0 265276  1764 ?        Ss   12:50   0:00  \_ postgres: writer process
        postgres  1575  0.0  0.0 265276   752 ?        Ss   12:50   0:00  \_ postgres: wal writer process
        postgres  1576  0.0  0.0 266140  2132 ?        Ss   12:50   0:00  \_ postgres: autovacuum launcher process
        postgres  1577  0.0  0.0 121084   968 ?        Ss   12:50   0:00  \_ postgres: stats collector process
        postgres  1591  0.0  0.0 266220  2392 ?        Ss   12:56   0:00  \_ postgres: wal sender process repl 192.168.56.117(39032) streaming 0/3013F48


    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1327  0.0  0.0 115548  2160 pts/0    S    09:14   0:00 -bash
        postgres  1512  0.0  0.0 155448  1840 pts/0    R+   14:00   0:00  \_ ps -u postgres uf
        postgres  1457  0.0  0.3 265316 12656 pts/0    S    12:56   0:00 /engine/pg9.3/bin/postgres -D /data/pg9.3
        postgres  1458  0.0  0.0 265492  2912 ?        Ss   12:56   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1459  0.0  0.0 265416  2264 ?        Ss   12:56   0:00  \_ postgres: checkpointer process
        postgres  1460  0.0  0.0 265316  1756 ?        Ss   12:56   0:00  \_ postgres: writer process
        postgres  1461  0.0  0.0 121084   728 ?        Ss   12:56   0:00  \_ postgres: stats collector process
        postgres  1462  0.0  0.0 272012  1984 ?        Ss   12:56   0:03  \_ postgres: wal receiver process   streaming 0/3013F48

        echo "select * from test;" | /engine/pg9.3/bin/psql -U app -p 5493 -d test

#######
9.4
#######

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg9.4/bin/psql -p 5494 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=hot_standby
        max_wal_senders = 3
        wal_keep_segments = 100 " >> $PGDATA/postgresql.conf
    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf
    pg restart



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5494 -D $PGDATA -U repl -P -v -X stream
    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf
    -- recovery.conf 파일 생성 ($PGDATA)
        echo "
        standby_mode='on'
        primary_conninfo='host=192.168.56.116 port=5494 user=repl password=repl1122'
        #primary_slot_name='repl_slot_01'
        trigger_file='/data/pg9.4/failover_trigger'" >> $PGDATA/recovery.conf

    stanby start...

master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg9.4/bin/psql -p 5494 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
        1466 |    16385 | repl    | walreceiver      | 192.168.56.117 |                 |       48862 | 2022-04-11 13:43:43.028452+09 |         1896 | streaming | 0/3000138     | 0/3000138      | 0/3000138      | 0/3000138       |             0 | async
        (1 row)



    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1450  0.0  0.3 270300 13276 pts/0    S    13:39   0:00 /engine/pg9.4/bin/postgres -D /data/pg9.4
        postgres  1451  0.0  0.0 123324   624 ?        Ss   13:39   0:00  \_ postgres: logger process
        postgres  1453  0.0  0.0 270416  2280 ?        Ss   13:39   0:00  \_ postgres: checkpointer process
        postgres  1454  0.0  0.0 270444  2236 ?        Ss   13:39   0:00  \_ postgres: writer process
        postgres  1455  0.0  0.1 270300  5132 ?        Ss   13:39   0:00  \_ postgres: wal writer process
        postgres  1456  0.0  0.0 270704  1840 ?        Ss   13:39   0:00  \_ postgres: autovacuum launcher process
        postgres  1457  0.0  0.0 125576  1008 ?        Ss   13:39   0:00  \_ postgres: stats collector process
        postgres  1466  0.0  0.0 270704  2064 ?        Ss   13:43   0:00  \_ postgres: wal sender process repl 192.168.56.117(48862) streaming 0/3000138


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1391  0.0  0.3 270332 13272 pts/0    S    13:43   0:00 /engine/pg9.4/bin/postgres -D /data/pg9.4
        postgres  1392  0.0  0.0 123324   624 ?        Ss   13:43   0:00  \_ postgres: logger process
        postgres  1393  0.0  0.0 270448  1580 ?        Ss   13:43   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1394  0.0  0.0 270332   920 ?        Ss   13:43   0:00  \_ postgres: checkpointer process
        postgres  1395  0.0  0.0 270332  1968 ?        Ss   13:43   0:00  \_ postgres: writer process
        postgres  1396  0.0  0.0 125444   656 ?        Ss   13:43   0:00  \_ postgres: stats collector process
        postgres  1397  0.0  0.0 274740  1844 ?        Ss   13:43   0:00  \_ postgres: wal receiver process   streaming 0/3000138


    -- logfile 확인

        2022-04-11 13:43:42 KST [1393]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 13:40:25 KST
        2022-04-11 13:43:43 KST [1393]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 13:43:43 KST [1393]: [3-1] user=,db=,remote=LOG:  redo starts at 0/2000060
        2022-04-11 13:43:43 KST [1393]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/2000128
        2022-04-11 13:43:43 KST [1391]: [3-1] user=,db=,remote=LOG:  database system is ready to accept read only connections
        2022-04-11 13:43:43 KST [1397]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg9.4/bin/psql -p 5494 -d postgres
        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
        1466 |    16385 | repl    | walreceiver      | 192.168.56.117 |                 |       48862 | 2022-04-11 13:43:43.028452+09 |         1902 | streaming | 0/3012590     | 0/3012590      | 0/3012590      | 0/3012590       |             0 | async
        (1 row)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1450  0.0  0.3 270300 13276 pts/0    S    13:39   0:00 /engine/pg9.4/bin/postgres -D /data/pg9.4
        postgres  1451  0.0  0.0 123324   624 ?        Ss   13:39   0:00  \_ postgres: logger process
        postgres  1453  0.0  0.0 270416  2528 ?        Ss   13:39   0:00  \_ postgres: checkpointer process
        postgres  1454  0.0  0.0 270444  2236 ?        Ss   13:39   0:00  \_ postgres: writer process
        postgres  1455  0.0  0.1 270300  5132 ?        Ss   13:39   0:00  \_ postgres: wal writer process
        postgres  1456  0.0  0.0 270704  1876 ?        Ss   13:39   0:00  \_ postgres: autovacuum launcher process
        postgres  1457  0.0  0.0 125576  1036 ?        Ss   13:39   0:00  \_ postgres: stats collector process
        postgres  1466  0.0  0.0 270836  2064 ?        Ss   13:43   0:00  \_ postgres: wal sender process repl 192.168.56.117(48862) streaming 0/3012590


    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1391  0.0  0.3 270332 13272 pts/0    S    13:43   0:00 /engine/pg9.4/bin/postgres -D /data/pg9.4
        postgres  1392  0.0  0.0 123324   624 ?        Ss   13:43   0:00  \_ postgres: logger process
        postgres  1393  0.0  0.0 270512  3348 ?        Ss   13:43   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1394  0.0  0.0 270448  2484 ?        Ss   13:43   0:00  \_ postgres: checkpointer process
        postgres  1395  0.0  0.0 270332  1968 ?        Ss   13:43   0:00  \_ postgres: writer process
        postgres  1396  0.0  0.0 125444   656 ?        Ss   13:43   0:00  \_ postgres: stats collector process
        postgres  1397  0.0  0.0 274868  2008 ?        Ss   13:43   0:00  \_ postgres: wal receiver process   streaming 0/3012590


        echo "select * from test;" | /engine/pg9.4/bin/psql -U app -p 5494 -d test

#######
9.5
#######

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg9.5/bin/psql -p 5495 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=hot_standby
        max_wal_senders = 3
        wal_keep_segments = 100 " >> $PGDATA/postgresql.conf
    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf
    pg restart



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5495 -D $PGDATA -U repl -P -v -X stream
    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf
    -- recovery.conf 파일 생성 ($PGDATA)
        echo "
        standby_mode='on'
        primary_conninfo='host=192.168.56.116 port=5495 user=repl password=repl1122'
        #primary_slot_name='repl_slot_01'
        trigger_file='/data/pg9.5/failover_trigger'" >> $PGDATA/recovery.conf

    stanby start...

master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg9.5/bin/psql -p 5495 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
        1677 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       52970 | 2022-04-11 14:07:45.938827+09 |         1835 | streaming | 0/3000060     | 0/3000060      | 0/3000060      | 0/3000060       |             0 | async
        (1 row)



    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1662  0.0  0.3 270864 13352 pts/0    S    14:04   0:00 /engine/pg9.5/bin/postgres -D /data/pg9.5
        postgres  1663  0.0  0.0 123744   680 ?        Ss   14:04   0:00  \_ postgres: logger process
        postgres  1665  0.0  0.0 270968  2524 ?        Ss   14:04   0:00  \_ postgres: checkpointer process
        postgres  1666  0.0  0.0 270864  2104 ?        Ss   14:04   0:00  \_ postgres: writer process
        postgres  1667  0.0  0.1 270864  5000 ?        Ss   14:04   0:00  \_ postgres: wal writer process
        postgres  1668  0.0  0.0 271300  1892 ?        Ss   14:04   0:00  \_ postgres: autovacuum launcher process
        postgres  1669  0.0  0.0 125996  1044 ?        Ss   14:04   0:00  \_ postgres: stats collector process
        postgres  1677  0.0  0.0 271172  2140 ?        Ss   14:07   0:00  \_ postgres: wal sender process repl 192.168.56.117(52970) streaming 0/3000060


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1535  0.1  0.3 270896 13340 pts/0    S    14:07   0:00 /engine/pg9.5/bin/postgres -D /data/pg9.5
        postgres  1536  0.0  0.0 123744   680 ?        Ss   14:07   0:00  \_ postgres: logger process
        postgres  1537  0.0  0.0 270968  1472 ?        Ss   14:07   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1538  0.0  0.0 270896   832 ?        Ss   14:07   0:00  \_ postgres: checkpointer process
        postgres  1539  0.0  0.0 270896  1076 ?        Ss   14:07   0:00  \_ postgres: writer process
        postgres  1540  0.0  0.0 125864   700 ?        Ss   14:07   0:00  \_ postgres: stats collector process
        postgres  1541  0.3  0.0 275172  1596 ?        Ss   14:07   0:00  \_ postgres: wal receiver process   streaming 0/3000060


    -- logfile 확인

        2022-04-11 14:07:45 KST [1537]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 14:05:56 KST
        2022-04-11 14:07:45 KST [1537]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 14:07:46 KST [1537]: [3-1] user=,db=,remote=LOG:  redo starts at 0/2000028
        2022-04-11 14:07:46 KST [1537]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/2000130
        2022-04-11 14:07:46 KST [1535]: [3-1] user=,db=,remote=LOG:  database system is ready to accept read only connections
        2022-04-11 14:07:46 KST [1541]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg9.5/bin/psql -p 5495 -d postgres
        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
        1677 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       52970 | 2022-04-11 14:07:45.938827+09 |         1841 | streaming | 0/30162F0     | 0/30162F0      | 0/30162F0      | 0/30162F0       |             0 | async
        (1 row)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1718  0.0  0.0 115548  2076 pts/0    S    14:11   0:00 -bash
        postgres  1749  0.0  0.0 155448  1840 pts/0    R+   14:11   0:00  \_ ps -u postgres uf
        postgres  1662  0.0  0.3 270864 13352 pts/0    S    14:04   0:00 /engine/pg9.5/bin/postgres -D /data/pg9.5
        postgres  1663  0.0  0.0 123744   680 ?        Ss   14:04   0:00  \_ postgres: logger process
        postgres  1665  0.0  0.0 270968  2524 ?        Ss   14:04   0:00  \_ postgres: checkpointer process
        postgres  1666  0.0  0.0 270864  2104 ?        Ss   14:04   0:00  \_ postgres: writer process
        postgres  1667  0.0  0.1 270864  5000 ?        Ss   14:04   0:00  \_ postgres: wal writer process
        postgres  1668  0.0  0.0 271300  1924 ?        Ss   14:04   0:00  \_ postgres: autovacuum launcher process
        postgres  1669  0.0  0.0 125996  1072 ?        Ss   14:04   0:00  \_ postgres: stats collector process
        postgres  1677  0.0  0.0 271480  2140 ?        Ss   14:07   0:00  \_ postgres: wal sender process repl 192.168.56.117(52970) streaming 0/30162F0


    standby

        [postgres@postgres ~]$ ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1572  0.0  0.0 115548  2080 pts/0    S    14:11   0:00 -bash
        postgres  1596  0.0  0.0 155448  1840 pts/0    R+   14:11   0:00  \_ ps -u postgres uf
        postgres  1535  0.0  0.3 270896 13340 pts/0    S    14:07   0:00 /engine/pg9.5/bin/postgres -D /data/pg9.5
        postgres  1536  0.0  0.0 123744   680 ?        Ss   14:07   0:00  \_ postgres: logger process
        postgres  1537  0.0  0.0 271004  3396 ?        Ss   14:07   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1538  0.0  0.0 270896   832 ?        Ss   14:07   0:00  \_ postgres: checkpointer process
        postgres  1539  0.0  0.0 270896  1868 ?        Ss   14:07   0:00  \_ postgres: writer process
        postgres  1540  0.0  0.0 125864   700 ?        Ss   14:07   0:00  \_ postgres: stats collector process
        postgres  1541  0.0  0.0 275352  1992 ?        Ss   14:07   0:00  \_ postgres: wal receiver process   streaming 0/30162F0


        echo "select * from test;" | /engine/pg9.5/bin/psql -U app -p 5495 -d test


#######
9.6
#######

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg9.6/bin/psql -p 5496 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=replica
        max_wal_senders = 3
        wal_keep_segments = 100 " >> $PGDATA/postgresql.conf
    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf
    pg restart



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5496 -D $PGDATA -U repl -P -v -X stream
    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf
    -- recovery.conf 파일 생성 ($PGDATA)
        echo "
        standby_mode='on'
        primary_conninfo='host=192.168.56.116 port=5495 user=repl password=repl1122'
        #primary_slot_name='repl_slot_01'
        trigger_file='/data/pg9.5/failover_trigger'" >> $PGDATA/recovery.conf

    stanby start...

master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg9.6/bin/psql -p 5496 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
        1812 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       59404 | 2022-04-11 15:17:43.739304+09 |         1762 | streaming | 0/3000060     | 0/3000060      | 0/3000060      | 0/3000060       |             0 | async
        (1 row)



    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1718  0.0  0.0 115548  2104 pts/0    S    14:11   0:00 -bash
        postgres  1817  0.0  0.0 155448  1840 pts/0    R+   15:18   0:00  \_ ps -u postgres uf
        postgres  1799  0.0  0.3 271128 13000 pts/0    S    15:16   0:00 /engine/pg9.6/bin/postgres -D /data/pg9.6
        postgres  1800  0.0  0.0 124024   696 ?        Ss   15:16   0:00  \_ postgres: logger process
        postgres  1802  0.0  0.0 271128  2380 ?        Ss   15:16   0:00  \_ postgres: checkpointer process
        postgres  1803  0.0  0.0 271128  2124 ?        Ss   15:16   0:00  \_ postgres: writer process
        postgres  1804  0.1  0.1 271128  5284 ?        Ss   15:16   0:00  \_ postgres: wal writer process
        postgres  1805  0.0  0.0 271556  1964 ?        Ss   15:16   0:00  \_ postgres: autovacuum launcher process
        postgres  1806  0.0  0.0 126276  1052 ?        Ss   15:16   0:00  \_ postgres: stats collector process
        postgres  1812  0.0  0.0 271532  2224 ?        Ss   15:17   0:00  \_ postgres: wal sender process repl 192.168.56.117(59404) streaming 0/3000060


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1633  0.0  0.0 115548  2156 pts/0    S    15:13   0:00 -bash
        postgres  1674  0.0  0.0 155448  1840 pts/0    R+   15:18   0:00  \_ ps -u postgres uf
        postgres  1664  0.1  0.3 271160 12996 pts/0    S    15:17   0:00 /engine/pg9.6/bin/postgres -D /data/pg9.6
        postgres  1665  0.0  0.0 124024   692 ?        Ss   15:17   0:00  \_ postgres: logger process
        postgres  1666  0.0  0.0 271224  1504 ?        Ss   15:17   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1667  0.0  0.0 271160   844 ?        Ss   15:17   0:00  \_ postgres: checkpointer process
        postgres  1668  0.0  0.0 271160  1352 ?        Ss   15:17   0:00  \_ postgres: writer process
        postgres  1669  0.0  0.0 126144   916 ?        Ss   15:17   0:00  \_ postgres: stats collector process
        postgres  1670  0.3  0.0 275436  1628 ?        Ss   15:17   0:00  \_ postgres: wal receiver process   streaming 0/3000060


    -- logfile 확인

        2022-04-11 15:17:42 KST [1666]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 15:16:57 KST
        2022-04-11 15:17:43 KST [1666]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 15:17:43 KST [1666]: [3-1] user=,db=,remote=LOG:  redo starts at 0/2000060
        2022-04-11 15:17:43 KST [1666]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/2000130
        2022-04-11 15:17:43 KST [1664]: [3-1] user=,db=,remote=LOG:  database system is ready to accept read only connections
        2022-04-11 15:17:43 KST [1670]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg9.6/bin/psql -p 5496 -d postgres
        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_location | write_location | flush_location | replay_location | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+---------------+----------------+----------------+-----------------+---------------+------------
        1812 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       59404 | 2022-04-11 15:17:43.739304+09 |         1768 | streaming | 0/30174A0     | 0/30174A0      | 0/30174A0      | 0/30174A0       |             0 | async
        (1 row)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1718  0.0  0.0 115660  2156 pts/0    S    14:11   0:00 -bash
        postgres  1832  0.0  0.0 155448  1840 pts/0    R+   15:22   0:00  \_ ps -u postgres uf
        postgres  1799  0.0  0.3 271128 13000 pts/0    S    15:16   0:00 /engine/pg9.6/bin/postgres -D /data/pg9.6
        postgres  1800  0.0  0.0 124024   696 ?        Ss   15:16   0:00  \_ postgres: logger process
        postgres  1802  0.0  0.0 271268  2640 ?        Ss   15:16   0:00  \_ postgres: checkpointer process
        postgres  1803  0.0  0.0 271128  2124 ?        Ss   15:16   0:00  \_ postgres: writer process
        postgres  1804  0.0  0.1 271128  5284 ?        Ss   15:16   0:00  \_ postgres: wal writer process
        postgres  1805  0.0  0.0 271556  1992 ?        Ss   15:16   0:00  \_ postgres: autovacuum launcher process
        postgres  1806  0.0  0.0 126276  1080 ?        Ss   15:16   0:00  \_ postgres: stats collector process
        postgres  1812  0.0  0.0 271800  2484 ?        Ss   15:17   0:00  \_ postgres: wal sender process repl 192.168.56.117(59404) streaming 0/30174A0


    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1633  0.0  0.0 115548  2156 pts/0    S    15:13   0:00 -bash
        postgres  1680  0.0  0.0 155448  1840 pts/0    R+   15:22   0:00  \_ ps -u postgres uf
        postgres  1664  0.0  0.3 271160 12996 pts/0    S    15:17   0:00 /engine/pg9.6/bin/postgres -D /data/pg9.6
        postgres  1665  0.0  0.0 124024   692 ?        Ss   15:17   0:00  \_ postgres: logger process
        postgres  1666  0.0  0.0 271224  3328 ?        Ss   15:17   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1667  0.0  0.0 271160   844 ?        Ss   15:17   0:00  \_ postgres: checkpointer process
        postgres  1668  0.0  0.0 271160  1880 ?        Ss   15:17   0:00  \_ postgres: writer process
        postgres  1669  0.0  0.0 126144   916 ?        Ss   15:17   0:00  \_ postgres: stats collector process
        postgres  1670  0.1  0.0 275568  1872 ?        Ss   15:17   0:00  \_ postgres: wal receiver process   streaming 0/30174A0


        echo "select * from test;" | /engine/pg9.6/bin/psql -U app -p 5496 -d test

#######
10.20
#######

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg10.20/bin/psql -p 5410 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=replica
        max_wal_senders = 3
        wal_keep_segments = 100 " >> $PGDATA/postgresql.conf
    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf
    pg restart



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5410 -D $PGDATA -U repl -P -v -X stream
    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf
    -- recovery.conf 파일 생성 ($PGDATA)
        echo "
        standby_mode='on'
        primary_conninfo='host=192.168.56.116 port=5410 user=repl password=repl1122'
        #primary_slot_name='repl_slot_01'
        trigger_file='/data/pg10.20/failover_trigger'" >> $PGDATA/recovery.conf

    stanby start...

master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg10.20/bin/psql -p 5410 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------
        1909 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       40628 | 2022-04-11 15:30:25.750393+09 |          557 | streaming | 0/3000060 | 0/3000060 | 0/3000060 | 0/3000060  |           |           |            |             0 | async
        (1 row)



    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115548  2140 pts/0    S    15:26   0:00 -bash
        postgres  1914  0.0  0.0 155448  1840 pts/0    R+   15:31   0:00  \_ ps -u postgres uf
        postgres  1896  0.0  0.3 271884 13132 pts/0    S    15:28   0:00 /engine/pg10.20/bin/postgres -D /data/pg10.20
        postgres  1897  0.0  0.0 124704   888 ?        Ss   15:28   0:00  \_ postgres: logger process
        postgres  1899  0.0  0.0 271884  2652 ?        Ss   15:28   0:00  \_ postgres: checkpointer process
        postgres  1900  0.0  0.0 271884  2136 ?        Ss   15:28   0:00  \_ postgres: writer process
        postgres  1901  0.1  0.1 271884  5292 ?        Ss   15:28   0:00  \_ postgres: wal writer process
        postgres  1902  0.0  0.0 272324  1988 ?        Ss   15:28   0:00  \_ postgres: autovacuum launcher process
        postgres  1903  0.0  0.0 126956  1092 ?        Ss   15:28   0:00  \_ postgres: stats collector process
        postgres  1904  0.0  0.0 272176  1564 ?        Ss   15:28   0:00  \_ postgres: bgworker: logical replication launcher
        postgres  1909  0.0  0.0 272288  2516 ?        Ss   15:30   0:00  \_ postgres: wal sender process repl 192.168.56.117(40628) streaming 0/3000060


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115548  2160 pts/0    S    15:26   0:00 -bash
        postgres  1732  0.0  0.0 155448  1840 pts/0    R+   15:31   0:00  \_ ps -u postgres uf
        postgres  1722  0.0  0.3 271884 13124 pts/0    S    15:30   0:00 /engine/pg10.20/bin/postgres -D /data/pg10.20
        postgres  1723  0.0  0.0 124704   888 ?        Ss   15:30   0:00  \_ postgres: logger process
        postgres  1724  0.0  0.0 271980  1612 ?        Ss   15:30   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1725  0.0  0.0 271884  1100 ?        Ss   15:30   0:00  \_ postgres: checkpointer process
        postgres  1726  0.0  0.0 271884  1636 ?        Ss   15:30   0:00  \_ postgres: writer process
        postgres  1727  0.0  0.0 126824   928 ?        Ss   15:30   0:00  \_ postgres: stats collector process
        postgres  1728  0.2  0.0 276272  1968 ?        Ss   15:30   0:00  \_ postgres: wal receiver process   streaming 0/3000060


    -- logfile 확인

        2022-04-11 15:30:24 KST [1724]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 15:29:28 KST
        2022-04-11 15:30:25 KST [1724]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 15:30:25 KST [1724]: [3-1] user=,db=,remote=LOG:  redo starts at 0/2000060
        2022-04-11 15:30:25 KST [1724]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/2000130
        2022-04-11 15:30:25 KST [1722]: [6-1] user=,db=,remote=LOG:  database system is ready to accept read only connections
        2022-04-11 15:30:25 KST [1728]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg10.20/bin/psql -p 5410 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------
        1909 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       40628 | 2022-04-11 15:30:25.750393+09 |          563 | streaming | 0/3018570 | 0/3018570 | 0/3018570 | 0/3018570  |           |           |            |             0 | async
        (1 row)


        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115548  2148 pts/0    S    15:26   0:00 -bash
        postgres  1928  0.0  0.0 155448  1840 pts/0    R+   15:34   0:00  \_ ps -u postgres uf
        postgres  1896  0.0  0.3 271884 13132 pts/0    S    15:28   0:00 /engine/pg10.20/bin/postgres -D /data/pg10.20
        postgres  1897  0.0  0.0 124704   888 ?        Ss   15:28   0:00  \_ postgres: logger process
        postgres  1899  0.0  0.0 272024  2652 ?        Ss   15:28   0:00  \_ postgres: checkpointer process
        postgres  1900  0.0  0.0 271884  2400 ?        Ss   15:28   0:00  \_ postgres: writer process
        postgres  1901  0.0  0.1 271884  5292 ?        Ss   15:28   0:00  \_ postgres: wal writer process
        postgres  1902  0.0  0.0 272324  2028 ?        Ss   15:28   0:00  \_ postgres: autovacuum launcher process
        postgres  1903  0.0  0.0 126956  1120 ?        Ss   15:28   0:00  \_ postgres: stats collector process
        postgres  1904  0.0  0.0 272176  1564 ?        Ss   15:28   0:00  \_ postgres: bgworker: logical replication launcher
        postgres  1909  0.0  0.0 272568  2516 ?        Ss   15:30   0:00  \_ postgres: wal sender process repl 192.168.56.117(40628) streaming 0/3018570


    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115548  2160 pts/0    S    15:26   0:00 -bash
        postgres  1738  0.0  0.0 155448  1844 pts/0    R+   15:34   0:00  \_ ps -u postgres uf
        postgres  1722  0.0  0.3 271884 13124 pts/0    S    15:30   0:00 /engine/pg10.20/bin/postgres -D /data/pg10.20
        postgres  1723  0.0  0.0 124704   888 ?        Ss   15:30   0:00  \_ postgres: logger process
        postgres  1724  0.0  0.0 271980  3696 ?        Ss   15:30   0:00  \_ postgres: startup process   recovering 000000010000000000000003
        postgres  1725  0.0  0.0 271884  1100 ?        Ss   15:30   0:00  \_ postgres: checkpointer process
        postgres  1726  0.0  0.0 271884  2164 ?        Ss   15:30   0:00  \_ postgres: writer process
        postgres  1727  0.0  0.0 126824   928 ?        Ss   15:30   0:00  \_ postgres: stats collector process
        postgres  1728  0.1  0.0 276404  2204 ?        Ss   15:30   0:00  \_ postgres: wal receiver process   streaming 0/3018570


        echo "select * from test;" | /engine/pg10.20/bin/psql -U app -p 5410 -d test

#######
11.15
#######

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg11.15/bin/psql -p 5411 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=replica
        max_wal_senders = 3
        wal_keep_segments = 100 " >> $PGDATA/postgresql.conf
    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf
    pg restart



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5411 -D $PGDATA -U repl -P -v -X stream
    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf
    -- recovery.conf 파일 생성 ($PGDATA)
        echo "
        standby_mode='on'
        primary_conninfo='host=192.168.56.116 port=5411 user=repl password=repl1122'
        #primary_slot_name='repl_slot_01'
        trigger_file='/data/pg11.15/failover_trigger'" >> $PGDATA/recovery.conf

    stanby start...

master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg11.15/bin/psql -p 5411 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------
        1975 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       53564 | 2022-04-11 15:42:04.211714+09 |          570 | streaming | 0/3000060 | 0/3000060 | 0/3000060 | 0/3000060  |           |           |            |             0 | async
        (1 row)



    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115548  2148 pts/0    S    15:26   0:00 -bash
        postgres  1980  0.0  0.0 155448  1844 pts/0    R+   15:42   0:00  \_ ps -u postgres uf
        postgres  1962  0.0  0.3 272196 13148 pts/0    S    15:40   0:00 /engine/pg11.15/bin/postgres -D /data/pg11.15
        postgres  1963  0.0  0.0 125008   880 ?        Ss   15:40   0:00  \_ postgres: logger
        postgres  1965  0.0  0.0 272308  2384 ?        Ss   15:40   0:00  \_ postgres: checkpointer
        postgres  1966  0.0  0.0 272196  2140 ?        Ss   15:40   0:00  \_ postgres: background writer
        postgres  1967  0.0  0.1 272196  5040 ?        Ss   15:40   0:00  \_ postgres: walwriter
        postgres  1968  0.0  0.0 272608  2048 ?        Ss   15:40   0:00  \_ postgres: autovacuum launcher
        postgres  1969  0.0  0.0 127272  1092 ?        Ss   15:40   0:00  \_ postgres: stats collector
        postgres  1970  0.0  0.0 272608  1588 ?        Ss   15:40   0:00  \_ postgres: logical replication launcher
        postgres  1975  0.0  0.0 272776  2548 ?        Ss   15:42   0:00  \_ postgres: walsender repl 192.168.56.117(53564) streaming 0/3000060


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2176 pts/0    S    15:26   0:00 -bash
        postgres  1770  0.0  0.0 155448  1840 pts/0    R+   15:42   0:00  \_ ps -u postgres uf
        postgres  1761  0.0  0.3 272196 13148 pts/0    S    15:42   0:00 /engine/pg11.15/bin/postgres -D /data/pg11.15
        postgres  1762  0.0  0.0 125008   880 ?        Ss   15:42   0:00  \_ postgres: logger
        postgres  1763  0.0  0.0 272356  1672 ?        Ss   15:42   0:00  \_ postgres: startup   recovering 000000010000000000000003
        postgres  1764  0.0  0.0 272196   848 ?        Ss   15:42   0:00  \_ postgres: checkpointer
        postgres  1765  0.0  0.0 272196  1372 ?        Ss   15:42   0:00  \_ postgres: background writer
        postgres  1766  0.0  0.0 127128   920 ?        Ss   15:42   0:00  \_ postgres: stats collector
        postgres  1767  0.2  0.0 276584  1852 ?        Rs   15:42   0:00  \_ postgres: walreceiver   streaming 0/3000060


    -- logfile 확인

        2022-04-11 15:42:03 KST [1763]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 15:41:07 KST
        2022-04-11 15:42:04 KST [1763]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 15:42:04 KST [1763]: [3-1] user=,db=,remote=LOG:  redo starts at 0/2000028
        2022-04-11 15:42:04 KST [1763]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/2000130
        2022-04-11 15:42:04 KST [1761]: [6-1] user=,db=,remote=LOG:  database system is ready to accept read only connections
        2022-04-11 15:42:04 KST [1767]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg11.15/bin/psql -p 5411 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn |    write_lag    |    flush_lag    |   replay_lag    | sync_priority | sync_state
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------------+-----------------+-----------------+---------------+------------
        1975 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       53564 | 2022-04-11 15:42:04.211714+09 |          576 | streaming | 0/3019140 | 0/3019140 | 0/3019140 | 0/3019140  | 00:00:00.000799 | 00:00:00.001503 | 00:00:00.001746 |             0 | async
        (1 row)


        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115664  2168 pts/0    S    15:26   0:00 -bash
        postgres  1990  0.0  0.0 155448  1836 pts/0    R+   15:45   0:00  \_ ps -u postgres uf
        postgres  1962  0.0  0.3 272196 13148 pts/0    S    15:40   0:00 /engine/pg11.15/bin/postgres -D /data/pg11.15
        postgres  1963  0.0  0.0 125008   880 ?        Ss   15:40   0:00  \_ postgres: logger
        postgres  1965  0.0  0.0 272308  2640 ?        Ss   15:40   0:00  \_ postgres: checkpointer
        postgres  1966  0.0  0.0 272196  2140 ?        Ss   15:40   0:00  \_ postgres: background writer
        postgres  1967  0.0  0.1 272196  5040 ?        Ss   15:40   0:00  \_ postgres: walwriter
        postgres  1968  0.0  0.0 272608  2048 ?        Ss   15:40   0:00  \_ postgres: autovacuum launcher
        postgres  1969  0.0  0.0 127272  1092 ?        Ss   15:40   0:00  \_ postgres: stats collector
        postgres  1970  0.0  0.0 272608  1588 ?        Ss   15:40   0:00  \_ postgres: logical replication launcher
        postgres  1975  0.0  0.0 272908  2548 ?        Ss   15:42   0:00  \_ postgres: walsender repl 192.168.56.117(53564) streaming 0/3019140


    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2176 pts/0    S    15:26   0:00 -bash
        postgres  1774  0.0  0.0 155448  1840 pts/0    R+   15:45   0:00  \_ ps -u postgres uf
        postgres  1761  0.0  0.3 272196 13148 pts/0    S    15:42   0:00 /engine/pg11.15/bin/postgres -D /data/pg11.15
        postgres  1762  0.0  0.0 125008   880 ?        Ss   15:42   0:00  \_ postgres: logger
        postgres  1763  0.0  0.0 272404  3560 ?        Ss   15:42   0:00  \_ postgres: startup   recovering 000000010000000000000003
        postgres  1764  0.0  0.0 272196   848 ?        Ss   15:42   0:00  \_ postgres: checkpointer
        postgres  1765  0.0  0.0 272196  1900 ?        Ss   15:42   0:00  \_ postgres: background writer
        postgres  1766  0.0  0.0 127128   920 ?        Ss   15:42   0:00  \_ postgres: stats collector
        postgres  1767  0.1  0.0 276788  2184 ?        Ss   15:42   0:00  \_ postgres: walreceiver   streaming 0/3019140


        echo "select * from test;" | /engine/pg11.15/bin/psql -U app -p 5411 -d test

#######
12.10
#######

standby 서버에서 recovery.conf 파일이 제거 되고, standby.signal 파일을 생성해 주어야 한다.
master서버에 대한 정보는 $PGDATA/postgresql.auto.conf 파일에 적어 넣으면 된다.

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg12.10/bin/psql -p 5412 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=replica
        max_wal_senders = 3
        wal_keep_segments = 100 " >> $PGDATA/postgresql.conf
    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf

    pg restart or echo "select pg_reload_conf(); " | /engine/pg12.10/bin/psql -p 5412 -d postgres



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5412 -D $PGDATA -U repl -P -v -X stream

    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf
    -- postgresql.auto.conf 수정 ($PGDATA)
        echo "
        primary_conninfo='host=192.168.56.116 port=5412 user=repl password=repl1122' " >> $PGDATA/postgresql.auto.conf
    -- standby.signal 파일 생성
        touch $PGDATA/standby.signal

    stanby start...

    -- master info 조회

        echo "
        SELECT name, setting, category, short_desc, context, pending_restart
        FROM pg_catalog.pg_settings
        WHERE category IN('Write-Ahead Log / Archive Recovery','Write-Ahead Log / Recovery Target')
        OR name IN ('primary_conninfo','primary_slot_name','promote_trigger_file','recovery_min_apply_delay')
        ORDER BY category, name; " | /engine/pg12.10/bin/psql -p 5412 -d postgres

                   name            |                          setting                          |              category              |                                   short_desc                                    |  context   | pending_restart
        ---------------------------+-----------------------------------------------------------+------------------------------------+---------------------------------------------------------------------------------+------------+-----------------
         primary_conninfo          | host=192.168.56.116 port=5412 user=repl password=repl1122 | Replication / Standby Servers      | Sets the connection string to be used to connect to the sending server.         | postmaster | f
         primary_slot_name         |                                                           | Replication / Standby Servers      | Sets the name of the replication slot to use on the sending server.             | postmaster | f
         promote_trigger_file      |                                                           | Replication / Standby Servers      | Specifies a file name whose presence ends recovery in the standby.              | sighup     | f
         recovery_min_apply_delay  | 0                                                         | Replication / Standby Servers      | Sets the minimum delay for applying changes during recovery.                    | sighup     | f
         archive_cleanup_command   |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be executed at every restart point.            | sighup     | f
         recovery_end_command      |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be executed once at the end of recovery.       | sighup     | f
         restore_command           |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will retrieve an archived WAL file.                 | postmaster | f
         recovery_target           |                                                           | Write-Ahead Log / Recovery Target  | Set to "immediate" to end recovery as soon as a consistent state is reached.    | postmaster | f
         recovery_target_action    | pause                                                     | Write-Ahead Log / Recovery Target  | Sets the action to perform upon reaching the recovery target.                   | postmaster | f
         recovery_target_inclusive | on                                                        | Write-Ahead Log / Recovery Target  | Sets whether to include or exclude transaction with recovery target.            | postmaster | f
         recovery_target_lsn       |                                                           | Write-Ahead Log / Recovery Target  | Sets the LSN of the write-ahead log location up to which recovery will proceed. | postmaster | f
         recovery_target_name      |                                                           | Write-Ahead Log / Recovery Target  | Sets the named restore point up to which recovery will proceed.                 | postmaster | f
         recovery_target_time      |                                                           | Write-Ahead Log / Recovery Target  | Sets the time stamp up to which recovery will proceed.                          | postmaster | f
         recovery_target_timeline  | latest                                                    | Write-Ahead Log / Recovery Target  | Specifies the timeline to recover into.                                         | postmaster | f
         recovery_target_xid       |                                                           | Write-Ahead Log / Recovery Target  | Sets the transaction ID up to which recovery will proceed.                      | postmaster | f
        (15 rows)

master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg12.10/bin/psql -p 5412 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state |          reply_time
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------+-------------------------------
        2058 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       40562 | 2022-04-11 16:04:30.913879+09 |          488 | streaming | 0/3000148 | 0/3000148 | 0/3000148 | 0/3000148  |           |           |            |             0 | async      | 2022-04-11 16:08:31.454985+09
        (1 row)


    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115664  2168 pts/0    S    15:26   0:00 -bash
        postgres  2071  0.0  0.0 155448  1840 pts/0    R+   16:08   0:00  \_ ps -u postgres uf
        postgres  2018  0.0  0.3 272436 13336 ?        Ss   15:50   0:00 /engine/pg12.10/bin/postgres -D /data/pg12.10
        postgres  2019  0.0  0.0 125024   880 ?        Ss   15:50   0:00  \_ postgres: logger
        postgres  2021  0.0  0.0 272548  2384 ?        Ss   15:50   0:00  \_ postgres: checkpointer
        postgres  2022  0.0  0.0 272436  2116 ?        Ss   15:50   0:00  \_ postgres: background writer
        postgres  2023  0.0  0.1 272436  5016 ?        Ss   15:50   0:00  \_ postgres: walwriter
        postgres  2024  0.0  0.0 272980  2080 ?        Ss   15:50   0:00  \_ postgres: autovacuum launcher
        postgres  2025  0.0  0.0 127288  1084 ?        Ss   15:50   0:00  \_ postgres: stats collector
        postgres  2026  0.0  0.0 272984  1576 ?        Ss   15:50   0:00  \_ postgres: logical replication launcher
        postgres  2058  0.0  0.0 273148  2580 ?        Ss   16:04   0:00  \_ postgres: walsender repl 192.168.56.117(40562) streaming 0/3000148


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2176 pts/0    S    15:26   0:00 -bash
        postgres  1847  0.0  0.0 155448  1840 pts/0    R+   16:08   0:00  \_ ps -u postgres uf
        postgres  1831  0.0  0.3 272436 13332 ?        Ss   16:04   0:00 /engine/pg12.10/bin/postgres -D /data/pg12.10
        postgres  1832  0.0  0.0 125024   880 ?        Ss   16:04   0:00  \_ postgres: logger
        postgres  1833  0.0  0.0 272596  1644 ?        Ss   16:04   0:00  \_ postgres: startup   recovering 000000010000000000000003
        postgres  1834  0.0  0.0 272436   836 ?        Ss   16:04   0:00  \_ postgres: checkpointer
        postgres  1835  0.0  0.0 272436  1884 ?        Ss   16:04   0:00  \_ postgres: background writer
        postgres  1836  0.0  0.0 127144   940 ?        Ss   16:04   0:00  \_ postgres: stats collector
        postgres  1837  0.1  0.0 276836  1872 ?        Ss   16:04   0:00  \_ postgres: walreceiver   streaming 0/3000148


    -- logfile 확인

        2022-04-11 16:04:30 KST [1833]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 15:51:21 KST
        2022-04-11 16:04:30 KST [1833]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 16:04:30 KST [1833]: [3-1] user=,db=,remote=LOG:  redo starts at 0/2000060
        2022-04-11 16:04:30 KST [1833]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/2000138
        2022-04-11 16:04:30 KST [1831]: [7-1] user=,db=,remote=LOG:  database system is ready to accept read only connections
        2022-04-11 16:04:30 KST [1837]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg12.10/bin/psql -p 5412 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state |          reply_time
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------+-------------------------------
        2058 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       40562 | 2022-04-11 16:04:30.913879+09 |          494 | streaming | 0/3014FF0 | 0/3014FF0 | 0/3014FF0 | 0/3014FF0  |           |           |            |             0 | async      | 2022-04-11 16:11:58.716019+09
        (1 row)


        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115664  2176 pts/0    S    15:26   0:00 -bash
        postgres  2088  0.0  0.0 155448  1844 pts/0    R+   16:12   0:00  \_ ps -u postgres uf
        postgres  2018  0.0  0.3 272436 13336 ?        Ss   15:50   0:00 /engine/pg12.10/bin/postgres -D /data/pg12.10
        postgres  2019  0.0  0.0 125024   880 ?        Ss   15:50   0:00  \_ postgres: logger
        postgres  2021  0.0  0.0 272548  2636 ?        Ss   15:50   0:00  \_ postgres: checkpointer
        postgres  2022  0.0  0.0 272436  2116 ?        Ss   15:50   0:00  \_ postgres: background writer
        postgres  2023  0.0  0.1 272436  5016 ?        Ss   15:50   0:00  \_ postgres: walwriter
        postgres  2024  0.0  0.0 273112  2112 ?        Ss   15:50   0:00  \_ postgres: autovacuum launcher
        postgres  2025  0.0  0.0 127288  1112 ?        Ss   15:50   0:00  \_ postgres: stats collector
        postgres  2026  0.0  0.0 272984  1576 ?        Ss   15:50   0:00  \_ postgres: logical replication launcher
        postgres  2058  0.0  0.0 273280  2580 ?        Ss   16:04   0:00  \_ postgres: walsender repl 192.168.56.117(40562) streaming 0/3014FF0


    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2176 pts/0    S    15:26   0:00 -bash
        postgres  1852  0.0  0.0 155448  1840 pts/0    R+   16:12   0:00  \_ ps -u postgres uf
        postgres  1831  0.0  0.3 272436 13332 ?        Ss   16:04   0:00 /engine/pg12.10/bin/postgres -D /data/pg12.10
        postgres  1832  0.0  0.0 125024   880 ?        Ss   16:04   0:00  \_ postgres: logger
        postgres  1833  0.0  0.0 272644  3512 ?        Ss   16:04   0:00  \_ postgres: startup   recovering 000000010000000000000003
        postgres  1834  0.0  0.0 272436  2136 ?        Ss   16:04   0:00  \_ postgres: checkpointer
        postgres  1835  0.0  0.0 272436  1884 ?        Ss   16:04   0:00  \_ postgres: background writer
        postgres  1836  0.0  0.0 127144   940 ?        Ss   16:04   0:00  \_ postgres: stats collector
        postgres  1837  0.1  0.0 277040  2188 ?        Ss   16:04   0:00  \_ postgres: walreceiver   streaming 0/3014FF0

        echo "select * from test;" | /engine/pg12.10/bin/psql -U app -p 5412 -d test

#######
13.6
#######

master 서버에서 wal_keep_segments는 사용 안하게 됨.

standby 서버에서 recovery.conf 파일이 제거 되고, standby.signal 파일을 생성해 주어야 한다.
master서버에 대한 정보는 $PGDATA/postgresql.auto.conf 파일에 적어 넣으면 된다.

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg13.6/bin/psql -p 5413 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=replica
        max_wal_senders = 3
        max_replication_slots = 2 " >> $PGDATA/postgresql.conf

    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf

    pg restart
    or
    echo "select pg_reload_conf(); " | /engine/pg13.6/bin/psql -p 5413 -d postgres

    slot 생성
        echo "select * from pg_create_physical_replication_slot('slot1') ; select * from pg_replication_slots ;"  | /engine/pg13.6/bin/psql -p 5413 -d postgres



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5413 -D $PGDATA -U repl -P -v -X stream

    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf

    -- postgresql.auto.conf 수정 ($PGDATA)
        echo "
        primary_conninfo='host=192.168.56.116 port=5413 user=repl password=repl1122'
        primary_slot_name='slot1'" >> $PGDATA/postgresql.auto.conf

    -- standby.signal 파일 생성
        touch $PGDATA/standby.signal

    stanby start...

    -- master info 조회

        echo "
        SELECT name, setting, category, short_desc, context, pending_restart
        FROM pg_catalog.pg_settings
        WHERE category IN('Write-Ahead Log / Archive Recovery','Write-Ahead Log / Recovery Target')
        OR name IN ('primary_conninfo','primary_slot_name','promote_trigger_file','recovery_min_apply_delay')
        ORDER BY category, name; " | /engine/pg13.6/bin/psql -p 5413 -d postgres

                   name            |                          setting                          |              category              |                                   short_desc                                    |  context   | pending_restart
        ---------------------------+-----------------------------------------------------------+------------------------------------+---------------------------------------------------------------------------------+------------+-----------------
         primary_conninfo          | host=192.168.56.116 port=5413 user=repl password=repl1122 | Replication / Standby Servers      | Sets the connection string to be used to connect to the sending server.         | sighup     | f
         primary_slot_name         | slot1                                                     | Replication / Standby Servers      | Sets the name of the replication slot to use on the sending server.             | sighup     | f
         promote_trigger_file      |                                                           | Replication / Standby Servers      | Specifies a file name whose presence ends recovery in the standby.              | sighup     | f
         recovery_min_apply_delay  | 0                                                         | Replication / Standby Servers      | Sets the minimum delay for applying changes during recovery.                    | sighup     | f
         archive_cleanup_command   |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be executed at every restart point.            | sighup     | f
         recovery_end_command      |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be executed once at the end of recovery.       | sighup     | f
         restore_command           |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be called to retrieve an archived WAL file.    | postmaster | f
         recovery_target           |                                                           | Write-Ahead Log / Recovery Target  | Set to "immediate" to end recovery as soon as a consistent state is reached.    | postmaster | f
         recovery_target_action    | pause                                                     | Write-Ahead Log / Recovery Target  | Sets the action to perform upon reaching the recovery target.                   | postmaster | f
         recovery_target_inclusive | on                                                        | Write-Ahead Log / Recovery Target  | Sets whether to include or exclude transaction with recovery target.            | postmaster | f
         recovery_target_lsn       |                                                           | Write-Ahead Log / Recovery Target  | Sets the LSN of the write-ahead log location up to which recovery will proceed. | postmaster | f
         recovery_target_name      |                                                           | Write-Ahead Log / Recovery Target  | Sets the named restore point up to which recovery will proceed.                 | postmaster | f
         recovery_target_time      |                                                           | Write-Ahead Log / Recovery Target  | Sets the time stamp up to which recovery will proceed.                          | postmaster | f
         recovery_target_timeline  | latest                                                    | Write-Ahead Log / Recovery Target  | Specifies the timeline to recover into.                                         | postmaster | f
         recovery_target_xid       |                                                           | Write-Ahead Log / Recovery Target  | Sets the transaction ID up to which recovery will proceed.                      | postmaster | f
        (15 rows)


master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg13.6/bin/psql -p 5413 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn |    write_lag    |    flush_lag    |   replay_lag    | sync_priority | sync_state |          reply_time
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------------+-----------------+-----------------+---------------+------------+-------------------------------
        2176 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       55360 | 2022-04-11 16:32:40.989917+09 |              | streaming | 0/5000148 | 0/5000148 | 0/5000148 | 0/5000148  | 00:00:00.000844 | 00:00:00.001715 | 00:00:00.001802 |             0 | async      | 2022-04-11 16:35:16.763246+09
        (1 row)


    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115664  2180 pts/0    S    15:26   0:00 -bash
        postgres  2182  0.0  0.0 155448  1844 pts/0    R+   16:35   0:00  \_ ps -u postgres uf
        postgres  2157  0.0  0.3 272464 13336 ?        Ss   16:29   0:00 /engine/pg13.6/bin/postgres -D /data/pg13.6
        postgres  2158  0.0  0.0 125052   908 ?        Ss   16:29   0:00  \_ postgres: logger
        postgres  2160  0.0  0.0 272576  2672 ?        Ss   16:29   0:00  \_ postgres: checkpointer
        postgres  2161  0.0  0.0 272464  2128 ?        Ss   16:29   0:00  \_ postgres: background writer
        postgres  2162  0.0  0.1 272464  5024 ?        Ss   16:29   0:00  \_ postgres: walwriter
        postgres  2163  0.0  0.0 273000  2124 ?        Ss   16:29   0:00  \_ postgres: autovacuum launcher
        postgres  2164  0.0  0.0 127316  1100 ?        Ss   16:29   0:00  \_ postgres: stats collector
        postgres  2165  0.0  0.0 272896  1584 ?        Ss   16:29   0:00  \_ postgres: logical replication launcher
        postgres  2176  0.0  0.0 273152  2620 ?        Ss   16:32   0:00  \_ postgres: walsender repl 192.168.56.117(55360) streaming 0/5000148


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2180 pts/0    S    15:26   0:00 -bash
        postgres  1912  0.0  0.0 155448  1840 pts/0    R+   16:35   0:00  \_ ps -u postgres uf
        postgres  1895  0.0  0.3 272464 13328 ?        Ss   16:32   0:00 /engine/pg13.6/bin/postgres -D /data/pg13.6
        postgres  1896  0.0  0.0 125052   900 ?        Ss   16:32   0:00  \_ postgres: logger
        postgres  1897  0.0  0.0 272624  1672 ?        Ss   16:32   0:00  \_ postgres: startup recovering 000000010000000000000005
        postgres  1898  0.0  0.0 272464  1092 ?        Ss   16:32   0:00  \_ postgres: checkpointer
        postgres  1899  0.0  0.0 272464  1888 ?        Ss   16:32   0:00  \_ postgres: background writer
        postgres  1900  0.0  0.0 127172   964 ?        Ss   16:32   0:00  \_ postgres: stats collector
        postgres  1901  0.1  0.0 276868  1888 ?        Ss   16:32   0:00  \_ postgres: walreceiver streaming 0/5000148


    -- logfile 확인

        2022-04-11 16:32:40 KST [1895]: [3-1] user=,db=,remote=LOG:  starting PostgreSQL 13.6 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
        2022-04-11 16:32:40 KST [1895]: [4-1] user=,db=,remote=LOG:  listening on IPv4 address "0.0.0.0", port 5413
        2022-04-11 16:32:40 KST [1895]: [5-1] user=,db=,remote=LOG:  listening on IPv6 address "::", port 5413
        2022-04-11 16:32:40 KST [1895]: [6-1] user=,db=,remote=LOG:  listening on Unix socket "/tmp/.s.PGSQL.5413"
        2022-04-11 16:32:40 KST [1897]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 16:30:16 KST
        2022-04-11 16:32:41 KST [1897]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 16:32:41 KST [1897]: [3-1] user=,db=,remote=LOG:  redo starts at 0/4000028
        2022-04-11 16:32:41 KST [1897]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/4000138
        2022-04-11 16:32:41 KST [1895]: [7-1] user=,db=,remote=LOG:  database system is ready to accept read only connections
        2022-04-11 16:32:41 KST [1901]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/5000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg13.6/bin/psql -p 5413 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn |    write_lag    |    flush_lag    |   replay_lag    | sync_priority | sync_state |          reply_time
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------------+-----------------+-----------------+---------------+------------+-------------------------------
        2176 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       55360 | 2022-04-11 16:32:40.989917+09 |              | streaming | 0/5013D08 | 0/5013D08 | 0/5013D08 | 0/5013D08  | 00:00:00.001912 | 00:00:00.004985 | 00:00:00.004992 |             0 | async      | 2022-04-11 16:37:52.923078+09
        (1 row)


        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115664  2180 pts/0    S    15:26   0:00 -bash
        postgres  2196  0.0  0.0 155448  1836 pts/0    R+   16:37   0:00  \_ ps -u postgres uf
        postgres  2157  0.0  0.3 272464 13336 ?        Ss   16:29   0:00 /engine/pg13.6/bin/postgres -D /data/pg13.6
        postgres  2158  0.0  0.0 125052   908 ?        Ss   16:29   0:00  \_ postgres: logger
        postgres  2160  0.0  0.0 272576  2672 ?        Ss   16:29   0:00  \_ postgres: checkpointer
        postgres  2161  0.0  0.0 272464  2128 ?        Ss   16:29   0:00  \_ postgres: background writer
        postgres  2162  0.0  0.1 272464  5024 ?        Ss   16:29   0:00  \_ postgres: walwriter
        postgres  2163  0.0  0.0 273132  2156 ?        Ss   16:29   0:00  \_ postgres: autovacuum launcher
        postgres  2164  0.0  0.0 127316  1128 ?        Ss   16:29   0:00  \_ postgres: stats collector
        postgres  2165  0.0  0.0 272896  1584 ?        Ss   16:29   0:00  \_ postgres: logical replication launcher
        postgres  2176  0.0  0.0 273468  2880 ?        Ss   16:32   0:00  \_ postgres: walsender repl 192.168.56.117(55360) streaming 0/5013D08

    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2180 pts/0    S    15:26   0:00 -bash
        postgres  1915  0.0  0.0 155448  1840 pts/0    R+   16:37   0:00  \_ ps -u postgres uf
        postgres  1895  0.0  0.3 272464 13328 ?        Ss   16:32   0:00 /engine/pg13.6/bin/postgres -D /data/pg13.6
        postgres  1896  0.0  0.0 125052   900 ?        Ss   16:32   0:00  \_ postgres: logger
        postgres  1897  0.0  0.0 272672  3524 ?        Ss   16:32   0:00  \_ postgres: startup recovering 000000010000000000000005
        postgres  1898  0.0  0.0 272464  2644 ?        Ss   16:32   0:00  \_ postgres: checkpointer
        postgres  1899  0.0  0.0 272464  1888 ?        Ss   16:32   0:00  \_ postgres: background writer
        postgres  1900  0.0  0.0 127172   964 ?        Ss   16:32   0:00  \_ postgres: stats collector
        postgres  1901  0.1  0.0 277076  2204 ?        Ss   16:32   0:00  \_ postgres: walreceiver streaming 0/5013D08

        echo "select * from test;" | /engine/pg13.6/bin/psql -U app -p 5413 -d test

#######
14.2
#######

master 서버에서 wal_keep_segments는 사용 안하게 됨.

standby 서버에서 recovery.conf 파일이 제거 되고, standby.signal 파일을 생성해 주어야 한다.
master서버에 대한 정보는 $PGDATA/postgresql.auto.conf 파일에 적어 넣으면 된다.

master
    replication user
        echo "create user repl with replication password 'repl1122' ; select * from pg_user ;" | /engine/pg14.2/bin/psql -p 5414 -d postgres
    postgresql.conf
        echo "
        # Replication setting
        wal_level=replica
        max_wal_senders = 3
        max_replication_slots = 2 " >> $PGDATA/postgresql.conf

    pg_hba.conf
        echo "

        # replication setting
        host     replication     repl            0.0.0.0/0                 md5" >> $PGDATA/pg_hba.conf

    pg restart
    or
    echo "select pg_reload_conf(); " | /engine/pg14.2/bin/psql -p 5414 -d postgres

    slot 생성
        echo "select * from pg_create_physical_replication_slot('slot1') ; select * from pg_replication_slots ;"  | /engine/pg14.2/bin/psql -p 5414 -d postgres



standby
    -- datadirectory 지움
        cd $PGDATA
        rm -rf *
    -- basebackup
        pg_basebackup -h 192.168.56.116 -p 5414 -D $PGDATA -U repl -P -v -X stream

    -- postgresql.conf 수정
        echo "

        ## Replication setting
        hot_standby = on
        hot_standby_feedback = on" >> $PGDATA/postgresql.conf

    -- postgresql.auto.conf 수정 ($PGDATA)
        echo "
        primary_conninfo='host=192.168.56.116 port=5414 user=repl password=repl1122'
        primary_slot_name='slot1'" >> $PGDATA/postgresql.auto.conf

    -- standby.signal 파일 생성
        touch $PGDATA/standby.signal

    stanby start...

    -- master info 조회

        echo "
        SELECT name, setting, category, short_desc, context, pending_restart
        FROM pg_catalog.pg_settings
        WHERE category IN('Write-Ahead Log / Archive Recovery','Write-Ahead Log / Recovery Target')
        OR name IN ('primary_conninfo','primary_slot_name','promote_trigger_file','recovery_min_apply_delay')
        ORDER BY category, name; " | /engine/pg14.2/bin/psql -p 5414 -d postgres

                   name            |                          setting                          |              category              |                                   short_desc                                    |  context   | pending_restart
        ---------------------------+-----------------------------------------------------------+------------------------------------+---------------------------------------------------------------------------------+------------+-----------------
         primary_conninfo          | host=192.168.56.116 port=5414 user=repl password=repl1122 | Replication / Standby Servers      | Sets the connection string to be used to connect to the sending server.         | sighup     | f
         primary_slot_name         | slot1                                                     | Replication / Standby Servers      | Sets the name of the replication slot to use on the sending server.             | sighup     | f
         promote_trigger_file      |                                                           | Replication / Standby Servers      | Specifies a file name whose presence ends recovery in the standby.              | sighup     | f
         recovery_min_apply_delay  | 0                                                         | Replication / Standby Servers      | Sets the minimum delay for applying changes during recovery.                    | sighup     | f
         archive_cleanup_command   |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be executed at every restart point.            | sighup     | f
         recovery_end_command      |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be executed once at the end of recovery.       | sighup     | f
         restore_command           |                                                           | Write-Ahead Log / Archive Recovery | Sets the shell command that will be called to retrieve an archived WAL file.    | sighup     | f
         recovery_target           |                                                           | Write-Ahead Log / Recovery Target  | Set to "immediate" to end recovery as soon as a consistent state is reached.    | postmaster | f
         recovery_target_action    | pause                                                     | Write-Ahead Log / Recovery Target  | Sets the action to perform upon reaching the recovery target.                   | postmaster | f
         recovery_target_inclusive | on                                                        | Write-Ahead Log / Recovery Target  | Sets whether to include or exclude transaction with recovery target.            | postmaster | f
         recovery_target_lsn       |                                                           | Write-Ahead Log / Recovery Target  | Sets the LSN of the write-ahead log location up to which recovery will proceed. | postmaster | f
         recovery_target_name      |                                                           | Write-Ahead Log / Recovery Target  | Sets the named restore point up to which recovery will proceed.                 | postmaster | f
         recovery_target_time      |                                                           | Write-Ahead Log / Recovery Target  | Sets the time stamp up to which recovery will proceed.                          | postmaster | f
         recovery_target_timeline  | latest                                                    | Write-Ahead Log / Recovery Target  | Specifies the timeline to recover into.                                         | postmaster | f
         recovery_target_xid       |                                                           | Write-Ahead Log / Recovery Target  | Sets the transaction ID up to which recovery will proceed.                      | postmaster | f
        (15 rows)


master
    -- 구성상태 확인 (location 값은 계속 변경되므로 쿼리나 ps를 수행하는 시점에 따라 동일하지 않을 수 있다.)

        echo "select * from pg_stat_replication ;" | /engine/pg14.2/bin/psql -p 5414 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state |          reply_time
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------+-------------------------------
        2227 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       43686 | 2022-04-11 16:44:34.124914+09 |              | streaming | 0/3000060 | 0/3000060 | 0/3000060 | 0/3000060  |           |           |            |             0 | async      | 2022-04-11 16:47:15.071001+09
        (1 row)

    -- process 확인 (wal sender)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115664  2180 pts/0    S    15:26   0:00 -bash
        postgres  2237  0.0  0.0 155448  1844 pts/0    R+   16:47   0:00  \_ ps -u postgres uf
        postgres  2208  0.0  0.3 272612 13108 ?        Ss   16:42   0:00 /engine/pg14.2/bin/postgres -D /data/pg14.2
        postgres  2209  0.0  0.0 125436   924 ?        Ss   16:42   0:00  \_ postgres: logger
        postgres  2211  0.0  0.0 272728  2844 ?        Ss   16:42   0:00  \_ postgres: checkpointer
        postgres  2212  0.0  0.0 272612  2132 ?        Ss   16:42   0:00  \_ postgres: background writer
        postgres  2213  0.0  0.1 272612  5288 ?        Ss   16:42   0:00  \_ postgres: walwriter
        postgres  2214  0.0  0.0 273160  2172 ?        Ss   16:42   0:00  \_ postgres: autovacuum launcher
        postgres  2215  0.0  0.0 127688  1136 ?        Ss   16:42   0:00  \_ postgres: stats collector
        postgres  2216  0.0  0.0 273048  1580 ?        Ss   16:42   0:00  \_ postgres: logical replication launcher
        postgres  2227  0.0  0.0 273300  2628 ?        Ss   16:44   0:00  \_ postgres: walsender repl 192.168.56.117(43686) streaming 0/3000060


standby
    -- process 확인 (wal receiver)

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2184 pts/0    S    15:26   0:00 -bash
        postgres  1946  0.0  0.0 155448  1844 pts/0    R+   16:47   0:00  \_ ps -u postgres uf
        postgres  1933  0.0  0.3 272612 13108 ?        Ss   16:44   0:00 /engine/pg14.2/bin/postgres -D /data/pg14.2
        postgres  1934  0.0  0.0 125436   924 ?        Ss   16:44   0:00  \_ postgres: logger
        postgres  1935  0.0  0.0 272776  1716 ?        Ss   16:44   0:00  \_ postgres: startup recovering 000000010000000000000003
        postgres  1936  0.0  0.0 272612  1092 ?        Ss   16:44   0:00  \_ postgres: checkpointer
        postgres  1937  0.0  0.0 272612  1896 ?        Ss   16:44   0:00  \_ postgres: background writer
        postgres  1938  0.0  0.0 127696  1000 ?        Ss   16:44   0:00  \_ postgres: stats collector
        postgres  1939  0.1  0.0 277024  1932 ?        Ss   16:44   0:00  \_ postgres: walreceiver streaming 0/3000060


    -- logfile 확인

        2022-04-11 16:44:33 KST [1933]: [3-1] user=,db=,remote=LOG:  starting PostgreSQL 14.2 on x86_64-pc-linux-gnu, compiled by gcc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-44), 64-bit
        2022-04-11 16:44:33 KST [1933]: [4-1] user=,db=,remote=LOG:  listening on IPv4 address "0.0.0.0", port 5414
        2022-04-11 16:44:33 KST [1933]: [5-1] user=,db=,remote=LOG:  listening on IPv6 address "::", port 5414
        2022-04-11 16:44:33 KST [1933]: [6-1] user=,db=,remote=LOG:  listening on Unix socket "/tmp/.s.PGSQL.5414"
        2022-04-11 16:44:33 KST [1935]: [1-1] user=,db=,remote=LOG:  database system was interrupted; last known up at 2022-04-11 16:43:51 KST
        2022-04-11 16:44:34 KST [1935]: [2-1] user=,db=,remote=LOG:  entering standby mode
        2022-04-11 16:44:34 KST [1935]: [3-1] user=,db=,remote=LOG:  redo starts at 0/2000028
        2022-04-11 16:44:34 KST [1935]: [4-1] user=,db=,remote=LOG:  consistent recovery state reached at 0/2000138
        2022-04-11 16:44:34 KST [1933]: [7-1] user=,db=,remote=LOG:  database system is ready to accept read-only connections
        2022-04-11 16:44:34 KST [1939]: [1-1] user=,db=,remote=LOG:  started streaming WAL from primary at 0/3000000 on timeline 1

replication test
    master
        create database test with encoding='UTF8' template=template0 lc_collate='C' lc_ctype='en_US.UTF-8' ;
        create user app password 'app1122' ;
        grant connect on database test to app ;
        grant all privileges on database test to app ;
        \c test app
        create table test ( id int ) ;
        insert into test values (1) ;
        select * from test ;
        \q

        echo "select * from pg_stat_replication ;" | /engine/pg14.2/bin/psql -p 5414 -d postgres

        pid  | usesysid | usename | application_name |  client_addr   | client_hostname | client_port |         backend_start         | backend_xmin |   state   | sent_lsn  | write_lsn | flush_lsn | replay_lsn | write_lag | flush_lag | replay_lag | sync_priority | sync_state |          reply_time
        ------+----------+---------+------------------+----------------+-----------------+-------------+-------------------------------+--------------+-----------+-----------+-----------+-----------+------------+-----------+-----------+------------+---------------+------------+-------------------------------
        2227 |    16384 | repl    | walreceiver      | 192.168.56.117 |                 |       43686 | 2022-04-11 16:44:34.124914+09 |              | streaming | 0/301AB38 | 0/301AB38 | 0/301AB38 | 0/301AB38  |           |           |            |             0 | async      | 2022-04-11 16:49:28.655154+09
        (1 row)


        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1852  0.0  0.0 115664  2184 pts/0    S    15:26   0:00 -bash
        postgres  2255  0.0  0.0 155448  1840 pts/0    R+   16:49   0:00  \_ ps -u postgres uf
        postgres  2208  0.0  0.3 272612 13108 ?        Ss   16:42   0:00 /engine/pg14.2/bin/postgres -D /data/pg14.2
        postgres  2209  0.0  0.0 125436   924 ?        Ss   16:42   0:00  \_ postgres: logger
        postgres  2211  0.0  0.0 272728  2844 ?        Ss   16:42   0:00  \_ postgres: checkpointer
        postgres  2212  0.0  0.0 272612  2132 ?        Ss   16:42   0:00  \_ postgres: background writer
        postgres  2213  0.0  0.1 272612  5288 ?        Ss   16:42   0:00  \_ postgres: walwriter
        postgres  2214  0.0  0.0 273160  2172 ?        Ss   16:42   0:00  \_ postgres: autovacuum launcher
        postgres  2215  0.0  0.0 127688  1136 ?        Ss   16:42   0:00  \_ postgres: stats collector
        postgres  2216  0.0  0.0 273048  1580 ?        Ss   16:42   0:00  \_ postgres: logical replication launcher
        postgres  2227  0.0  0.0 273644  2888 ?        Ss   16:44   0:00  \_ postgres: walsender repl 192.168.56.117(43686) streaming 0/301AB38

    standby

        ps -u postgres uf
        USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
        postgres  1691  0.0  0.0 115668  2184 pts/0    S    15:26   0:00 -bash
        postgres  1950  0.0  0.0 155448  1840 pts/0    R+   16:49   0:00  \_ ps -u postgres uf
        postgres  1933  0.0  0.3 272612 13108 ?        Ss   16:44   0:00 /engine/pg14.2/bin/postgres -D /data/pg14.2
        postgres  1934  0.0  0.0 125436   924 ?        Ss   16:44   0:00  \_ postgres: logger
        postgres  1935  0.0  0.0 272824  3572 ?        Ss   16:44   0:00  \_ postgres: startup recovering 000000010000000000000003
        postgres  1936  0.0  0.0 272612  2404 ?        Ss   16:44   0:00  \_ postgres: checkpointer
        postgres  1937  0.0  0.0 272612  1896 ?        Ss   16:44   0:00  \_ postgres: background writer
        postgres  1938  0.0  0.0 127696  1000 ?        Ss   16:44   0:00  \_ postgres: stats collector
        postgres  1939  0.1  0.0 277232  2184 ?        Ss   16:44   0:00  \_ postgres: walreceiver streaming 0/301AB38

        echo "select * from test;" | /engine/pg14.2/bin/psql -U app -p 5414 -d test

세션 모니터링

★깜장별★ — Wed, 6 Apr 2022 18:27:44 +0900

수행 시간을 주고 세션 모니터링 진행

active(1) : 1초 이상 수행중인 세션 모니터링

active(0) : 전체 세션

function active(ss){

cop = db.currentOp()

print ('------------------------------') ;

print ('Total connections :' , cop.inprog.length) ;

print ('------------------------------') ;

print (' ' ) ;

print ('Over ' , ss ,' secs session List');

print (' ') ;

cop.inprog.forEach(function(op){

els_sec = op.secs_running ;

if ( els_sec > ss ) {

print (' ') ;

print ( 'Type :' , op.type , '/ Opid :' , op.opid , '/ Kill op : db.killOp(', op.opid, ') / OP info :' , op.desc , '/ Client :' , op.client , '/ Operation :' , op.op , '/ NS :' , op.ns ) ;

print ( 'Active :' , op.active , '/ Secs_runing :' , els_sec ) ;

print ( 'Command :' );

printjson( op.command ) ;

print ('Plan :' ,op.planSummary) ;

print('Locks waiting :' , op.waitingForLock , '/ Num Yield :' , op.numYields ) ;

printjson(op.lockStats) ;

print ('--------------------') ;

}

)

print ('------------------------------') ;

print ('Total connections :' , cop.inprog.length) ;

print ('------------------------------') ;

}

active(1)

Cent7 + PostgreSQL 9.3 ~ 14 설치하기(source)

★깜장별★ — Wed, 6 Apr 2022 17:41:00 +0900

PostgreSQL 9.3 ~ 14을 테스트 용으로 하나의 서버에 구성해야 할 필요가 생겨서 설치했던 문서 공유

Cent7.9 minimal + PostgreSQL source

Download : https://www.postgresql.org/ftp/source/

#############
설치파일 업로드
#############

/source 에 압축 파일들 업로드
cd /source
ls
postgresql-10.20.tar.gz  postgresql-12.10.tar.gz  postgresql-14.2.tar.gz    postgresql-9.4.26.tar.gz  postgresql-9.6.24.tar.gz
postgresql-11.15.tar.gz  postgresql-13.6.tar.gz   postgresql-9.3.25.tar.gz  postgresql-9.5.25.tar.gz

#############
Limit
#############

vi  /etc/security/limits.conf
*                -       nofile          65536

#############
Firewall
#############

systemctl stop firewalld
systemctl disable firewalld

#############
Packages
#############

yum -y update
yum -y net-tools
yum -y install epel-release
yum -y install gcc gcc-c++ make autoconf readline readline-devel zlib zlib-devel openssl openssl-devel gettext gettext-devel python python-devel wget

##################
Add user & group
##################

useradd postgres

###################
Make Directories
###################

mkdir -p /engine/pg10.20
mkdir -p /engine/pg11.15
mkdir -p /engine/pg12.10
mkdir -p /engine/pg13.6
mkdir -p /engine/pg14.2
mkdir -p /engine/pg9.3
mkdir -p /engine/pg9.4
mkdir -p /engine/pg9.5
mkdir -p /engine/pg9.6

mkdir -p /data/pg10.20
mkdir -p /data/pg11.15
mkdir -p /data/pg12.10
mkdir -p /data/pg13.6
mkdir -p /data/pg14.2
mkdir -p /data/pg9.3
mkdir -p /data/pg9.4
mkdir -p /data/pg9.5
mkdir -p /data/pg9.6

chown -R postgres:postgres /source /engine /data

############
Install
############

#########
9.3
#########

su - postgres
cd /source
tar xvfz postgresql-9.3.25.tar.gz
cd postgresql-9.3.25

./configure --prefix=/engine/pg9.3
make
make world
make install
make install-world

/engine/pg9.3/bin/initdb -D /data/pg9.3

vi /data/pg9.3/postgresql.conf
listen_addresses = '*'
port = 5493

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg9.3/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg9.3/bin/pg_ctl -D /data/pg9.3 -l logfile start
/engine/pg9.3/bin/pg_ctl -D /data/pg9.3 status
/engine/pg9.3/bin/psql -p 5493 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg9.3/bin/pg_ctl -D /data/pg9.3 stop -mf -w

#########
9.4
#########

su - postgres
cd /source
tar xvfz postgresql-9.4.26.tar.gz
cd postgresql-9.4.26

./configure --prefix=/engine/pg9.4
make
make world
make install
make install-world

/engine/pg9.4/bin/initdb -D /data/pg9.4

vi /data/pg9.4/postgresql.conf
listen_addresses = '*'
port = 5494

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg9.4/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg9.4/bin/pg_ctl -D /data/pg9.4 -l logfile start
/engine/pg9.4/bin/pg_ctl -D /data/pg9.4 status
/engine/pg9.4/bin/psql -p 5494 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg9.4/bin/pg_ctl -D /data/pg9.4 stop -mf -w

#########
9.5
#########

su - postgres
cd /source
tar xvfz postgresql-9.5.25.tar.gz
cd postgresql-9.5.25

./configure --prefix=/engine/pg9.5
make
make world
make install
make install-world

/engine/pg9.5/bin/initdb -D /data/pg9.5

vi /data/pg9.5/postgresql.conf
listen_addresses = '*'
port = 5495

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg9.5/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg9.5/bin/pg_ctl -D /data/pg9.5 -l logfile start
/engine/pg9.5/bin/pg_ctl -D /data/pg9.5 status
/engine/pg9.5/bin/psql -p 5495 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg9.5/bin/pg_ctl -D /data/pg9.5 stop -mf -w

#########
9.6
#########

su - postgres
cd /source
tar xvfz postgresql-9.6.24.tar.gz
cd postgresql-9.6.24

./configure --prefix=/engine/pg9.6
make
make world
make install
make install-world

/engine/pg9.6/bin/initdb -D /data/pg9.6

vi /data/pg9.6/postgresql.conf
listen_addresses = '*'
port = 5496

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg9.6/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg9.6/bin/pg_ctl -D /data/pg9.6 -l logfile start
/engine/pg9.6/bin/pg_ctl -D /data/pg9.6 status
/engine/pg9.6/bin/psql -p 5496 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg9.6/bin/pg_ctl -D /data/pg9.6 stop -mf -w

#########
10
#########

su - postgres
cd /source
tar xvfz postgresql-10.20.tar.gz
cd postgresql-10.20

./configure --prefix=/engine/pg10.20
make
make world
make install
make install-world

/engine/pg10.20/bin/initdb -D /data/pg10.20

vi /data/pg10.20/postgresql.conf
listen_addresses = '*'
port = 5410

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg10.20/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg10.20/bin/pg_ctl -D /data/pg10.20 -l logfile start
/engine/pg10.20/bin/pg_ctl -D /data/pg10.20 status
/engine/pg10.20/bin/psql -p 5410 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg10.20/bin/pg_ctl -D /data/pg10.20 stop -mf -w

#########
11
#########

su - postgres
cd /source
tar xvfz postgresql-11.15.tar.gz
cd postgresql-11.15

./configure --prefix=/engine/pg11.15
make
make world
make install
make install-world

/engine/pg11.15/bin/initdb -D /data/pg11.15

vi /data/pg11.15/postgresql.conf
listen_addresses = '*'
port = 5411

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg11.15/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg11.15/bin/pg_ctl -D /data/pg11.15 -l logfile start
/engine/pg11.15/bin/pg_ctl -D /data/pg11.15 status
/engine/pg11.15/bin/psql -p 5411 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg11.15/bin/pg_ctl -D /data/pg11.15 stop -mf -w

#########
12
#########

su - postgres
cd /source
tar xvfz postgresql-12.10.tar.gz
cd postgresql-12.10

./configure --prefix=/engine/pg12.10
make
make world
make install
make install-world

/engine/pg12.10/bin/initdb -D /data/pg12.10

vi /data/pg12.10/postgresql.conf
listen_addresses = '*'
port = 5412

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg12.10/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg12.10/bin/pg_ctl -D /data/pg12.10 -l logfile start
/engine/pg12.10/bin/pg_ctl -D /data/pg12.10 status
/engine/pg12.10/bin/psql -p 5412 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg12.10/bin/pg_ctl -D /data/pg12.10 stop -mf -w

#########
13
#########

su - postgres
cd /source
tar xvfz postgresql-13.6.tar.gz
cd postgresql-13.6

./configure --prefix=/engine/pg13.6
make
make world
make install
make install-world

/engine/pg13.6/bin/initdb -D /data/pg13.6

vi /data/pg13.6/postgresql.conf
listen_addresses = '*'
port = 5413

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg13.6/pg_hba.conf
host all all 0.0.0.0/0 md5

/engine/pg13.6/bin/pg_ctl -D /data/pg13.6 -l logfile start
/engine/pg13.6/bin/pg_ctl -D /data/pg13.6 status
/engine/pg13.6/bin/psql -p 5413 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg13.6/bin/pg_ctl -D /data/pg13.6 stop -mf -w

#########
14
#########

su - postgres
cd /source
tar xvfz postgresql-14.2.tar.gz
cd postgresql-14.2

./configure --prefix=/engine/pg14.2
make
make world
make install
make install-world

/engine/pg14.2/bin/initdb -D /data/pg14.2

vi /data/pg14.2/postgresql.conf
listen_addresses = '*'
port = 5414

log_destination = 'stderr'

logging_collector = on

log_directory = 'pg_log'

log_filename = 'postgresql-%Y-%m-%d_%H%M%S.log'

log_rotation_age = 1d

log_line_prefix = '%t [%p]: [%l-1] user=%u,db=%d,remote=%r'

vi /data/pg14.2/pg_hba.conf
host    all             all             0.0.0.0/0               md5

/engine/pg14.2/bin/pg_ctl -D /data/pg14.2 -l logfile start
/engine/pg14.2/bin/pg_ctl -D /data/pg14.2 status
/engine/pg14.2/bin/psql -p 5414 -d postgres
\l
select * from pg_available_extensions;
alter user postgres with password 'postgres' ;
\q
--원격 접속 확인 (dbeaver같은 툴 이용)
/engine/pg14.2/bin/pg_ctl -D /data/pg14.2 stop -mf -w

####################
Profile Setting
####################

su - postgres

vi .bash_profile

### For PG

BASEPATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/root/bin

list_version=( "9.3" "9.4" "9.5" "9.6" "10.20" "11.15" "12.10" "13.6" "14.2")

echo " "
echo "------------------------------------ "
echo "     select PostgreSQL Version       "
echo "------------------------------------ "

PS3=`echo -e "\n>> "`
select select_version in "${list_version[@]}"
do

    if [ -n "$select_version" ]
    then
    value_ver=$select_version

    pghome=`echo /engine/pg$value_ver`
    pgdata=`echo /data/pg$value_ver`

    echo -e "\n\n\n * You selected $value_ver version\n\n\n"

    unset PATH
    unset PGHOME
    unset PGDATA

    export PATH=$pghome/bin:$BASEPATH
    export PGHOME=$pghome
    export PGDATA=$pgdata
    export PGDATABASE=postgres

    break;

    else
         echo -e "\n. . . Please choose the correct number . . ."
    fi
done

## Aliases

alias start_pg93='/engine/pg9.3/bin/pg_ctl -D /data/pg9.3 -l logfile start'
alias status_pg93='/engine/pg9.3/bin/pg_ctl -D /data/pg9.3 status'
alias client_pg93='/engine/pg9.3/bin/psql -p 5493 -d postgres'
alias stop_pg93='/engine/pg9.3/bin/pg_ctl -D /data/pg9.3 stop -mf -w'

alias start_pg94='/engine/pg9.4/bin/pg_ctl -D /data/pg9.4 -l logfile start'
alias status_pg94='/engine/pg9.4/bin/pg_ctl -D /data/pg9.4 status'
alias client_pg94='/engine/pg9.4/bin/psql -p 5494 -d postgres'
alias stop_pg94='/engine/pg9.4/bin/pg_ctl -D /data/pg9.4 stop -mf -w'

alias start_pg95='/engine/pg9.5/bin/pg_ctl -D /data/pg9.5 -l logfile start'
alias status_pg95='/engine/pg9.5/bin/pg_ctl -D /data/pg9.5 status'
alias client_pg95='/engine/pg9.5/bin/psql -p 5495 -d postgres'
alias stop_pg95='/engine/pg9.5/bin/pg_ctl -D /data/pg9.5 stop -mf -w'

alias start_pg96='/engine/pg9.6/bin/pg_ctl -D /data/pg9.6 -l logfile start'
alias status_pg96='/engine/pg9.6/bin/pg_ctl -D /data/pg9.6 status'
alias client_pg96='/engine/pg9.6/bin/psql -p 5496 -d postgres'
alias stop_pg96='/engine/pg9.6/bin/pg_ctl -D /data/pg9.6 stop -mf -w'

alias start_pg10='/engine/pg10.20/bin/pg_ctl -D /data/pg10.20 -l logfile start'
alias status_pg10='/engine/pg10.20/bin/pg_ctl -D /data/pg10.20 status'
alias client_pg10='/engine/pg10.20/bin/psql -p 5410 -d postgres'
alias stop_pg10='/engine/pg10.20/bin/pg_ctl -D /data/pg10.20 stop -mf -w'

alias start_pg11='/engine/pg11.15/bin/pg_ctl -D /data/pg11.15 -l logfile start'
alias status_pg11='/engine/pg11.15/bin/pg_ctl -D /data/pg11.15 status'
alias client_pg11='/engine/pg11.15/bin/psql -p 5411 -d postgres'
alias stop_pg11='/engine/pg11.15/bin/pg_ctl -D /data/pg11.15 stop -mf -w'

alias start_pg12='/engine/pg12.10/bin/pg_ctl -D /data/pg12.10 -l logfile start'
alias status_pg12='/engine/pg12.10/bin/pg_ctl -D /data/pg12.10 status'
alias client_pg12='/engine/pg12.10/bin/psql -p 5412 -d postgres'
alias stop_pg12='/engine/pg12.10/bin/pg_ctl -D /data/pg12.10 stop -mf -w'

alias start_pg13='/engine/pg13.6/bin/pg_ctl -D /data/pg13.6 -l logfile start'
alias status_pg13='/engine/pg13.6/bin/pg_ctl -D /data/pg13.6 status'
alias client_pg13='/engine/pg13.6/bin/psql -p 5413 -d postgres'
alias stop_pg13='/engine/pg13.6/bin/pg_ctl -D /data/pg13.6 stop -mf -w'

alias start_pg14='/engine/pg14.2/bin/pg_ctl -D /data/pg14.2 -l logfile start'
alias status_pg14='/engine/pg14.2/bin/pg_ctl -D /data/pg14.2 status'
alias client_pg14='/engine/pg14.2/bin/psql -p 5414 -d postgres'
alias stop_pg14='/engine/pg14.2/bin/pg_ctl -D /data/pg14.2 stop -mf -w'

Oracle to EDB (MTK)

★깜장별★ — Tue, 5 Apr 2022 13:38:58 +0900

EDB에서 제공하는 Migration Tool Kit

소스는 Oracle, Mysql , EDB , PG 다 될 수 있고, 타겟은 EDB만 가능

사용가능 DBMS

• PostgreSQL versions 9.5, 9.6, 10,11, and 12
• Advanced Server versions 9.5, 9.6, 10,11 and 12
• Oracle 10g Release 2
• Oracle 11g Release 2
• Oracle 12c Release 1
• SQL Server 2008
• SQL Server 2012
• SQL Server 2014
• MySQL 5.5.36
• Sybase Adaptive Server Enterprise 15.7

필요 요구사항 : 자바 1.7이상

EDB 사이트 가입 아이디 필요 (설치 시에 물어 봄)

EDB owner(enterprisedb) Oracle Client 필요.

1. JAVA 설치 (root 유저)

Linux Java 압축 해제

tar xvfz jdk-8u241-linux-x64.tar.gz

Linux를 rpm으로 설치 한 게 아니라서 패스에 등록하기 위한 디렉터리 생성 및 java 파일 이동

mkdir /usr/local/JDK

mv jdk1.8.0_241 /usr/local/JDK/

Java Path 등록

vi /etc/profile (파일 제일 밑에 다음 추가)

#JDK Path

export JAVA_HOME=/usr/local/JDK/jdk1.8.0_241

export PATH=$PATH:$JAVA_HOME/bin

서버 Rebooting 없이 Java 패스 적용

source /etc/profile

Java 확인

which java

/usr/local/JDK/jdk1.8.0_241/bin/java

java -version

java version "1.8.0_241"

Java(TM) SE Runtime Environment (build 1.8.0_241-b07)

Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

2. MTK 설치 (root 유저)

unzip edb-migrationtoolkit-51.0.1-1-linux-x64.run.zip

chmod 777 edb-migrationtoolkit-51.0.1-1-linux-x64.run

./edb-migrationtoolkit-51.0.1-1-linux-x64.run

설치는 영어로 하고, 설치 중간에 EDB 가입 이 메일과 비번 물어보면 입력

기본 설치 경로는 /opt/edb/mtk

소스와 타겟 정보를 설정해 주는 파일 백업

cd /opt/edb/mtk/etc

cp toolkit.properties toolkit.properties.bak (혹시 모르니 백업)

소스와 타겟 DB 설정

vi toolkit.properties (소스,타겟 수정)

소스 : 192.168.56.200 / ora11g / SH유저의 데이터 마이그레이션 예정

타겟 : 192.168.56.110 / tpch DB로 sh 스키마로 데이터 이전 예정

RC_DB_URL=jdbc:oracle:thin:@192.168.56.200:1511:ora11g
SRC_DB_USER=system
SRC_DB_PASSWORD=oracle

TARGET_DB_URL=jdbc:edb://192.168.56.110:5444/tpch
TARGET_DB_USER=sh
TARGET_DB_PASSWORD=sh

기타 DB 설정 참고

다른 예)
ex) EDB와 PG만 source/target 다 되고 나머지 Oracle,Mysql은 source만 가능

[PG]
SRC_DB_URL=jdbc:postgresql://192.168.56.110:5411/postgres
SRC_DB_USER=postgres
SRC_DB_PASSWORD=root

TARGET_DB_URL=jdbc:postgresql://192.168.56.110:5411/postgres
TARGET_DB_USER=postgres
TARGET_DB_PASSWORD=root

[Oracle]
SRC_DB_URL=jdbc:oracle:thin:@192.168.56.200:1511:ora11g
SRC_DB_USER=sh
SRC_DB_PASSWORD=sh

[Mysql]
SRC_DB_URL=jdbc:mysql://192.168.56.100[:5432]/tpch
SRC_DB_USER=soe
SRC_DB_PASSWORD=soe

소스가 Oracle인데 오라클 jar 파일 없으니 다음 파일 업로드 후에 /opt/edb/mtk/lib에 넣어 줌

[root@pgmechine lib]# pwd

/opt/edb/mtk/lib

[root@pgmechine lib]# ls

commons-lang3-3.1.jar edb-commons.jar edb-jdbc17.jar log4j-1.2.17.jar ojdbc6.jar ojdbc7.jar ojdbc8.jar

3. MTK 사용 (root 유저)

cd /opt/edb/mtk/bin

3.1 스키마만 먼저 conversion

./runMTK.sh -schemaOnly SH <-- 유저 명 대문자

3.2 스키마는 다 만들어 졌으니 data만 conversion

./runMTK.sh -dataOnly -fastCopy SH <-- fastCopy는 벌크 로딩으로 속도를 빠르게 하기 위함

CLOB 테이블은 마이그레이션 하는데 다음과 같은 에러 남

Loading Large Objects into table: TEST_CLOB ...

MTK-17001: Error Loading Data into Table: TEST_CLOB

DB-42501: com.edb.util.PSQLException: ERROR: permission denied for relation pg_class

LOB는 fastCopy를 사용하지 못하고 copyViaDBLinkOra를 사용해서 넣어야 함.

3.3 LOB를 가져오기 위해서는 EDB가 기동 될 때 오라클 클라이언트를 물고 기동해야 한다.

오라클 클라이언트 설치 (enterprisedb 유저)

su - enterprisedb

unzip instantclient-basic-linux.x64-19.5.0.0.0dbru.zip

unzip instantclient-sqlplus-linux.x64-19.5.0.0.0dbru.zip

unzip instantclient-sdk-linux.x64-19.5.0.0.0dbru.zip

cd ~

vi .bash_profile

export ORACLE_HOME=/home/enterprisedb/instantclient_19_5
export TNS_ADMIN=/home/enterprisedb/instantclient_19_5

export PATH=$PATH:$ORACLE_HOME
export LD_LIBRARY_PATH=$ORACLE_HOME

3.4 EDB postgresql.conf에 환경변수 oracle_home 설정

vi postgresql.conf

oracle_home ='/home/enterprisedb/instantclient_19_5'

EDB 재 기동

3.5 Error났던 LOB 테이블만 다시 데이터 밀어 넣기

./runMTK.sh -dataOnly -copyViaDBLinkOra -tables TEST_CLOB SH <-- 테이블 명, 유저 명 대문자

혹시 이 명령어 수행하다가 아래와 같은 에러 나면

MTK-11014: Error connecting to DBLinkOra
DB-HV00N: com.edb.util.PSQLException: ERROR: unable to load OCI library: libnnz19.so: cannot open shared object file: No such file
Hint: Either the oracle_home guc setting or Oracle library must be in path at the time you start the server.

enterprisedb 유저의 ORACLE_HOME 설정 과 postgresql.conf에 oracle_home 설정이 제대로 되어 있는지 확인하고, 제대로 되어 있는데도 에러가 나면 수동으로 enterprisedb 재 기동 해줌

서버 start시에 자동으로 edb가 기동되게 되어 있는데 이때 환경변수나 postgresql.conf를 잘 못 읽어 가는 듯 함.

./runMTK.sh -dataOnly -copyViaDBLinkOra -tables TEST_CLOB SH

수행 중 아래 에러가 또 발생 함. LOB를 가져오기 위해서는 super 유저 권한이 필요 함.

MTK-11014: Error connecting to DBLinkOra
DB-42501: com.edb.util.PSQLException: ERROR: must be superuser to use dblink_ora

enterprisedb로 로그인해서 superuser 권한을 SH에게 부여

alter user sh with superuser ;

./runMTK.sh -dataOnly -copyViaDBLinkOra -tables TEST_CLOB SH --> 정상적으로 마이그레이션이 됨.

4. 다른 유저로 데이터 넘기기

기본적으로 MTK를 사용하면 오라클의 스키마와 같은 스키마로 데이터를 마이그레이션 한다.

SH 유저에 superuser 권한이 부여 되어 있는 상태이니 toolkit.properties 파일은 수정하지 않고 오라클의 SH 유저의 테이블을 EDB의 soe 유저로 옮기는 테스트를 진행해 본다.

넘어 가는 지만 테스트 하면 되니, 일반 테이블과 LOB 테이블 두 개만 테스트 해 보기로 ...

create tablespace tbs2 location '/edb10.5/tbs2' ;
create user soe identified by soe ;
create database tpcc owner soe tablespace tbs2 ;

./runMTK.sh -schemaOnly -targetSchema soe -tables CHANNELS,TEST_CLOB SH

toolkit.properties파일에는 EDB 로그인을 SH 유저로 하고 , targetschema는 soe로 했더니

pg_tables를 조회해 보면 chanels와 test_clob 테이블의 스키마는 soe로 되었지만, table_owner는 sh로 됨.

LOB 데이터도 넘겨야 하니 EDB에서 superuser 권한을 soe에게 주고 toolkit.properties 파일의 EDB 로그인 유저를 soe로 바꿔서 다시 수행해 보자.

alter user soe with superuser ;

SRC_DB_URL=jdbc:oracle:thin:@192.168.56.200:1511:ora11g
SRC_DB_USER=system
SRC_DB_PASSWORD=oracle

TARGET_DB_URL=jdbc:edb://192.168.56.110:5444/tpcc
TARGET_DB_USER=soe
TARGET_DB_PASSWORD=soe

./runMTK.sh -schemaOnly -targetSchema soe -tables CHANNELS,TEST_CLOB SH

정상적으로 schema및 tableowner가 soe로 생성 됨.

이제 데이터 넣어보기

./runMTK.sh -dataOnly -fastCopy -targetSchema soe -tables CHANNELS SH

./runMTK.sh -dataOnly -copyViaDBLinkOra -targetSchema soe -tables TEST_CLOB SH

두 개의 테이블 모두 정상적으로 데이터가 들어 감.

5. 스키마의 여러 테이블 중에서 특정 테이블만 넘기기

scott의 여러 테이블 중에서 2개의 테이블만 넘기고자 한다면

./runMTK.sh -tables EMP,DEPT SCOTT

******************** Migration Summary ********************

Tables: 2 out of 2

Total objects: 2

Successful count: 2

Failed count: 0

Invalid count: 0

*************************************************************

위와 같이 넘기면 오로지 테이블 생성과 테이블 데이터만 넘어감. emp와 dept는 각자 PK와 FK를 가지고 있으며, PK에 해당하는 인덱스가 존재 함.

SQL> select constraint_name , table_name , constraint_type from user_constraints ;

CONSTRAINT_NAME                TABLE_NAME                      C
------------------------------ ------------------------------ -
FK_DEPTNO                       EMP                              R
PK_DEPT                        DEPT                              P
PK_EMP                               EMP                              P

SQL> create index emp_idx01 on emp ( empno , sal ) ;
Index created.

SQL> select index_name , table_name from user_indexes ;

INDEX_NAME                       TABLE_NAME
------------------------------ ------------------------------
PK_EMP                               EMP
EMP_IDX01                       EMP
PK_DEPT                        DEPT

제약 조건 및 인덱스를 생성하고자 다음과 같이 옵션을 주고 다시 시도하면

./runMTK.sh -indexes -constraints -tables EMP,DEPT SCOTT

******************** Migration Summary ********************

Tables: 2 out of 2

Constraints: 3 out of 3

Indexes: 1 out of 1

Total objects: 6

Successful count: 6

Failed count: 0

Invalid count: 0

*************************************************************

위의 결과를 보면 인덱스는 1개 만 마이그레이션 된 것 처럼 보이지만 대한 언급은 없지만 실제 EDB에서 조사를 해 보면 PK에 해당하는 인덱스가 생성 됨

6. MTK 옵션

참고 URL : https://www.enterprisedb.com/edb-docs/d/edb-postgres-advanced-server/user-guides/migration-guide/9.4/Postgres_Plus_Migration_Guide.1.24.html#

Feature:	Section	Relevant Options:
Offline Migration	7.1	-offlineMigration
Import Options	7.2	-sourcedbtype, -targetdbtype, -schemaOnly, -dataOnly
Schema Creation Options	7.3	-dropSchema, -targetSchema
Schema Object Selection Options	7.4	-allTables, -tables, -constraints, -ignoreCheckConstFilter, -skipCKConst, -skipFKConst, -skipColDefaultClause, -indexes, -triggers, -allViews, -views, -allSequences, -sequences, -allProcs, -procs, -allFuncs, -funcs, -checkFunctionBodies, -allPackages, -packages, -allRules,
Migration Options	7.5	-truncLoad, -enableConstBeforeDataLoad, -retryCount, -safeMode, -fastCopy, -analyze, vacuumAnalyze, -replaceNullChar, -copyDelimiter, -batchSize, -cpBatchSize, -fetchSize, -filterProp -customColTypeMapping, -customColTypeMappingFile
Oracle Specific Options	7.6	-allUsers, -users, -importPartitionAsTable, -objectTypes, -copyViaDBLinkOra, -allDBLinks -allSynonyms, -allPublicSynonyms, -allPrivateSynonyms, -useOraCase
Miscellaneous Options	7.7	-help, -logDir, -logFileCount, -logFileSize, -logBadSQL -verbose, -version

정기점검 스크립트 (HTML 및 OUT 파일 생성)

★깜장별★ — Tue, 5 Apr 2022 13:32:11 +0900

MySQL / MariaDB/ Percona 정기점검용 스크립트

스크립트를 돌리면 점검결과가 HMTL 파일로 생성되며, 좀 더 많은 내용은 out 파일에 저장 됨.

점검스크립트_2022_04_04.txt

0.05MB

ENUM 데이터 타입

★깜장별★ — Tue, 5 Apr 2022 13:25:58 +0900

enum 데이터 타입은 캐릭터형 데이터 타입이며, 해당 칼럼에 들어올 수 있는 값을 지정해 줄 수 있다. 테이블 생성 시 주어 진 list안의 데이터만 값을 입력 받고, 잘못 된 데이터는 '' 값으로 처리 하는 데이터 타입이다. 간단하게 비유 하자면 체크 제약 조건이라고 보면 된다.

해당 칼럼에 NOT NULL 조건이 없으면 들어갈 수 있는 데이터 값으로는 NULL , '' , List에 지정된 문자 가 칼럼에 값으로 들어갈 수 있다.

enum 카럼은 값으로도 조회가 가능하지만, 칼럼의 값(list)의 index값으로도 제어가 가능하다.

칼럼의 index로 데이터를 제어가 가능한 것은 장점이 될 수도 있고, 단점이 될 수도 있다.

칼럼에서 list의 순서가 변경 된다면, 칼럼의 index가 변경이 되어서, index를 가지고 insert나 select를 할 경우 기존과 다른 값을 입력하거나 조회 할 수 있게 된다.

1. 테스트 테이블 생성

CREATE TABLE test_enum
(id int NOT NULL ,
type_enum enum ('a','b','c','d','e') ) ;

위와 같이 생성 시에 type_enum 칼럼에 들어갈 수 있는 값과 인덱스 값은 다음과 같이 기대할 수 있다.

인덱스	칼럼 입력 값
NULL	NULL
0	'' 잘못 입력된 값 대신 들어가는 값
1	a
2	b
3	c
4	d
5	e

2. 테스트

#정상적인 데이터 Insert

INSERT INTO test_enum VALUES (1,'c'),(2,'a'),(3,'A'),(4,'e'),(5,1),(6,null);

COMMIT ;

SELECT id , type_enum , type_enum+0 FROM test_enum ;

id type_enum         type_enum+0
1      c         3
2      a         1
3      a         1
4      e         5
5      a         1
6     null     null

id가 5인 경우, 값이 아니라 인덱스 1값인 a가 Insert가 되었다.

# 리스트에 없는 데이터 Insert

INSERT INTO test_enum VALUES (7,'z') ;

COMMIT ;

SELECT id , type_enum , type_enum+0 FROM test_enum ;

id type_enum        type_enum+0
1      c        3
2      a        1
3      a        1
4      e        5
5      a        1
6    null      null
7               0

enum 리스트에 없는 데이터는 '' 값으로 들어가며, Index값은 0이다.

# enum 리스트에 새로운 값 추가(제일 뒤)

ALTER TABLE test_enum MODIFY type_enum enum ('a','b','c','d','e','z') ;

INSERT INTO test_enum VALUES (8,'z') ;

COMMIT ;

SELECT id , type_enum , type_enum+0 FROM test_enum ;

id   type_enum   type_enum+0
1        c           3
2        a           1
3        a           1
4        e           5
5        a           1
6       null       null
7                    0
8        z           6

리스트에 값을 추가 한 후에 추가된 값이 정상적으로 Insert 되는 것을 확인 할 수 있음.

# 칼럼 값 조회

SELECT * FROM test_enum WHERE type_enum = 1 ;

id type_enum
2     a
3     a
5     a

SELECT * FROM test_enum WHERE type_enum = 'a' ;

id   type_enum
2     a
3     a
5     a

SELECT * FROM test_enum WHERE type_enum = 'A' ;

id type_enum
2     a
3     a
5     a

칼럼 값의 대소 문자를 가리지 않고 조회 됨.

# 리스트의 값은 같지만, 리스트 순서를 변경 함.
이 경우 입력할 수 있는 값은 똑같지만, 값의 인덱스가 변경이 된다.

ALTER TABLE test_enum MODIFY type_enum enum ('z','e','d','c','b','a') ;

COMMIT ;

SELECT id , type_enum , type_enum+0 FROM test_enum ;

id   type_enum   type_enum+0
1       c          4
2       a          6
3       a          6
4       e          2
5       a          6
6      null      null
7                  0
8       z          1

테이블에 들어있는 칼럼의 값은 변화가 없지만, 인덱스 값의 변화가 있음을 확인 할 수 있다.

SELECT * FROM test_enum WHERE type_enum = 1 ;

id   type_enum
8       z

기존의 인덱스 1의 값은 a 였으나, 변경된 인덱스 1의 값은 z 임을 다시 확인 할 수 있다.
enum 사용시 주의 해야 할 부분.

# 조회시에 대소문자를 가리지 않는 것으로 확인이 되어서, Insert 시에도 가리지 않는지 확인

ALTER TABLE test_enum MODIFY type_enum enum ('z','e','d','c','b','a','A') ;

SELECT * FROM test_enum WHERE type_enum = 'A' ;

id   type_enum
2      a
3      a
5      a

SELECT * FROM test_enum WHERE type_enum = 'a' ;

id   type_enum
2      a
3      a
5      a

INSERT INTO test_enum VALUES (9,'A') ;

COMMIT ;

SELECT id , type_enum , type_enum+0 FROM test_enum ;

id   type_enum   type_enum+0
1      c           4
2      a           6
3      a           6
4      e           2
5      a           6
6     null       null
7                  0
8      z           1
9      a           6

대문자 A를 입력했는데, 값은 a로 들어 감.

SELECT * FROM test_enum WHERE type_enum = 'A' ;

id   type_enum
2      a
3      a
5      a
9      a

SELECT * FROM test_enum WHERE type_enum = 'a' ;

id   type_enum
2      a
3      a
5      a
9      a

INSERT INTO test_enum VALUES (10,7) ;

COMMIT ;

SELECT id , type_enum , type_enum+0 FROM test_enum ;

id        type_enum        type_enum+0
1         c            4
2         a            6
3         a            6
4         e            2
5         a            6
6       null         null
7                      0
8         z            1
9         a            6
10        A            7

SELECT * FROM test_enum WHERE type_enum = 'A' ;

id   type_enum
2       a
3       a
5       a
9       a
10      A

SELECT * FROM test_enum WHERE type_enum = 'a' ;

id   type_enum
2       a
3       a
5       a
9       a
10      A

동일한 문자의 대소는 가리지 않고 값으로 조회가 가능 함.
enum을 사용 시 주의 해야 할 부분

# 대소문자가 섞여 있는 리스트를 생성해 봄

ALTER TABLE test_enum MODIFY type_enum enum ('z','e','d','c','b','a','AbCd') ;

INSERT INTO test_enum VALUES (11,'abcd') ;
INSERT INTO test_enum VALUES (12,'ABCD') ;
INSERT INTO test_enum VALUES (13,'aBcD') ;

COMMIT;

SELECT id , type_enum , type_enum+0 FROM test_enum ;

id   type_enum   type_enum+0
1       c            4
2       a            6
3       a            6
4       e            2
5       a            6
6     null         null
7                    0
8       z            1
9       a            6
10      a            6
11    AbCd           7
12    AbCd           7
13    AbCd           7

SELECT * FROM test_enum WHERE type_enum = 'AbCd' ;

id   type_enum
11     AbCd
12     AbCd
13     AbCd

SELECT * FROM test_enum WHERE type_enum = 'abcd' ;

id   type_enum
11     AbCd
12     AbCd
13     AbCd

SELECT * FROM test_enum WHERE type_enum = 'ABCD' ;

id   type_enum
11     AbCd
12     AbCd
13     AbCd

3. enum 칼럼에 지정된 값 이외에는 들어가지 않게 제한 주기

# 테이블 구조 확인

MariaDB [maria5]> desc test_enum ;
+-----------+------------------------------------------+------+-----+---------+-------+
| Field     | Type                                     | Null | Key | Default | Extra |
+-----------+------------------------------------------+------+-----+---------+-------+
| id        | int(11)                                  | NO   |     | NULL    |       |
| type_enum | enum('z','e','d','c','b','a','A','AbCd') | YES |     | NULL    |       |
+-----------+------------------------------------------+------+-----+---------+-------+

# SQL_MODE 확인

MariaDB [maria5]> select @@sql_mode ;
+------------+
| @@sql_mode |
+------------+
|            |
+------------+

MariaDB [maria5]> insert into test_enum values (15,'y'),(16,'x') ;
MariaDB [maria5]> select * from test_enum ;
+----+-----------+
| id | type_enum |
+----+-----------+
| 1 | c         |
| 2 | a         |
| 3 | a         |
| 4 | e         |
| 5 | a         |
| 6 | NULL      |
| 7 |           |
| 8 | z         |
| 9 | a         |
| 10 | A         |
| 11 | AbCd      |
| 12 | AbCd      |
| 13 | AbCd      |
| 14 |           |
| 15 |           |
| 16 |           |
+----+-----------+

# SQL_MODE 변경

MariaDB [maria5]> set global sql_mode ='STRICT_TRANS_TABLES' ;
MariaDB [maria5]> show global variables like '%sql_mode%' ;
+---------------+---------------------+
| Variable_name | Value               |
+---------------+---------------------+
| sql_mode      | STRICT_TRANS_TABLES |
+---------------+---------------------+

앞으로 로그인 하는 세션들은 영향을 받음.

[root@CMaria ~]# mysql5 -uroot -p
MariaDB [maria5]> insert into test_enum values (17 ,'y') ;     <-- 리스트에 없는 값 에러 발생
ERROR 1265 (01000): Data truncated for column 'type_enum' at row 1

MariaDB [maria5]> insert into test_enum values (17 ,'ABCD') ;   <-- 여전히 대소문자는 구별 안 함
Query OK, 1 row affected (0.08 sec)

MariaDB [maria5]> select * from test_enum ;
+----+-----------+
| id | type_enum |
+----+-----------+
| 1 | c         |
| 2 | a         |
| 3 | a         |
| 4 | e         |
| 5 | a         |
| 6 | NULL      |
| 7 |           |
| 8 | z         |
| 9 | a         |
| 10 | A         |
| 11 | AbCd      |
| 12 | AbCd      |
| 13 | AbCd      |
| 14 |           |
| 15 |           |
| 16 |           |
| 17 | AbCd      |
+----+-----------+

events_statements_summary_by_digest view를 이용한 쿼리 추출

★깜장별★ — Tue, 5 Apr 2022 13:19:39 +0900

performance_schema를 활성화 시켰을 경우 SQL별 통계정보가 쌓이는 뷰를 이용하여 튜닝 대상 선정에 유용하다.

해당 뷰는 performance_schema_digests_size라는 파라미터 값의 사이즈에 영향을 받는다.

해당 파라미터는 events_statements_summary_by_digest 뷰에 저장될 수 있는 row 수를 제한한다. -1은 auto sizing.

해당 파라미터 값 만큼의 데이터가 다 차면 데이터가 NULL로 된다. truncate 시키면 이전 수집 데이터는 다 지워지고 새로 정보들이 쌓인다.

show global variables like 'performance_schema_digests_size' ;

performance_schema_digests_size = 1000

조회 쿼리 :

SELECT schema_name,

DIGEST_TEXT AS query,

IF(SUM_NO_GOOD_INDEX_USED > 0 OR SUM_NO_INDEX_USED > 0, '*', '') AS full_scan,

COUNT_STAR AS exec_count,

SEC_TO_TIME(SUM_TIMER_WAIT/1000000000000) AS exec_time_total,

SEC_TO_TIME(MAX_TIMER_WAIT/1000000000000) AS exec_time_max,

SEC_TO_TIME(AVG_TIMER_WAIT/1000000000000) AS exec_time_avg,

SUM_ROWS_EXAMINED / COUNT_STAR AS rows_scanned_avg ,

SUM_ROWS_SENT AS rows_sent,

ROUND(SUM_ROWS_SENT / COUNT_STAR) AS rows_sent_avg,

SUM_ERRORS AS err_count,

SUM_WARNINGS AS warn_count,

DIGEST AS digest

FROM performance_schema.events_statements_summary_by_digest

WHERE SEC_TO_TIME(AVG_TIMER_WAIT/1000000000000) > 1

ORDER BY 8 DESC ;

Temp 테이블을 사용하는 쿼리 확인

SELECT IF(LENGTH(DIGEST_TEXT) > 64, CONCAT(LEFT(DIGEST_TEXT, 30), ' ... ', RIGHT(DIGEST_TEXT, 30)), DIGEST_TEXT) AS query,

COUNT_STAR AS exec_count,

SUM_CREATED_TMP_TABLES AS memory_tmp_tables,

SUM_CREATED_TMP_DISK_TABLES AS disk_tmp_tables,

ROUND(SUM_CREATED_TMP_TABLES / COUNT_STAR) AS avg_tmp_tables_per_query,

ROUND((SUM_CREATED_TMP_DISK_TABLES / SUM_CREATED_TMP_TABLES) * 100) AS tmp_tables_to_disk_pct,

DIGEST AS digest

FROM performance_schema.events_statements_summary_by_digest

WHERE SUM_CREATED_TMP_TABLES > 0

ORDER BY SUM_CREATED_TMP_DISK_TABLES DESC, SUM_CREATED_TMP_TABLES DESC LIMIT 5;

Sort 쿼리

SELECT IF(LENGTH(DIGEST_TEXT) > 64, CONCAT(LEFT(DIGEST_TEXT, 30), ' ... ', RIGHT(DIGEST_TEXT, 30)), DIGEST_TEXT) AS query,

COUNT_STAR AS exec_count,

SUM_SORT_MERGE_PASSES AS sort_merge_passes,

ROUND(SUM_SORT_MERGE_PASSES / COUNT_STAR) AS avg_sort_merges,

SUM_SORT_SCAN AS sorts_using_scans,

SUM_SORT_RANGE AS sort_using_range,

SUM_SORT_ROWS AS rows_sorted,

ROUND(SUM_SORT_ROWS / COUNT_STAR) AS avg_rows_sorted,

DIGEST AS digest

FROM performance_schema.events_statements_summary_by_digest

WHERE SUM_SORT_ROWS > 0

ORDER BY SUM_SORT_MERGE_PASSES DESC, SUM_SORT_SCAN DESC, SUM_SORT_ROWS DESC LIMIT 5;

Full Table Scan

SELECT IF(LENGTH(DIGEST_TEXT) > 64, CONCAT(LEFT(DIGEST_TEXT, 30), ' ... ', RIGHT(DIGEST_TEXT, 30)), DIGEST_TEXT) AS query,

COUNT_STAR AS exec_count,

SUM_NO_INDEX_USED AS no_index_used_count,

SUM_NO_GOOD_INDEX_USED AS no_good_index_used_count,

ROUND((SUM_NO_INDEX_USED / COUNT_STAR) * 100) no_index_used_pct,

DIGEST AS digest

FROM performance_schema.events_statements_summary_by_digest

WHERE SUM_NO_INDEX_USED > 0

OR SUM_NO_GOOD_INDEX_USED > 0

ORDER BY no_index_used_pct DESC, exec_count DESC LIMIT 5;

Error or Warning 쿼리

SELECT IF(LENGTH(DIGEST_TEXT) > 64, CONCAT(LEFT(DIGEST_TEXT, 30), ' ... ', RIGHT(DIGEST_TEXT, 30)), DIGEST_TEXT) AS query,

COUNT_STAR AS exec_count,

SUM_ERRORS AS errors,

(SUM_ERRORS / COUNT_STAR) * 100 as error_pct,

SUM_WARNINGS AS warnings,

(SUM_WARNINGS / COUNT_STAR) * 100 as warning_pct,

DIGEST AS digest

FROM performance_schema.events_statements_summary_by_digest

WHERE SUM_ERRORS > 0

OR SUM_WARNINGS > 0

ORDER BY SUM_ERRORS DESC, SUM_WARNINGS DESC;

Bucketing 데이터를 unwind 후 CSV로 데이터 추출

★깜장별★ — Mon, 4 Apr 2022 17:18:15 +0900

0. 요구사항

현재 데이터는 고객 아이디 별로 Bucketing(CUST_CNTA_EXP_H 필드)되어 있어서 하루 전 데이터를 export할 경우, 해당 데이터가 아닌 것도 데이터가 같이 추출 됨 (Document단위이기 때문)

Hadoop으로 데이터를 가져가야 해서 CSV형태로 데이터를 추출해서 저장해야 함.

1. 구현 방법

1. 하루 전 데이터를 aggregate pipeline으로 새로운 collection으로 저장

2. 새로운 collection을 export

2. 구현 Shell

2.1 Linux Shell

#!/bin/bash

yesterDay=`date -d yesterday +%Y-%m-%d`
toDay=`date +%Y-%m-%d`

echo "Start --------------------------" >> /home/mongo/scripts/$toDay.out
echo "Export $yesterDay data..... " >> /home/mongo/scripts/$toDay.out
date >> /home/mongo/scripts/$toDay.out
echo "--------------------------" >> /home/mongo/scripts/$toDay.out
echo " " >> /home/mongo/scripts/$toDay.out
echo "To make $yesterDay data to new collection " >> /home/mongo/scripts/$toDay.out
echo " " >> /home/mongo/scripts/$toDay.out

nohup /mongo_sw/ent/bin/mongo "mongodb://ceppadm:Pceppadm%21123@172.23.1.11:27017,172.23.1.12:27017,172.23.1.13:27019/pcepp?authSource=admin&replicaSet=pceppReplicaSet" < /home/mongo/scripts/make_for_hadoop.js >> /home/mongo/scripts/$toDay.out 2>&1 &

echo " " >> /home/mongo/scripts/$toDay.out
echo "Waiting...." >> /home/mongo/scripts/$toDay.out

while true
do
date >> /home/mongo/scripts/$toDay.out
checkEnding=`grep -rnw /home/mongo/scripts/$toDay.out -e 'Collection created'`
if [ "$checkEnding" != "" ]
then
break
fi
sleep 300
done

echo "---------------------------" >> /home/mongo/scripts/$toDay.out
date >> /home/mongo/scripts/$toDay.out
echo "Finished to create collection " >> /home/mongo/scripts/$toDay.out
echo "---------------------------" >> /home/mongo/scripts/$toDay.out
echo " " >> /home/mongo/scripts/$toDay.out

echo "Start export" >> /home/mongo/scripts/$toDay.out
date >> /home/mongo/scripts/$toDay.out
echo "---------------------------" >> /home/mongo/scripts/$toDay.out
echo " " >> /home/mongo/scripts/$toDay.out
nohup /mongo_sw/ent/bin/mongoexport -c for_hadoop -o $toDay.csv --type=csv -f "CUST_NO,YYMM,CUST_CNTA_EXP_H" --uri="mongodb://ceppadm:Pceppadm%21123@172.23.1.11:27017,172.23.1.12:27017,172.23.1.13:27019/pcepp?authSource=admin&replicaSet=pceppReplicaSet" >> /home/mongo/scripts/$toDay.out 2>&1 &

2.2 새로운 컬렉션을 만드는 MQL

기본으로 몽고 DB는 UTC 시간을 사용하는데, LG U+에서는 한국 시간으로 데이터를 넣고 있어서 한국 시간으로 변형 필요

하루 전 날짜는 - (1000*60*60*24*1) 해 주면 됨

make_for_hadoop.js

var YD=new Date(ISODate() - (1000*60*60*24*1) )
YD
var yyyy=YD.getFullYear();
var mm=YD.getMonth()+1;
var dd=YD.getDate();
var hh=YD.getHours();
var min=YD.getMinutes();

if(mm<10){mm='0'+mm}
if(dd<10){dd='0'+dd}

var yyyymm=yyyy+mm ;
var startTime=yyyy+'-'+mm+'-'+dd+' 00:00:00.000+0900';
var endTime=yyyy+'-'+mm+'-'+dd+' 23:59:59.000+0900' ;

yyyymm
startTime
endTime

db.tb_cc_cust_cnta_exp_h.aggregate([
{ $match : { YYMM : yyyymm , "CUST_CNTA_EXP_H.SYS_CREATION_DATE" : { $gte : startTime }} } ,
{ $unwind : "$CUST_CNTA_EXP_H" } ,
{ $match : { "CUST_CNTA_EXP_H.SYS_CREATION_DATE" : { $gte : startTime , $lte : endTime} } } ,
{ $project : {_id : 0 , HST_CNT : 0 , CUCT_DTTM_MIN : 0 , CUCT_DTTM_MAX : 0 , CUST_EXP_EXTR_DT : 0} } ,
{ $out : "for_hadoop" }
] ,
{allowDiskUse : true} )

db.for_hadoop.count({ })

print ("Collection created...")

위와 같이 하면 CUST_CNTA_EXP_H Key값에 Sub Document Type으로 되어서 export 시에 이상하게 됨

아래와 같이 변경 하여 Key : Value 형태로 변경을 다시 한번 해 줘야 함.

make_for_hadoop.js

var YD=new Date(ISODate() - (1000*60*60*24*1) )
YD
var yyyy=YD.getFullYear();
var mm=YD.getMonth()+1;
var dd=YD.getDate();
var hh=YD.getHours();
var min=YD.getMinutes();

if(mm<10){mm='0'+mm}
if(dd<10){dd='0'+dd}

var yyyymm=yyyy+mm ;
var startTime=yyyy+'-'+mm+'-'+dd+' 00:00:00.000+0900';
var endTime=yyyy+'-'+mm+'-'+dd+' 23:59:59.000+0900' ;

yyyymm
startTime
endTime

db.tb_cc_cust_cnta_exp_h.aggregate([
{ $match : { YYMM : yyyymm , "CUST_CNTA_EXP_H.SYS_CREATION_DATE" : { $gte : startTime }} } ,
{ $unwind : "$CUST_CNTA_EXP_H" } ,
{ $match : { "CUST_CNTA_EXP_H.SYS_CREATION_DATE" : { $gte : startTime , $lte : endTime} } } ,
{ $replaceRoot: { newRoot : { $mergeObject : [ "$CUST_CNTA_EXP_H" , "$$ROOT" ] } } } ,
{ $project : {_id : 0 , HST_CNT : 0 , CUCT_DTTM_MIN : 0 , CUCT_DTTM_MAX : 0 , CUST_EXP_EXTR_DT : 0 , CUST_CNTA_EXP_H : 0 } } ,
{ $out : "for_hadoop" }
] ,
{allowDiskUse : true} )

db.for_hadoop.count({ })

print ("Collection created...")

$unwind 한 결과가 배열로 나오는지 ,Document로 나오는지에 따라서 다음과 같은 방법으로 상위로 올릴 수 있다.

// 결과가 Array 로 나올 시
{ $replaceRoot: { newRoot: { $mergeObjects: [ { $arrayElemAt: [ "$CUST_CNTA_EXP_H", 0 ] }, "$$ROOT" ] } } },
{ $project:{_id:0,CUST_CNTA_EXP_H:0}}

// 결과가 Document 로 나올 시
{ $replaceRoot: { newRoot: { $mergeObjects: [ $CUST_CNTA_EXP_H, "$$ROOT" ] } } },
{ $project:{_id:0,CUST_CNTA_EXP_H:0}}

timediff 계산

★깜장별★ — Mon, 4 Apr 2022 17:08:03 +0900

Aggregation 을 이용한 Field 시간 계산

use('hmc');

// db.test.drop();

// db.test.insertOne(

// {

// vin : "KNFSTZ76BFK964124",

// event_time : ISODate("2021-03-16T14:14:45.000Z"),

// system_time : ISODate("2021-03-16T14:14:49.000Z")

// }

// )

db.test.aggregate(

{

$match:

{

vin : "KNFSTZ76BFK964124" ,

event_time : { "$gte" : ISODate("2021-03-16T14:14:40Z"), "$lt" : ISODate("2021-03-16T14:14:50Z") }

}

{

$project:

{

_id:0,

vin : 1,

event_time : 1,

system_time : 1,

timeDiff : {

$cond: {

if: { $gte: ["$system_time", "$event_time"] },

then: { $subtract: [ "$system_time", "$event_time" ] },

else : 0

}

{$match : {timeDiff:{$gte:0}}}

)

system_time 과 event_time 차이가 1초 이상 나는 데이터 조회

use('hmc');

let timeDiff = {

$cond: {

if: { $gte: ["$system_time", "$event_time"] },

then: { $subtract: [ "$system_time", "$event_time" ] },

else : 0

}

};

db.test.find(

{

vin : "KNFSTZ76BFK964124" ,

event_time : { "$gte" : ISODate("2021-03-16T14:14:40Z"), "$lt" : ISODate("2021-03-16T14:14:50Z") },

$expr: { $gte:[ timeDiff, NumberDecimal("1000") ] }

}

,{_id:0, vin : 1, event_time : 1, system_time : 1}

)

MySQL to MongoDB 마이그레이션

★깜장별★ — Mon, 4 Apr 2022 17:06:10 +0900

Oracle Sample schema Download : https://www.oracletutorial.com/getting-started/oracle-sample-database/

테이블 생성 시 VARCHAR2 --> VARCHAR, NUMBER --> DOUBLE, to_date --> STR_TO_DATE('07-JUN-16','%d-%b-%y')

ERD

Table Info

MySQL 5.7 이상에서 데이터를 추출

JSON 함수를 이용하여 데이터 추출

JSON 함수에 지정해준 순서대로 추출되지는 않음 --;

오라클(value)과 다르게 콤마(,)로 구분 됨

칼럼헤더가 output 파일에 들어가지 않게 -sN 옵션 사용

/tmp/myorder.sql

SELECT 'use ot' ;
SELECT CONCAT( 'db.myorder.insert(' , JSON_OBJECT(
            'customer_id' ,o.customer_id
            , 'order_id' , o.order_id
            , 'status' , o.status
            , 'order_date' , o.order_date
            , 'sales_id' , o.salesman_id
            , 'order_detail' , ( SELECT JSON_ARRAYAGG( JSON_OBJECT(
                                                          'product_name' , p.product_name
                                                         , 'quantity' , oi.quantity
                                                         , 'unit_price' , oi.unit_price ))
                                                   FROM   order_items oi , products p
                                                   WHERE oi.order_id = o.order_id AND oi.product_id = p.product_id
                                   )
            , 'order_sum_price' , ( select sum(quantity*unit_price) from order_items oi where oi.order_id = o.order_id)
             ) , ' )' )
FROM    orders o ;

/engine/mysql-5.7/bin/mysql -h 127.0.0.1 -P 3357 -u root -p'root1122' -D ot -sN < /tmp/myorder.sql > /tmp/myorder.json

use ot

db.myorder.insert({"status": "Pending", "order_id": 1, "sales_id": 56, "order_date": "2017-10-15", "customer_id": 4, "order_detail": [{"quantity": 116, "unit_price": 469.99, "product_name": "Asus Z10PE-D16 WS"}, {"quantity": 77, "unit_price": 686.99, "product_name": "G.Skill Ripjaws V Series"}, {"quantity": 52, "unit_price": 525.99, "product_name": "Supermicro H8DG6-F"}, {"quantity": 131, "unit_price": 645.99, "product_name": "G.Skill Ripjaws V Series"}, {"quantity": 95, "unit_price": 640.99, "product_name": "G.Skill Ripjaws V Series"}, {"quantity": 41, "unit_price": 645.2, "product_name": "G.Skill Ripjaws V Series"}, {"quantity": 129, "unit_price": 383.98, "product_name": "Asus X99-DELUXE II"}, {"quantity": 139, "unit_price": 1099.99, "product_name": "Corsair Vengeance LPX"}, {"quantity": 147, "unit_price": 525.99, "product_name": "Supermicro H8DG6-F"}, {"quantity": 105, "unit_price": 1844.89, "product_name": "Intel Xeon E5-2683 V4 (OEM/Tray)"}, {"quantity": 103, "unit_price": 309.85, "product_name": "Asus PRIME X299-A"}, {"quantity": 121, "unit_price": 1999.89, "product_name": "HP C2J95AT"}], "order_sum_price": 1054107.74} )

db.myorder.insert({"status": "Shipped", "order_id": 2, "sales_id": null, "order_date": "2015-04-26", "customer_id": 4, "order_detail": [{"quantity": 111, "unit_price": 298.98, "product_name": "Asus MAXIMUS IX CODE"}, {"quantity": 121, "unit_price": 1431.99, "product_name": "G.Skill Trident Z"}, {"quantity": 39, "unit_price": 1055.99, "product_name": "G.Skill Ripjaws 4 Series"}, {"quantity": 49, "unit_price": 799, "product_name": "Crucial"}, {"quantity": 30, "unit_price": 1908.73, "product_name": "Intel Xeon E5-2690 V3"}, {"quantity": 95, "unit_price": 949.89, "product_name": "PNY VCQK4200-PB"}, {"quantity": 100, "unit_price": 686.99, "product_name": "G.Skill Ripjaws V Series"}, {"quantity": 97, "unit_price": 419.99, "product_name": "Supermicro X11SSL-CF"}, {"quantity": 75, "unit_price": 620.95, "product_name": "Intel Core i7-4790K"}], "order_sum_price": 590302.91} )

db.myorder.insert({"status": "Shipped", "order_id": 3, "sales_id": null, "order_date": "2017-04-26", "customer_id": 5, "order_detail": [{"quantity": 111, "unit_price": 66.89, "product_name": "Seagate ST2000DM006"}, {"quantity": 65, "unit_price": 16.99, "product_name": "Western Digital WD2500AAJS"}, {"quantity": 63, "unit_price": 479.99, "product_name": "Supermicro X9SRH-7TF"}, {"quantity": 46, "unit_price": 525.99, "product_name": "Supermicro H8DG6-F"}, {"quantity": 112, "unit_price": 644, "product_name": "Kingston"}, {"quantity": 81, "unit_price": 1299.99, "product_name": "Corsair Vengeance LPX"}, {"quantity": 73, "unit_price": 768.99, "product_name": "G.Skill Trident Z"}, {"quantity": 127, "unit_price": 948.99, "product_name": "Supermicro X10SDV-8C-TLN4F"}], "order_sum_price": 417049.24} )

mongo --host mongo:9999 -u root -p root1122 --authenticationDatabase admin < /tmp/order.json

TTL Index

★깜장별★ — Mon, 4 Apr 2022 16:59:50 +0900

TTL 인덱스는 날짜 필드에 기간을 주고 인덱스를 생성하게 되면, 자동으로 해당 시간이 경과한 데이터를 삭제해 주는 기능이 있다.

데이터를 보관 주기 이후에 삭제해야 하는 경우에 주로 사용된다.

1. 인덱스 생성 시 인덱스 이름을 주고 생성하는 방법

db.runCommand({
createIndexes : "컬렉션이름" ,
indexes : [
{ name : "index이름" ,
key : { key1 : 1 , key2 : -1 } ,
background : false ,
unique : false ,
expireAfterSeconds : "TTL Index일 경우 초 설정"
}
]
})

2. TTL Index 모니터링 주기 확인

db.adminCommand({getParameter:1, ttlMonitorSleepSecs: 1});

3.TTL Monitoring 상태 확인 및 disable/enable

db.adminCommand({getParameter:1, ttlMonitorEnabled:1});

db.adminCommand({setParameter:1, ttlMonitorEnabled:false});

db.adminCommand({getParameter:1, ttlMonitorEnabled:true});

4.TTL index에 의해서 지워진 데이터 확인

db.serverStatus().metrics.ttl.deleteDocuments

5. 잘 지워 지고 있는지 확인하기 위하여 해당 시간의 날짜 확인

new Date( ISODate() - (1000*60*60*24*200) ) -- 200일 전의 시간 확인 (UTC)

oplog Resizing

★깜장별★ — Mon, 4 Apr 2022 16:56:01 +0900

MongoDB 3.4 이전에서는 oplog 컬렉션을 삭제 하고 다시 만들어 줘야 함.

3.4 이상에서는 replSetResizeOplog 명령을 이용하여 oplog의 크기를 변경.

use local
rs.printReplicationInfo() -- OPlog Size 및 보관 기간 확인
db.oplog.rs.stats().maxSize --size 확인 후

secondary부터 oplog 사이즈 변경 하고 나중에 Primary 변경 해 준다
db.adminCommand({replSetResizeOplog: 1, size: 16000}) <--- 16GB로 변경 (16000MB)

예) 3.4.24 버전 OPlog Size 변경 (1G --> 5G)

3.4.24 버전임

Primary : 27342

Secondary : 27341, 27343

repl34:PRIMARY> rs.status()

{

"set" : "repl34",

"date" : ISODate("2021-11-22T06:32:52.007Z"),

"myState" : 1,

"term" : NumberLong(2),

"syncingTo" : "",

"syncSourceHost" : "",

"syncSourceId" : -1,

"heartbeatIntervalMillis" : NumberLong(2000),

"optimes" : {

"lastCommittedOpTime" : {

"ts" : Timestamp(1637562764, 1),

"t" : NumberLong(2)

"appliedOpTime" : {

"ts" : Timestamp(1637562764, 1),

"t" : NumberLong(2)

"durableOpTime" : {

"ts" : Timestamp(1637562764, 1),

"t" : NumberLong(2)

}

"members" : [

{

"_id" : 0,

"name" : "mongoSvr:27341",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 949,

생략 ...

{

"_id" : 1,

"name" : "mongoSvr:27342",

"health" : 1,

"state" : 1,

"stateStr" : "PRIMARY",

"uptime" : 951,

생략 ...

{

"_id" : 2,

"name" : "mongoSvr:27343",

"health" : 1,

"state" : 2,

"stateStr" : "SECONDARY",

"uptime" : 944,

생략 ...

}

"ok" : 1

}

-- Secondary 변경

mongo --port 27343 -uroot -proot1122 admin

repl34:SECONDARY> use local

switched to db local

repl34:SECONDARY> rs.printReplicationInfo() ;

configured oplog size: 1024MB

log length start to end: 1633secs (0.45hrs)

oplog first event time: Mon Nov 22 2021 15:13:01 GMT+0900 (KST)

oplog last event time: Mon Nov 22 2021 15:40:14 GMT+0900 (KST)

now: Mon Nov 22 2021 15:40:18 GMT+0900 (KST)

repl34:SECONDARY> db.adminCommand({replSetResizeOplog: 1, size: 5000})

{ "ok" : 1 }

repl34:SECONDARY> rs.printReplicationInfo() ;

configured oplog size: 5000MB

log length start to end: 1683secs (0.47hrs)

oplog first event time: Mon Nov 22 2021 15:13:01 GMT+0900 (KST)

oplog last event time: Mon Nov 22 2021 15:41:04 GMT+0900 (KST)

now: Mon Nov 22 2021 15:41:07 GMT+0900 (KST)

-- Secondary 변경

mongo --port 27341 -uroot -proot1122 admin

repl34:SECONDARY> use local

switched to db local

repl34:SECONDARY> rs.printReplicationInfo() ;

configured oplog size: 1024MB

log length start to end: 1743secs (0.48hrs)

oplog first event time: Mon Nov 22 2021 15:13:01 GMT+0900 (KST)

oplog last event time: Mon Nov 22 2021 15:42:04 GMT+0900 (KST)

now: Mon Nov 22 2021 15:42:05 GMT+0900 (KST)

repl34:SECONDARY> db.adminCommand({replSetResizeOplog: 1, size: 5000})

{ "ok" : 1 }

repl34:SECONDARY> rs.printReplicationInfo() ;

configured oplog size: 5000MB

log length start to end: 1753secs (0.49hrs)

oplog first event time: Mon Nov 22 2021 15:13:01 GMT+0900 (KST)

oplog last event time: Mon Nov 22 2021 15:42:14 GMT+0900 (KST)

now: Mon Nov 22 2021 15:42:22 GMT+0900 (KST)

-- Primary 변경

mongo --port 27342 -uroot -proot1122 admin

repl34:PRIMARY> use local

switched to db local

repl34:PRIMARY> rs.printReplicationInfo() ;

configured oplog size: 1024MB

log length start to end: 1793secs (0.5hrs)

oplog first event time: Mon Nov 22 2021 15:13:01 GMT+0900 (KST)

oplog last event time: Mon Nov 22 2021 15:42:54 GMT+0900 (KST)

now: Mon Nov 22 2021 15:42:54 GMT+0900 (KST)

repl34:PRIMARY> db.adminCommand({replSetResizeOplog: 1, size: 5000})

{ "ok" : 1 }

repl34:PRIMARY> rs.printReplicationInfo() ;

configured oplog size: 5000MB

log length start to end: 1793secs (0.5hrs)

oplog first event time: Mon Nov 22 2021 15:13:01 GMT+0900 (KST)

oplog last event time: Mon Nov 22 2021 15:42:54 GMT+0900 (KST)

now: Mon Nov 22 2021 15:43:03 GMT+0900 (KST)

쿼리 결과가 20건 이상일 경우

★깜장별★ — Mon, 4 Apr 2022 16:52:44 +0900

몽고 쿼리를 수행하게 되면 기본으로 20건만 가져오게 되는데, 이게 쉘 에서 돌려도 20건만 나옴

그래서 쉘 제일 위쪽에서 지정해 주고 수행

test.js

DBQuery.shellBatchSize = 1000;

DBQuery.prototype._prettyShell = true;

db.test.find()

mongo < test.js > test.out 하면 1000건이 저장 됨

sysbench

★깜장별★ — Mon, 4 Apr 2022 16:49:22 +0900

MySQL 부하 테스트 (MySQL, MariaDB, Percona)

설치

curl -s https://packagecloud.io/install/repositories/akopytov/sysbench/script.rpm.sh | sudo bash

sudo yum -y install sysbench

[root@OpenSource ~]# which sysbench

/usr/bin/sysbench

[root@OpenSource ~]# sysbench --version

sysbench 1.0.20

[root@OpenSource sysbench]# cd /usr/share/sysbench

[root@OpenSource sysbench]# ls

bulk_insert.lua oltp_delete.lua oltp_point_select.lua oltp_read_write.lua oltp_update_non_index.lua select_random_points.lua tests

oltp_common.lua oltp_insert.lua oltp_read_only.lua oltp_update_index.lua oltp_write_only.lua select_random_ranges.lua

[root@OpenSource ~]# alias | grep client

alias client_commu5.7='/engine/mysql/commu5.7/bin/mysql -uroot -p1 -S /data/mysql/commu5.7/mysql.sock'

alias client_commu8.0='/engine/mysql/commu8.0/bin/mysql -uroot -p1 -S /data/mysql/commu8.0/mysql.sock'

alias client_enter5.7='/engine/mysql/enter5.7/bin/mysql -uroot -p1 -S /data/mysql/enter5.7/mysql.sock'

alias client_enter8.0='/engine/mysql/enter8.0/bin/mysql -uroot -p1 -S /data/mysql/enter8.0/mysql.sock'

alias client_maria10.3='/engine/mariadb/maria10.3/bin/mysql -uroot -p1 -S /data/mariadb/maria10.3/mysql.sock'

alias client_maria10.5='/engine/mariadb/maria10.5/bin/mysql -uroot -p1 -S /data/mariadb/maria10.5/mysql.sock'

alias client_maria10.6='/engine/mariadb/maria10.6/bin/mysql -uroot -p1 -S /data/mariadb/maria10.6/mysql.sock'

alias client_maria10.7='/engine/mariadb/maria10.7/bin/mysql -uroot -p1 -S /data/mariadb/maria10.7/mysql.sock'

alias client_percona5.6='/engine/percona/percona5.6/bin/mysql -uroot -p1 -S /data/percona/percona5.6/mysql.sock'

alias client_percona5.7='/engine/percona/percona5.7/bin/mysql -uroot -p1 -S /data/percona/percona5.7/mysql.sock'

alias client_percona8.0='/engine/percona/percona8.0/bin/mysql -uroot -p1 -S /data/percona/percona8.0/mysql.sock'

유저 / DB 생성

MySQL 5.7 ~ 8.0

client_commu5.7 -e "create database sysbench; create user 'sysbench'@'%' identified WITH mysql_native_password by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_commu8.0 -e "create database sysbench; create user 'sysbench'@'%' identified WITH mysql_native_password by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_enter5.7 -e "create database sysbench; create user 'sysbench'@'%' identified WITH mysql_native_password by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_enter8.0 -e "create database sysbench; create user 'sysbench'@'%' identified WITH mysql_native_password by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_commu5.7 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_commu8.0 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_enter5.7 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_enter8.0 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

MariaDB 10.3 ~ 10.7

client_maria10.3 -e "create database sysbench; create user 'sysbench'@'%' identified by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_maria10.5 -e "create database sysbench; create user 'sysbench'@'%' identified by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_maria10.6 -e "create database sysbench; create user 'sysbench'@'%' identified by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_maria10.7 -e "create database sysbench; create user 'sysbench'@'%' identified by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_maria10.3 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_maria10.5 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_maria10.6 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_maria10.7 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

percona 5.6 ~ 8.0

client_percona5.6 -e "create database sysbench; create user 'sysbench'@'%' identified by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_percona5.7 -e "create database sysbench; create user 'sysbench'@'%' identified by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_percona8.0 -e "create database sysbench; create user 'sysbench'@'%' identified by 'sysbench'; grant all on sysbench.* to 'sysbench'@'%' ; flush privileges;"

client_percona5.6 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_percona5.7 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

client_percona8.0 -e "select user , host from mysql.user where user = 'sysbench' ; show databases ;"

Sysbench

1. 테스트 데이터 생성

sysbench /usr/share/sysbench/oltp_read_write.lua \
--time=20 --max-requests=0 \
--table_size=100000 --tables=15 --report-interval=1 \
--mysql-host=localhost --mysql-port=3304 --mysql-socket=/data/mysql/commu5.7/mysql.sock \
--db-driver=mysql \
--mysql-db=sysbench --mysql-user=sysbench --mysql-password=sysbench \
--threads=100 \
prepare

2. 수행

3. 테스트 데이터 삭제

YCSB (Yahoo! Cloud Serving Benchmark)

★깜장별★ — Mon, 4 Apr 2022 16:42:02 +0900

NoSQL 관련 다양한 DB에 성능테스트를 해 볼 수 있다.

다운로드 : https://github.com/brianfrankcooper/YCSB

하나의 데이터가 1K인 데이터를 이용하여 SELECT, INSERT, UPDATE 등을 테스트 할 수 있게 제공한다.

1. 설치

1. 자바가 설치 되어 있어야 한다. (기존에 설치 되어 있는 상태에서 진행을 해서, 꼭 필요한지는 확인 필요)

2. ycsb 파일을 다운 받는다.

wget https://github.com/brianfrankcooper/YCSB/releases/download/0.17.0/ycsb-0.17.0.tar.gz

tar xfvz ycsb-0.17.0.tar.gz

2. 사용 법

Load : 테스트 데이터를 대상 DB에 넣는다.

Run : 테스트 데이터를 이용하여 벤치 마크 수행

workloads 디렉터리 밑에 존재하는 템플릿 중에서 자신이 테스트 하고자 하는 상황에 맞는 파일을 선택하거나, 수정해서 사용 할 수 있다.

각 각의 파일들을 한번 씩 열어 보면 Read/Write 비율이 각각 다름을 확인 할 수 있다.

3. MongoDB 테스트

테스트 대상 몽고 DB 상황

Admin DB에 root / root1122 유저로 인증을 해야 하며, 포트는 9999 이다.

MongoDB 테스트를 위해서 mongodb-binding 디렉터리 밑에 README.md 파일을 보면 자세한 사용법이 나와 있다.

테스트 데이터 생성

[root@pgmongo ycsb-0.17.0]# ./bin/ycsb load mongodb -s -P workloads/workloada -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
java -cp /media/ycsb/ycsb-0.17.0/mongodb-binding/conf:/media/ycsb/ycsb-0.17.0/conf:/media/ycsb/ycsb-0.17.0/lib/core-0.17.0.jar:/media/ycsb/ycsb-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-core-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/HdrHistogram-2.1.4.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongo-java-driver-3.8.0.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-classic-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/slf4j-api-1.7.25.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-async-driver-2.0.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/snappy-java-1.1.7.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-core-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-binding-0.17.0.jar site.ycsb.Client -db site.ycsb.db.MongoDbClient -s -P workloads/workloada -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -load
Command line: -db site.ycsb.db.MongoDbClient -s -P workloads/workloada -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -load
YCSB Client 0.17.0

Loading workload...
Starting test.
2020-09-10 15:05:26:356 0 sec: 0 operations; est completion in 0 second
mongo client connection created with mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
2020-09-10 15:05:27:285 0 sec: 1000 operations; 1052.63 current ops/sec; [CLEANUP: Count=1, Max=1633, Min=1633, Avg=1633, 90=1633, 99=1633, 99.9=1633, 99.99=1633] [INSERT: Count=1000, Max=136703, Min=203, Avg=548.51, 90=666, 99=1211, 99.9=2923, 99.99=136703]
[OVERALL], RunTime(ms), 950
[OVERALL], Throughput(ops/sec), 1052.6315789473683
[TOTAL_GCS_PS_Scavenge], Count, 0
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 0
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.0
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 0
[TOTAL_GC_TIME], Time(ms), 0
[TOTAL_GC_TIME_%], Time(%), 0.0
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 1633.0
[CLEANUP], MinLatency(us), 1633
[CLEANUP], MaxLatency(us), 1633
[CLEANUP], 95thPercentileLatency(us), 1633
[CLEANUP], 99thPercentileLatency(us), 1633
[INSERT], Operations, 1000
[INSERT], AverageLatency(us), 548.506
[INSERT], MinLatency(us), 203
[INSERT], MaxLatency(us), 136703
[INSERT], 95thPercentileLatency(us), 816
[INSERT], 99thPercentileLatency(us), 1211
[INSERT], Return=OK, 1000

Read/Write 테스트

[root@pgmongo ycsb-0.17.0]# ./bin/ycsb run mongodb -s -P workloads/workloada -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
java -cp /media/ycsb/ycsb-0.17.0/mongodb-binding/conf:/media/ycsb/ycsb-0.17.0/conf:/media/ycsb/ycsb-0.17.0/lib/core-0.17.0.jar:/media/ycsb/ycsb-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-core-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/HdrHistogram-2.1.4.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongo-java-driver-3.8.0.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-classic-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/slf4j-api-1.7.25.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-async-driver-2.0.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/snappy-java-1.1.7.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-core-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-binding-0.17.0.jar site.ycsb.Client -db site.ycsb.db.MongoDbClient -s -P workloads/workloada -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -t
Command line: -db site.ycsb.db.MongoDbClient -s -P workloads/workloada -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -t
YCSB Client 0.17.0

Loading workload...
Starting test.
2020-09-10 15:10:02:204 0 sec: 0 operations; est completion in 0 second
mongo client connection created with mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
2020-09-10 15:10:03:232 1 sec: 1000 operations; 945.18 current ops/sec; [READ: Count=491, Max=141823, Min=264, Avg=795.13, 90=812, 99=1165, 99.9=141823, 99.99=141823] [CLEANUP: Count=1, Max=1614, Min=1614, Avg=1614, 90=1614, 99=1614, 99.9=1614, 99.99=1614] [UPDATE: Count=509, Max=9599, Min=269, Avg=514.3, 90=688, 99=1132, 99.9=1529, 99.99=9599]
[OVERALL], RunTime(ms), 1058
[OVERALL], Throughput(ops/sec), 945.179584120983
[TOTAL_GCS_PS_Scavenge], Count, 0
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 0
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.0
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 0
[TOTAL_GC_TIME], Time(ms), 0
[TOTAL_GC_TIME_%], Time(%), 0.0
[READ], Operations, 491
[READ], AverageLatency(us), 795.1262729124236
[READ], MinLatency(us), 264
[READ], MaxLatency(us), 141823
[READ], 95thPercentileLatency(us), 934
[READ], 99thPercentileLatency(us), 1165
[READ], Return=OK, 491
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 1614.0
[CLEANUP], MinLatency(us), 1614
[CLEANUP], MaxLatency(us), 1614
[CLEANUP], 95thPercentileLatency(us), 1614
[CLEANUP], 99thPercentileLatency(us), 1614
[UPDATE], Operations, 509
[UPDATE], AverageLatency(us), 514.2986247544204
[UPDATE], MinLatency(us), 269
[UPDATE], MaxLatency(us), 9599
[UPDATE], 95thPercentileLatency(us), 831
[UPDATE], 99thPercentileLatency(us), 1132
[UPDATE], Return=OK, 509

workloads/workloada 파일은 1000건의 데이터를 가지고 테스트를 하기에 너무 작아서 100,000 건으로 테스트 하기 위해서 파일을 수정해서 다시 시작 한다.

[root@pgmongo ycsb-0.17.0]# cd workloads/

[root@pgmongo workloads]# cp workloada workloada_100000

[root@pgmongo workloads]# vi workloada_100000

recordcount=100000

operationcount=100000

두 개의 값을 변경해 주고 저장

다시 테스트를 하기 위해서 몽고 DB에 접속해서 기존의 데이터 삭제

use ycsb

db.usertable.drop()

다시 데이터 만들고 테스트 진행

[root@pgmongo ycsb-0.17.0]# ./bin/ycsb load mongodb -s -P workloads/workloada_100000 -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
java -cp /media/ycsb/ycsb-0.17.0/mongodb-binding/conf:/media/ycsb/ycsb-0.17.0/conf:/media/ycsb/ycsb-0.17.0/lib/core-0.17.0.jar:/media/ycsb/ycsb-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-core-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/HdrHistogram-2.1.4.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongo-java-driver-3.8.0.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-classic-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/slf4j-api-1.7.25.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-async-driver-2.0.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/snappy-java-1.1.7.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-core-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-binding-0.17.0.jar site.ycsb.Client -db site.ycsb.db.MongoDbClient -s -P workloads/workloada_100000 -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -load
Command line: -db site.ycsb.db.MongoDbClient -s -P workloads/workloada_100000 -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -load
YCSB Client 0.17.0

Loading workload...
Starting test.
2020-09-10 15:16:19:871 0 sec: 0 operations; est completion in 106751991167300 days 15 hours
mongo client connection created with mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
2020-09-10 15:16:29:851 10 sec: 58140 operations; 5814 current ops/sec; est completion in 8 second [INSERT: Count=58173, Max=146175, Min=120, Avg=161.82, 90=189, 99=430, 99.9=859, 99.99=5167]
2020-09-10 15:16:35:711 15 sec: 100000 operations; 7140.91 current ops/sec; [CLEANUP: Count=1, Max=3025, Min=3024, Avg=3025, 90=3025, 99=3025, 99.9=3025, 99.99=3025] [INSERT: Count=41827, Max=7331, Min=117, Avg=137.75, 90=155, 99=195, 99.9=321, 99.99=700]
[OVERALL], RunTime(ms), 15863
[OVERALL], Throughput(ops/sec), 6303.977809998109
[TOTAL_GCS_PS_Scavenge], Count, 5
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 64
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.4034545798398789
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 5
[TOTAL_GC_TIME], Time(ms), 64
[TOTAL_GC_TIME_%], Time(%), 0.4034545798398789
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 3025.0
[CLEANUP], MinLatency(us), 3024
[CLEANUP], MaxLatency(us), 3025
[CLEANUP], 95thPercentileLatency(us), 3025
[CLEANUP], 99thPercentileLatency(us), 3025
[INSERT], Operations, 100000
[INSERT], AverageLatency(us), 151.75476
[INSERT], MinLatency(us), 117
[INSERT], MaxLatency(us), 146175
[INSERT], 95thPercentileLatency(us), 195
[INSERT], 99thPercentileLatency(us), 279
[INSERT], Return=OK, 100000

[root@pgmongo ycsb-0.17.0]# ./bin/ycsb run mongodb -s -P workloads/workloada_100000 -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
java -cp /media/ycsb/ycsb-0.17.0/mongodb-binding/conf:/media/ycsb/ycsb-0.17.0/conf:/media/ycsb/ycsb-0.17.0/lib/core-0.17.0.jar:/media/ycsb/ycsb-0.17.0/lib/htrace-core4-4.1.0-incubating.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-mapper-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/jackson-core-asl-1.9.4.jar:/media/ycsb/ycsb-0.17.0/lib/HdrHistogram-2.1.4.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongo-java-driver-3.8.0.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-classic-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/slf4j-api-1.7.25.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-async-driver-2.0.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/snappy-java-1.1.7.1.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/logback-core-1.1.2.jar:/media/ycsb/ycsb-0.17.0/mongodb-binding/lib/mongodb-binding-0.17.0.jar site.ycsb.Client -db site.ycsb.db.MongoDbClient -s -P workloads/workloada_100000 -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -t
Command line: -db site.ycsb.db.MongoDbClient -s -P workloads/workloada_100000 -p mongodb.url=mongodb://root:root1122@localhost:9999/ycsb?authSource=admin -t
YCSB Client 0.17.0

Loading workload...
Starting test.
2020-09-10 15:17:28:584 0 sec: 0 operations; est completion in 0 second
mongo client connection created with mongodb://root:root1122@localhost:9999/ycsb?authSource=admin
DBWrapper: report latency for each error is false and specific error codes to track for latency are: []
2020-09-10 15:17:38:555 10 sec: 49095 operations; 4909.5 current ops/sec; est completion in 11 seconds [READ: Count=24488, Max=39647, Min=137, Avg=182.8, 90=218, 99=548, 99.9=977, 99.99=6879] [UPDATE: Count=24638, Max=137855, Min=151, Avg=200.71, 90=228, 99=520, 99.9=1053, 99.99=8519]
2020-09-10 15:17:46:875 18 sec: 100000 operations; 6117.65 current ops/sec; [READ: Count=25465, Max=8471, Min=130, Avg=154.23, 90=172, 99=208, 99.9=287, 99.99=1328] [CLEANUP: Count=1, Max=2343, Min=2342, Avg=2343, 90=2343, 99=2343, 99.9=2343, 99.99=2343] [UPDATE: Count=25409, Max=3313, Min=144, Avg=168.14, 90=188, 99=227, 99.9=288, 99.99=576]
[OVERALL], RunTime(ms), 18321
[OVERALL], Throughput(ops/sec), 5458.217346214727
[TOTAL_GCS_PS_Scavenge], Count, 8
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 86
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.4694066917744664
[TOTAL_GCS_PS_MarkSweep], Count, 0
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 0
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 0.0
[TOTAL_GCs], Count, 8
[TOTAL_GC_TIME], Time(ms), 86
[TOTAL_GC_TIME_%], Time(%), 0.4694066917744664
[READ], Operations, 49953
[READ], AverageLatency(us), 168.23125738193903
[READ], MinLatency(us), 130
[READ], MaxLatency(us), 39647
[READ], 95thPercentileLatency(us), 219
[READ], 99thPercentileLatency(us), 371
[READ], Return=OK, 49953
[CLEANUP], Operations, 1
[CLEANUP], AverageLatency(us), 2343.0
[CLEANUP], MinLatency(us), 2342
[CLEANUP], MaxLatency(us), 2343
[CLEANUP], 95thPercentileLatency(us), 2343
[CLEANUP], 99thPercentileLatency(us), 2343
[UPDATE], Operations, 50047
[UPDATE], AverageLatency(us), 184.17291745758988
[UPDATE], MinLatency(us), 144
[UPDATE], MaxLatency(us), 137855
[UPDATE], 95thPercentileLatency(us), 230
[UPDATE], 99thPercentileLatency(us), 353
[UPDATE], Return=OK, 50047

MariaDB open_files_limit 값 변경

★깜장별★ — Wed, 30 Mar 2022 14:07:44 +0900

일반적으로는 my.cnf 파일에 적용한 후 재 기동 해 주면 변경이 되지만 , 간혹 systemctl(service)로 기동하는 경우 적용이 안 되는 경우가 있다.

이럴 경우 다음과 같이 설정해서 변경해 주면 된다.

1. O/S 에서 /etc/security/limits.conf 파일에서 파일 limit 제한 해제

*                -       nofile          65536

OR

*                hard    nofile          65536
*                soft nofile          65536

OR mysql 유저

mysql               hard    nofile          65536
mysql               soft nofile          65536

2. systemctl로 기동 될때 사용하는 파라미터 파일들에서 "LimitNOFILE" 값을 설정

읽어 들이는 파일 순서는 다음과 같다. (limits.conf 파일은 없으니 생성한 후에 거기다가 설정 하면 된다.)

그래서 먼저 읽어 들인 값을 사용하게 된다.

limits.conf 파일을 생성해서 사용 할거면 migrated-from-my.cnf-settings.conf 파일에 LimitNOFILE 값이 있는지 확인해야 함.

1. /etc/systemd/system/mariadb.service.d/migrated-from-my.cnf-settings.conf
2. /etc/systemd/system/mysqld.service.d/limits.conf
3. /usr/lib/systemd/system/mariadb.service
4. /usr/lib/systemd/system/mariadb.service
5. /etc/systemd/system/mysql.service
6. /etc/systemd/system/mysqld.service

/etc/systemd/system/mysqld.service.d/limits.conf 파일에 다음 추가

infinity는 자동으로 Max값을 넣는다.

[Service]
LimitNOFILE=infinity

3. 설정 reload 및 재 시작

systemctl daemon-reload

systemctl restart mariadb

MariaDB [(none)]> select @@open_files_limit ;

+--------------------------+

| @@open_files_limit |

+--------------------------+

| 16291 |

+--------------------------+

Percona MySQL Server Install (5.6~8.0)

★깜장별★ — Mon, 7 Mar 2022 09:50:20 +0900

우리팀 막내들을 위해서 만들었던 문서 공유 ~~

Cent7 Minimal에 설치 (7.9)

#######################

# Percona 5.6

#######################

Download : https://www.percona.com/downloads/Percona-Server-5.6/LATEST/

--> Linux Generic

--> Percona-Server-5.6.51-rel91.0-Linux.x86_64.ssl101.tar.gz

-- 패키지 설치

yum install -y zlib1g-dev gcc g++

yum install -y grub-pc-bin

yum install -y openssl

yum install -y libncurses5-dev libncursesw5-dev

yum install -y bzip2 libaio1 libaio-dev

yum install -y expect gzip vim

yum install -y perl perl-Data-Dumper -- 5.6 설치에 필수

-- 그룹 및 유저 생성

groupadd mysql

useradd -g mysql mysql

-- 설치 디렉터리 생성 및 권한 부여

mkdir -p /engine/

mkdir -p /data/5.6

mkdir -p /log/5.6

chown -R mysql:mysql /data

chown -R mysql:mysql /log

chown -R mysql:mysql /engine -- 선택

-- 설치 파일 압축 해제

tar xvf Percona-Server-5.6.51-rel91.0-Linux.x86_64.ssl101.tar.gz

mv Percona-Server-5.6.51-rel91.0-Linux.x86_64.ssl101 /engine/

cd /engine

mv Percona-Server-5.6.51-rel91.0-Linux.x86_64.ssl101 5.6

-- mysql config file 생성

vi /engine/5.6/my5.6.cnf

[mysql]

socket=/data/5.6/mysql.sock

port=3356

[mysqld]

basedir=/engine/5.6

datadir=/data/5.6

log-error=/log/5.6/error.log

pid-file=/data/5.6/mysql.pid

socket=/data/5.6/mysql.sock

slow_query_log_file=/log/5.6/MySQL-slow.log

port=3356

user=mysql

-- DB 생성

perl 및 perl-Data-Dumper가 설치 되어 있지 않으면 실행 안됨

/engine/5.6/scripts/mysql_install_db \

--defaults-file=/engine/5.6/my5.6.cnf \

--basedir=/engine/5.6 \

--datadir=/data/5.6

-- 파일 권한 다시 부여 (확인해 보고 mysql:mysql 이면 안 줘도 됨. 확인 귀찮으니 그냥 준다~~)

chown -R mysql:mysql /data

chown -R mysql:mysql /log

-- DB 기동

/engine/5.6/bin/mysqld --defaults-file=/engine/5.6/my5.6.cnf &

-- Root Password 설정

/engine/5.6/bin/mysqladmin -uroot password 'root1122' --socket=/data/5.6/mysql.sock

-- 접속 테스트

/engine/5.6/bin/mysql -uroot -p --socket=/data/5.6/mysql.sock

/engine/5.6/bin/mysql -h127.0.0.1 -P 3356 -uroot -p

-- 종료

/engine/5.6/bin/mysqladmin -uroot -p --socket=/data/5.6/mysql.sock shutdown

#######################

# Percona 5.7

#######################

Download : https://www.percona.com/downloads/Percona-Server-5.7/LATEST/

--> Linux Generic

--> Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz

-- 패키지 설치

yum install -y zlib1g-dev gcc g++

yum install -y grub-pc-bin

yum install -y openssl

yum install -y libncurses5-dev libncursesw5-dev

yum install -y bzip2 libaio1 libaio-dev

yum install -y expect gzip vim

yum install -y perl perl-Data-Dumper -- 5.6 설치에 필수

-- 그룹 및 유저 생성

groupadd mysql

useradd -g mysql mysql

-- 설치 디렉터리 생성 및 권한 부여

mkdir -p /engine/

mkdir -p /data/5.7

mkdir -p /log/5.7

chown -R mysql:mysql /data

chown -R mysql:mysql /log

chown -R mysql:mysql /engine -- 선택

-- 설치 파일 압축 해제

tar xvf Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12.tar.gz

mv Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12 /engine/

cd /engine

mv Percona-Server-5.7.36-39-Linux.x86_64.glibc2.12 5.7

-- mysql config file 생성

vi /engine/5.7/my5.7.cnf

[mysql]

socket=/data/5.7/mysql.sock

port=3357

[mysqld]

basedir=/engine/5.7

datadir=/data/5.7

log-error=/log/5.7/error.log

pid-file=/data/5.7/mysql.pid

socket=/data/5.7/mysql.sock

slow_query_log_file=/log/5.7/MySQL-slow.log

port=3357

user=mysql

-- DB 생성

/engine/5.7/bin/mysqld \

--defaults-file=/engine/5.7/my5.7.cnf \

--explicit_defaults_for_timestamp \

--initialize

-- 파일 권한 다시 부여 (확인해 보고 mysql:mysql 이면 안 줘도 됨. 확인 귀찮으니 그냥 준다~~)

chown -R mysql:mysql /data

chown -R mysql:mysql /log

-- DB 기동

/engine/5.7/bin/mysqld --defaults-file=/engine/5.7/my5.7.cnf &

-- Root Password 설정

임시 패스워드 확인

grep password /log/5.7/error.log

2022-03-02T14:37:53.993193Z 1 [Note] A temporary password is generated for root@localhost: MNhJq*/>1y.K

/engine/5.7/bin/mysql_secure_installation -h127.0.0.1 -P3357

--> 임시패스워드 입력 : MNhJq*/>1y.K

-- 접속 테스트

/engine/5.7/bin/mysql -uroot -p --socket=/data/5.7/mysql.sock

/engine/5.7/bin/mysql -h127.0.0.1 -P 3357 -uroot -p

-- 종료

/engine/5.7/bin/mysqladmin -uroot -p --socket=/data/5.7/mysql.sock shutdown

#######################

# Percona 8.0

#######################

Download : https://www.percona.com/downloads/Percona-Server-8.0/LATEST/

--> Linux Generic

--> Percona-Server-8.0.26-17-Linux.x86_64.glibc2.12.tar.gz

-- 패키지 설치

yum install -y zlib1g-dev gcc g++

yum install -y grub-pc-bin

yum install -y openssl

yum install -y libncurses5-dev libncursesw5-dev

yum install -y bzip2 libaio1 libaio-dev

yum install -y expect gzip vim

yum install -y perl perl-Data-Dumper -- 5.6 설치에 필수

-- 그룹 및 유저 생성

groupadd mysql

useradd -g mysql mysql

-- 설치 디렉터리 생성 및 권한 부여

mkdir -p /engine/

mkdir -p /data/8.0

mkdir -p /log/8.0

chown -R mysql:mysql /data

chown -R mysql:mysql /log

chown -R mysql:mysql /engine -- 선택

-- 설치 파일 압축 해제

tar xvf Percona-Server-8.0.26-17-Linux.x86_64.glibc2.12.tar.gz

mv Percona-Server-8.0.26-17-Linux.x86_64.glibc2.12 /engine/

cd /engine

mv Percona-Server-8.0.26-17-Linux.x86_64.glibc2.12 8.0

-- mysql config file 생성

vi /engine/8.0/my8.0.cnf

[mysql]

socket=/data/8.0/mysql.sock

port=3380

[mysqld]

basedir=/engine/8.0

datadir=/data/8.0

log-error=/log/8.0/error.log

pid-file=/data/8.0/mysql.pid

socket=/data/8.0/mysql.sock

slow_query_log_file=/log/8.0/MySQL-slow.log

port=3380

user=mysql

-- DB 생성

/engine/8.0/bin/mysqld \

--defaults-file=/engine/8.0/my8.0.cnf \

--explicit_defaults_for_timestamp \

--initialize

-- 파일 권한 다시 부여 (확인해 보고 mysql:mysql 이면 안 줘도 됨. 확인 귀찮으니 그냥 준다~~)

chown -R mysql:mysql /data

chown -R mysql:mysql /log

-- DB 기동

/engine/8.0/bin/mysqld --defaults-file=/engine/8.0/my8.0.cnf &

-- Root Password 설정

임시 패스워드 확인

grep password /log/8.0/error.log

2022-03-02T15:17:38.702030Z 6 [Note] [MY-010454] [Server] A temporary password is generated for root@localhost: %)a4Pw876y>w

/engine/8.0/bin/mysql_secure_installation -h127.0.0.1 -P3380

--> 임시패스워드 입력 :

-- 접속 테스트

/engine/8.0/bin/mysql -uroot -p --socket=/data/8.0/mysql.sock

/engine/8.0/bin/mysql -h127.0.0.1 -P 3380 -uroot -p

-- 종료

/engine/8.0/bin/mysqladmin -uroot -p --socket=/data/8.0/mysql.sock shutdown

############################

# alias 등록

############################

vi /root/.bash_profile

alias start5.6='/engine/5.6/bin/mysqld --defaults-file=/engine/5.6/my5.6.cnf & '

alias stop5.6='/engine/5.6/bin/mysqladmin -uroot -p --socket=/data/5.6/mysql.sock shutdown'

alias mysql5.6='/engine/5.6/bin/mysql -h 127.0.0.1 -P 3356'

alias root5.6='/engine/5.6/bin/mysql -h 127.0.0.1 -P 3356 -u root -proot1122'

alias start5.7='/engine/5.7/bin/mysqld --defaults-file=/engine/5.7/my5.7.cnf & '

alias stop5.7='/engine/5.7/bin/mysqladmin -uroot -p --socket=/data/5.7/mysql.sock shutdown'

alias mysql5.7='/engine/5.7/bin/mysql -h 127.0.0.1 -P 3357'

alias root5.7='/engine/5.7/bin/mysql -h 127.0.0.1 -P 3357 -u root -proot1122'

alias start8.0='/engine/8.0/bin/mysqld --defaults-file=/engine/8.0/my8.0.cnf & '

alias stop8.0='/engine/8.0/bin/mysqladmin -uroot -p --socket=/data/8.0/mysql.sock shutdown'

alias mysql8.0='/engine/8.0/bin/mysql -h 127.0.0.1 -P 3380'

alias root8.0='/engine/8.0/bin/mysql -h 127.0.0.1 -P 3380 -u root -proot1122'

echo "

===========

Percona :

5.6 : start5.6 / stop5.6 / mysql5.6 / root5.6 / 3356

5.7 : start5.7 / stop5.7 / mysql5.7 / root5.7 / 3357

8.0 : start8.0 / stop8.0 / mysql8.0 / root8.0 / 3380

Directory :

Software : /engine

Data : /data

Logs : /log

==========

[MariaDB] Xpand + MaxScale #6

★깜장별★ — Fri, 4 Mar 2022 19:46:47 +0900

노드 장애 테스트

현재 MaxScale (1대) + Xpand Node(4대)로 구성된 상태에서 , HammerDB를 이용하여 tpcc 데이터를 만들고,

HammerDB를 이용하여 40세션을 붙여서 부하를 주고 있는 상황에서 , xpand3 Server가 Down되는 시나리오.

시나리오에서는 xpand3 서버를 종료 했다가, 일정 시간이 지난 후 다시 기동함.

만약 진짜 xpand3 서버의 O/S가 문제가 있어서 xpand node를 신규로 구성해야 한다면 "[MariaDB] Xpand + MaxScale #5" 포스팅을 확인 하여 추가를 시켜주면 된다.

현 상태 확인

-- maxscale 에서 확인

40개 세션이 xpand1 ~ xpand4까지 고르게 분산 되어 있음을 확인 할 수 있다.

-- Xpand Node에서 확인

mysql --user=root --password --no-auto-rehash
use system

MySQL [system]> select nodeid , session_id , source_ip , user , trx_state , statement_state , substr(last_statement,1,20)
    -> from sessions where database = 'tpcc' ;
+--------+------------+--------------+---------------------+-----------+-----------------+-----------------------------+
| nodeid | session_id | source_ip    | user                | trx_state | statement_state | substr(last_statement,1,20) |
+--------+------------+--------------+---------------------+-----------+-----------------+-----------------------------+
|      1 |      10241 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      1 |      19457 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      1 |      11265 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      1 |      15361 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      1 |      13313 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      1 |      14337 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      1 |      12289 | 10.70.101.20 | 7069997881645258753 | open      | executing       | SELECT i_price, i_na        |
|      1 |      16385 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      1 |       9217 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @os_c_id,@os_        |
|      1 |      18433 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      4 |      30724 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @os_c_id,@os_        |
|      4 |      28676 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      4 |      23556 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      4 |      29700 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      4 |      31748 | 10.70.101.20 | 7069997881645258753 | open      | finished        | CALL SLEV(7,2,16)           |
|      4 |      25604 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      4 |      27652 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @os_c_id,@os_        |
|      4 |      26628 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      4 |      32772 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      4 |      24580 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      3 |      21507 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      3 |      19459 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      3 |      24579 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      3 |      22531 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      3 |      17411 | 10.70.101.20 | 7069997881645258753 | open      | finished        | set @p_c_id = 2758,         |
|      3 |      23555 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @os_c_id,@os_        |
|      2 |      75778 | 10.70.101.20 | 7069997881645258753 | open      | executing       | SET no_s_quantity =         |
|      3 |      16387 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      2 |      69634 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      3 |      25603 | 10.70.101.20 | 7069997881645258753 | open      | finished        | CALL DELIVERY(7,9,20        |
|      2 |      76802 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      3 |      20483 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      2 |      71682 | 10.70.101.20 | 7069997881645258753 | open      | finished        | CALL DELIVERY(9,2,20        |
|      3 |      18435 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      2 |      77826 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      2 |      73730 | 10.70.101.20 | 7069997881645258753 | open      | finished        | CALL SLEV(4,2,18)           |
|      2 |      74754 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @p_c_id,@p_c_        |
|      2 |      68610 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      2 |      70658 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
|      2 |      72706 | 10.70.101.20 | 7069997881645258753 | open      | finished        | select @disc,@last,@        |
+--------+------------+--------------+---------------------+-----------+-----------------+-----------------------------+
40 rows in set (0.00 sec)

MySQL [system]> select username , host , user from users ;
+------------------+--------------+---------------------+
| username         | host         | user                |
+------------------+--------------+---------------------+
| app              | %            | 7069997881645258753 |
| clx_maint        | 127.0.0.1    |                4110 |
| clx_view_definer | 127.0.0.1    |                4112 |
| max              | 10.70.101.20 | 7068584483821799425 |
| mysql_slave      |              |                4100 |
| root             | %            | 7069555885833806849 |
| root             | 127.0.0.1    |                4099 |
| xpand            | localhost    | 7068526298150719489 |
| xpandm           | localhost    | 7068526299347743745 |
+------------------+--------------+---------------------+
9 rows in set (0.00 sec)

MySQL [system]> select main.`table` , main.`index` , main.slice , main.nodeid , main.rank , repl.nodeid as replica_node , repl.rank as replca_rank , main.bytes , (main.bytes + repl.bytes)/1024/1024 as slice_sum_mb
    -> from   (select * from table_replicas where database = 'tpcc' and rank = 0 ) main ,
    ->        (select * from table_replicas where database = 'tpcc' and rank = 1 ) repl
    -> where main.`table` = repl.`table`
    -> and    main.`index` = repl.`index`
    -> and    main.slice = repl.slice
    -> order by main.`table`, main.`index` , main.nodeid ;
+------------+---------------------------+---------------------+--------+------+--------------+-------------+-----------+--------------+
| table      | index                     | slice               | nodeid | rank | replica_node | replca_rank | bytes     | slice_sum_mb |
+------------+---------------------------+---------------------+--------+------+--------------+-------------+-----------+--------------+
| customer   | c_w_id                    | 7070001415015847937 |      1 |    0 |            4 |           1 |   8814592 | 16.93750000 |
| customer   | c_w_id                    | 7070001415015848961 |      2 |    0 |            3 |           1 |   6225920 | 16.96875000 |
| customer   | c_w_id                    | 7070001415015851009 |      3 |    0 |            2 |           1 | 10354688 | 15.31250000 |
| customer   | c_w_id                    | 7070001415015849985 |      4 |    0 |            1 |           1 |   9994240 | 19.06250000 |
| customer   | __idx_customer__PRIMARY   | 7070001415015835649 |      1 |    0 |            3 |           1 | 37027840 | 71.65625000 |
| customer   | __idx_customer__PRIMARY   | 7070001415015836673 |      2 |    0 |            4 |           1 | 53510144 | 109.81250000 |
| customer   | __idx_customer__PRIMARY   | 7070001415015837697 |      3 |    0 |            1 |           1 | 40271872 | 76.75000000 |
| customer   | __idx_customer__PRIMARY   | 7070001415015834625 |      4 |    0 |            2 |           1 | 58720256 | 112.00000000 |
| district   | __idx_district__PRIMARY   | 7070001415239200769 |      1 |    0 |            3 |           1 |     65536 |   0.12500000 |
| district   | __idx_district__PRIMARY   | 7070001415239199745 |      2 |    0 |            4 |           1 |     65536 |   0.12500000 |
| district   | __idx_district__PRIMARY   | 7070001415239202817 |      3 |    0 |            1 |           1 |     65536 |   0.12500000 |
| district   | __idx_district__PRIMARY   | 7070001415239201793 |      4 |    0 |            2 |           1 |     65536 |   0.12500000 |
| history    | __base_history            | 7070001415325110273 |      1 |    0 |            4 |           1 |   7798784 | 15.06250000 |
| history    | __base_history            | 7070001415325107201 |      2 |    0 |            3 |           1 |   7766016 | 20.65625000 |
| history    | __base_history            | 7070001415325109249 |      3 |    0 |            2 |           1 | 13828096 | 20.62500000 |
| history    | __base_history            | 7070001415325108225 |      4 |    0 |            1 |           1 |   7995392 | 15.00000000 |
| item       | __idx_item__PRIMARY       | 7070001415402424321 |      1 |    0 |            2 |           1 |   5013504 |   7.43750000 |
| item       | __idx_item__PRIMARY       | 7070001415402426369 |      2 |    0 |            1 |           1 |   2785280 |   7.43750000 |
| item       | __idx_item__PRIMARY       | 7070001415402425345 |      3 |    0 |            4 |           1 |   5144576 |   7.56250000 |
| item       | __idx_item__PRIMARY       | 7070001415402427393 |      4 |    0 |            3 |           1 |   2752512 |   7.50000000 |
| new_order | __idx_new_order__PRIMARY | 7070001415479744513 |      1 |    0 |            2 |           1 |   2260992 |   3.75000000 |
| new_order | __idx_new_order__PRIMARY | 7070001415479742465 |      2 |    0 |            1 |           1 |   1114112 |   2.75000000 |
| new_order | __idx_new_order__PRIMARY | 7070001415479743489 |      3 |    0 |            4 |           1 |   4390912 |   6.93750000 |
| new_order | __idx_new_order__PRIMARY | 7070001415479741441 |      4 |    0 |            3 |           1 |   2195456 |   5.21875000 |
| orders     | o_w_id                    | 7070001415557074945 |      1 |    0 |            3 |           1 |   7110656 | 14.09375000 |
| orders     | o_w_id                    | 7070001415557071873 |      2 |    0 |            4 |           1 |   6356992 | 12.12500000 |
| orders     | o_w_id                    | 7070001415557072897 |      3 |    0 |            1 |           1 |   7143424 | 13.21875000 |
| orders     | o_w_id                    | 7070001415557073921 |      4 |    0 |            2 |           1 |   8388608 | 16.00000000 |
| orders     | __idx_orders__PRIMARY     | 7070001415557059585 |      1 |    0 |            3 |           1 |   7733248 | 15.25000000 |
| orders     | __idx_orders__PRIMARY     | 7070001415557058561 |      2 |    0 |            4 |           1 |   8093696 | 15.43750000 |
| orders     | __idx_orders__PRIMARY     | 7070001415557061633 |      3 |    0 |            1 |           1 |   9732096 | 17.93750000 |
| orders     | __idx_orders__PRIMARY     | 7070001415557060609 |      4 |    0 |            2 |           1 |   9863168 | 18.81250000 |
| order_line | __idx_order_line__PRIMARY | 7070001415647263745 |      1 |    0 |            4 |           1 | 48889856 | 93.34375000 |
| order_line | __idx_order_line__PRIMARY | 7070001415647260673 |      2 |    0 |            3 |           1 | 73465856 | 140.12500000 |
| order_line | __idx_order_line__PRIMARY | 7070001415647262721 |      3 |    0 |            2 |           1 | 72515584 | 139.06250000 |
| order_line | __idx_order_line__PRIMARY | 7070001415647261697 |      4 |    0 |            1 |           1 | 46202880 | 88.12500000 |
| stock      | __idx_stock__PRIMARY      | 7070001415724580865 |      1 |    0 |            3 |           1 | 73629696 | 140.43750000 |
| stock      | __idx_stock__PRIMARY      | 7070001415724579841 |      2 |    0 |            4 |           1 | 107806720 | 221.62500000 |
| stock      | __idx_stock__PRIMARY      | 7070001415724578817 |      3 |    0 |            1 |           1 | 77561856 | 147.93750000 |
| stock      | __idx_stock__PRIMARY      | 7070001415724577793 |      4 |    0 |            2 |           1 | 99418112 | 195.65625000 |
| warehouse | __idx_warehouse__PRIMARY | 7070001415810488321 |      1 |    0 |            3 |           1 |     65536 |   0.12500000 |
| warehouse | __idx_warehouse__PRIMARY | 7070001415810487297 |      2 |    0 |            4 |           1 |     65536 |   0.12500000 |
| warehouse | __idx_warehouse__PRIMARY | 7070001415810486273 |      3 |    0 |            1 |           1 |     65536 |   0.12500000 |
| warehouse | __idx_warehouse__PRIMARY | 7070001415810485249 |      4 |    0 |            2 |           1 |     65536 |   0.12500000 |
+------------+---------------------------+---------------------+--------+------+--------------+-------------+-----------+--------------+

Xpand3 Server down

HammerDB로 부하를 주고 있는 상황에서 xpand3 System 종료 진행

-- maxscale 에서 확인

xpand3 서버 다운 발생

-- xpand node에서 확인

다운 전

[xpand@xpand2 ~]$ clx space
nid | Hostname | Status |       Undo      |       Perm      |    Temp    |       Used      | DB Total | FS Free
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
1 |   xpand1 |    OK | 512.0K (0.00%) | 393.1M (2.58%) | 0 (0.00%) | 394.0M (2.59%) |    14.9G |    3.8G
2 |   xpand2 |    OK | 864.0K (0.01%) | 561.1M (3.68%) | 0 (0.00%) | 562.5M (3.69%) |    14.9G |    3.8G
3 |   xpand3 |    OK | 640.0K (0.00%) | 443.5M (2.91%) | 0 (0.00%) | 444.6M (2.92%) |    14.9G |    3.8G
4 |   xpand4 |    OK | 800.0K (0.01%) | 503.3M (3.30%) | 0 (0.00%) | 504.6M (3.31%) |    14.9G |    3.8G
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
                             2.8M (0.00%) |    1.9G (3.12%) | 0 (0.00%) |    1.9G (3.13%) |    59.5G |   15.2G

다운 후

[xpand@xpand2 ~]$ clx status
Cluster Name:    cl90c9aff68a607030
Cluster Version: 5.3.14
Cluster Status:   ATTN
Cluster Size:    4 nodes - 8 CPUs per Node
Current Node:    xpand2 - nid 2

nid | Hostname | Status |   IP Address | Zone | TPS |       Used      | Total
----+----------+---------+---------------+------+------+-----------------+-------
1 |   xpand1 |     OK | 10.70.101.16 |    0 | 123 | 432.3M (2.84%) | 14.9G
2 |   xpand2 |     OK | 10.70.101.17 |    0 |   74 | 582.0M (3.82%) | 14.9G
3 |   xpand3 |   DOWN | 10.70.101.18 |    0 |    0 |       0 (0.00%) |      0
4 |   xpand4 |     OK | 10.70.101.19 |    0 |   51 | 546.5M (3.59%) | 14.9G
----+----------+---------+---------------+------+------+-----------------+-------
                                                   248 |    1.5G (3.42%) | 44.6G

[xpand@xpand2 ~]$ clx space
nid | Hostname | Status |       Undo      |       Perm      |      Temp      |       Used      | DB Total | FS Free
----+----------+---------+-----------------+-----------------+----------------+-----------------+----------+--------
1 |   xpand1 |     OK | 448.0K (0.00%) | 438.4M (2.88%) |      0 (0.00%) | 439.3M (2.88%) |    14.9G |    3.8G
2 |   xpand2 |     OK |    4.2M (0.03%) | 647.7M (4.25%) |      0 (0.00%) | 652.4M (4.28%) |    14.9G |    3.8G
3 |   xpand3 |   DOWN |                 |                 |                |                 |          |
4 |   xpand4 |     OK | 864.0K (0.01%) | 544.1M (3.57%) | 32.0K (0.00%) | 545.4M (3.58%) |    14.9G |    3.8G
----+----------+---------+-----------------+-----------------+----------------+-----------------+----------+--------
                              5.4M (0.01%) |    1.6G (3.57%) | 32.0K (0.00%) |    1.6G (3.58%) |    44.6G |   11.4G

xpand1 , xpand2 , xpand4의 사용량이 증가 하고 있음을 확인 할 수 있다.

-- 쿼리로 확인

mysql --user=root --password --no-auto-rehash
use system

MySQL [system]> select nodeid , hostname , iface_ip , mysql_port from nodeinfo ;
+--------+----------+--------------+------------+
| nodeid | hostname | iface_ip     | mysql_port |
+--------+----------+--------------+------------+
|      1 | xpand1   | 10.70.101.16 |       3306 |
|      2 | xpand2   | 10.70.101.17 |       3306 |
|      4 | xpand4   | 10.70.101.19 |       3306 |
+--------+----------+--------------+------------+

서버 종료 되고 초기에는 xpand3이 가지고 있던 데이터가 사라진 상태이기 때문에 replica가 없는 슬라이스 들이 존재 한다.

MySQL [system]> select * from table_replicas where `index` = 'c_w_id' ;
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| Database | Table    | Index | slice               | replica             | nodeid | rank | status | bytes   |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070001415015841793 |      4 |    0 |      1 | 9994240 |
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070001415015842817 |      1 |    1 |      1 | 9994240 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070001415015846913 |      1 |    0 |      1 | 8814592 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070001415015845889 |      4 |    1 |      1 | 8945664 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070001415015840769 |      2 |    1 |      1 | 5701632 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070001415015844865 |      2 |    0 |      1 | 6225920 |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+

7070001415015849985   node4 node1
7070001415015847937   node1 node4
7070001415015851009 node2
7070001415015848961   node2

위의 쿼리를 보면 슬라이스 7070001415015851009, 7070001415015848961 는 아직 replica가 없는 1개씩 만 존재 하는 슬라이스들이다.
조금 시간이 지난 후 다시 쿼리를 해 보면

MySQL [system]> select * from table_replicas where `index` = 'c_w_id' ;
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| Database | Table    | Index | slice               | replica             | nodeid | rank | status | bytes   |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070001415015841793 |      4 |    0 |      1 | 9994240 |
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070001415015842817 |      1 |    1 |      1 | 9994240 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070001415015846913 |      1 |    0 |      1 | 8814592 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070001415015845889 |      4 |    1 |      1 | 8945664 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070001415015840769 |      2 |    1 |      1 | 5701632 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070045848110571524 |      4 |    2 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070045791243413508 |      1 |    0 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070001415015844865 |      2 |    1 |      1 | 6225920 |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+

7070001415015849985 node4 node1
7070001415015847937 node1 node4
7070001415015851009 node2 node4
7070001415015848961 node1 node2

MySQL [system]> select * from rebalancer_replicas where slice in ('7070001415015851009' , '7070001415015848961');
+---------------------+--------+-------+------+---------+
| slice               | nodeid | devid | rank | bytes   |
+---------------------+--------+-------+------+---------+
| 7070001415015851009 |      2 |     1 |    0 | 5701632 |
| 7070001415015851009 |      4 |     1 |    1 | 2719744 |
| 7070001415015848961 |      1 |     1 |    0 | 2719744 |
| 7070001415015848961 |      2 |     1 |    1 | 6225920 |
+---------------------+--------+-------+------+---------+

MySQL [system]> select *
    -> from   rebalancer_activity_log
    -> where representation = 'c_w_id'
    -> order by started desc ;
+---------------------+------+----------------+----------+----------+----------------+---------+-----------------------+-----------------------+-------+
| id                  | op   | reason         | database | relation | representation | bytes   | started               | finished              | error |
+---------------------+------+----------------+----------+----------+----------------+---------+-----------------------+-----------------------+-------+
| 7070051227670267905 | move | load imbalance | tpcc     | customer | c_w_id         | 8814592 | 2022-03-01 08:49:38.0 | 2022-03-01 08:49:43.6 | NULL |
| 7070045847554371586 | copy | reprotect      | tpcc     | customer | c_w_id         | 5701632 | 2022-03-01 08:28:45.3 | 2022-03-01 08:28:46.4 | NULL |
| 7070045790321948674 | copy | reprotect      | tpcc     | customer | c_w_id         | 6225920 | 2022-03-01 08:28:32.0 | 2022-03-01 08:28:32.9 | NULL |
+---------------------+------+----------------+----------+----------+----------------+---------+-----------------------+-----------------------+-------+

위의 결과 처럼 기존에 4개 슬라이스 8개 replica가 존재하던 것을 3개 서버로 나눠야 하니, 3개+2개로 서버가 데이터를 나눠 가지게 된다.
node1, node4는 3개의 슬라이스, node2는 2개의 슬라이스 씩 데이터를 나눠 가짐

현재 더 진행 중인 작업이 있는지 확인

MySQL [system]> select *
    -> from   rebalancer_activity_log
    -> where started > date_add(now() , interval -5 minute)
    -> order by started desc ;
+---------------------+------+----------------+----------+----------+----------------------+----------+-----------------------+-----------------------+-------+
| id                  | op   | reason         | database | relation | representation       | bytes    | started               | finished              | error |
+---------------------+------+----------------+----------+----------+----------------------+----------+-----------------------+-----------------------+-------+
| 7070054448998790145 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 76513280 | 2022-03-01 09:02:08.0 | NULL                  | NULL |
| 7070054320144704513 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 76283904 | 2022-03-01 09:01:38.0 | 2022-03-01 09:01:45.6 | NULL |
| 7070054191288836097 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 75759616 | 2022-03-01 09:01:08.0 | 2022-03-01 09:01:17.2 | NULL |
| 7070054062442878977 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 75431936 | 2022-03-01 09:00:38.0 | 2022-03-01 09:00:44.4 | NULL |
| 7070053933584769025 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 74907648 | 2022-03-01 09:00:08.0 | 2022-03-01 09:00:15.9 | NULL |
| 7070053804737675265 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 74579968 | 2022-03-01 08:59:38.0 | 2022-03-01 08:59:44.9 | NULL |
| 7070053675881125889 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 74317824 | 2022-03-01 08:59:08.0 | 2022-03-01 08:59:15.4 | NULL |
| 7070053547022120961 | move | load imbalance | tpcc     | stock    | __idx_stock__PRIMARY | 73990144 | 2022-03-01 08:58:38.0 | 2022-03-01 08:58:45.3 | NULL |
+---------------------+------+----------------+----------+----------+----------------------+----------+-----------------------+-----------------------+-------+

아직도 계속 진행 중인 것을 확인 할 수 있다.

[xpand@xpand2 ~]$ clx space
nid | Hostname | Status |       Undo      |       Perm      |    Temp    |       Used      | DB Total | FS Free
----+----------+---------+-----------------+-----------------+------------+-----------------+----------+--------
1 |   xpand1 |     OK | 576.0K (0.00%) | 691.3M (4.54%) | 0 (0.00%) | 692.3M (4.55%) |    14.9G |    3.8G
2 |   xpand2 |     OK | 800.0K (0.01%) | 696.3M (4.57%) | 0 (0.00%) | 697.6M (4.58%) |    14.9G |    3.8G
3 |   xpand3 |   DOWN |                 |                 |            |                 |          |
4 |   xpand4 |     OK | 672.0K (0.00%) | 696.7M (4.57%) | 0 (0.00%) | 697.8M (4.58%) |    14.9G |    3.8G
----+----------+---------+-----------------+-----------------+------------+-----------------+----------+--------
                              2.0M (0.00%) |    2.0G (4.56%) | 0 (0.00%) |    2.0G (4.57%) |    44.6G |   11.4G

Xpand3 Server 기동

HammerDB는 아직도 계속해서 부하 주는 중 ...

-- xpand3 에서 확인

[xpand@xpand3 ~]$ clx status
Cluster Name:    cl90c9aff68a607030
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    4 nodes - 8 CPUs per Node
Current Node:    xpand3 - nid 3

nid | Hostname | Status |   IP Address | TPS |       Used      | Total
----+----------+--------+---------------+-----+-----------------+-------
1 |   xpand1 |    OK | 10.70.101.16 |   0 | 521.8M (3.43%) | 14.9G
2 |   xpand2 |    OK | 10.70.101.17 |   0 | 631.3M (4.14%) | 14.9G
3 |   xpand3 |    OK | 10.70.101.18 |   0 | 454.3M (2.98%) | 14.9G
4 |   xpand4 |    OK | 10.70.101.19 |   0 | 681.2M (4.47%) | 14.9G
----+----------+--------+---------------+-----+-----------------+-------
                                            0 |    2.2G (3.76%) | 59.5G

mysql --user=root --password --no-auto-rehash
use system

MySQL [system]> select nodeid , hostname , iface_ip , mysql_port from nodeinfo ;
+--------+----------+--------------+------------+
| nodeid | hostname | iface_ip     | mysql_port |
+--------+----------+--------------+------------+
|      3 | xpand3   | 10.70.101.18 |       3306 |
|      1 | xpand1   | 10.70.101.16 |       3306 |
|      4 | xpand4   | 10.70.101.19 |       3306 |
|      2 | xpand2   | 10.70.101.17 |       3306 |
+--------+----------+--------------+------------+

-- maxscale에서 확인

-- 리발란서 체크

MySQL [system]> select *
    -> from   rebalancer_activity_log
    -> where started > date_add(now() , interval -5 minute)
    -> order by started desc ;
+---------------------+--------+-----------------------------+----------------+-----------------+--------------------------------+----------+-----------------------+-----------------------+-------------------+
| id                  | op     | reason                      | database       | relation        | representation                 | bytes    | started               | finished              | error             |
+---------------------+--------+-----------------------------+----------------+-----------------+--------------------------------+----------+-----------------------+-----------------------+-------------------+
| 7070057224240590852 | move   | load imbalance              | clustrix_statd | hotness_history | __idx_hotness_history__PRIMARY |   688128 | 2022-03-01 09:12:54.2 | 2022-03-01 09:12:59.4 | NULL              |
| 7070057201853034500 | move   | load imbalance              | clustrix_statd | hotness_history | timestamp_2                    |   557056 | 2022-03-01 09:12:49.0 | 2022-03-01 09:12:54.2 | NULL              |
| 7070057178865233922 | move   | distribution load imbalance | clustrix_statd | statd_history   | __idx_statd_history__PRIMARY   | 1310720 | 2022-03-01 09:12:43.6 | 2022-03-01 09:12:48.9 | NULL              |
| 7070057178855275521 | move   | distribution load imbalance | tpcc           | new_order       | __idx_new_order__PRIMARY       | 2785280 | 2022-03-01 09:12:43.6 | 2022-03-01 09:12:48.9 | NULL              |
| 7070057123086937089 | move   | load imbalance              | tpcc           | new_order       | __idx_new_order__PRIMARY       | 2785280 | 2022-03-01 09:12:30.6 | 2022-03-01 09:12:36.0 | NULL              |
| 7070057098480262145 | move   | load imbalance              | clustrix_statd | qpc_history     | __idx_qpc_history__PRIMARY     | 4161536 | 2022-03-01 09:12:24.9 | 2022-03-01 09:12:30.6 | NULL              |
| 7070057076009735172 | move   | distribution load imbalance | tpcc           | new_order       | __idx_new_order__PRIMARY       | 1015808 | 2022-03-01 09:12:19.7 | 2022-03-01 09:12:24.9 | NULL              |
| 7070057050019202052 | move   | distribution load imbalance | clustrix_statd | qpc_history     | __idx_qpc_history__PRIMARY     | 4227072 | 2022-03-01 09:12:13.6 | 2022-03-01 09:12:19.6 | NULL              |
| 7070056944761846787 | move   | load imbalance              | tpcc           | item            | __idx_item__PRIMARY            | 1966080 | 2022-03-01 09:11:49.1 | 2022-03-01 09:11:49.1 | No source replica |
| 7070056921902192643 | move   | distribution load imbalance | tpcc           | item            | __idx_item__PRIMARY            | 1966080 | 2022-03-01 09:11:43.8 | 2022-03-01 09:11:49.1 | NULL              |
| 7070056921474142209 | move   | distribution load imbalance | tpcc           | item            | __idx_item__PRIMARY            | 1998848 | 2022-03-01 09:11:43.7 | 2022-03-01 09:11:49.0 | NULL              |
| 7070056921366020098 | rerank | read imbalance              | tpcc           | history         | __base_history                 | 9732096 | 2022-03-01 09:11:43.7 | 2022-03-01 09:11:43.7 | NULL              |
| 7070056921645957121 | rerank | read imbalance              | tpcc           | orders          | o_w_id                         | 8159232 | 2022-03-01 09:11:43.7 | 2022-03-01 09:11:43.8 | NULL              |
| 7070056921130296324 | rerank | distribution read imbalance | tpcc           | customer        | __idx_customer__PRIMARY        | 54919168 | 2022-03-01 09:11:43.6 | 2022-03-01 09:11:43.7 | NULL              |
| 7070056921130888193 | rerank | read imbalance              | tpcc           | orders          | __idx_orders__PRIMARY          | 10485760 | 2022-03-01 09:11:43.6 | 2022-03-01 09:11:43.7 | NULL              |
| 7070056792276204547 | move   | load imbalance              | clustrix_statd | statd_history   | timestamp                      |   983040 | 2022-03-01 09:11:13.6 | 2022-03-01 09:11:19.0 | NULL              |
| 7070056792301891588 | move   | distribution load imbalance | tpcc           | item            | __idx_item__PRIMARY            | 1966080 | 2022-03-01 09:11:13.6 | 2022-03-01 09:11:19.1 | NULL              |
| 7070056688169977858 | move   | distribution load imbalance | clustrix_statd | statd_history   | timestamp                      | 1179648 | 2022-03-01 09:10:49.4 | 2022-03-01 09:10:54.6 | NULL              |
| 7070056663427659780 | move   | distribution load imbalance | tpcc           | customer        | c_w_id                         | 9994240 | 2022-03-01 09:10:43.6 | 2022-03-01 09:10:49.4 | NULL              |
| 7070056663446284289 | move   | load imbalance              | clustrix_statd | statd_history   | __idx_statd_history__PRIMARY   | 2719744 | 2022-03-01 09:10:43.6 | 2022-03-01 09:10:49.1 | NULL              |
| 7070056559896604674 | move   | distribution load imbalance | clustrix_statd | statd_history   | __idx_statd_history__PRIMARY   | 2686976 | 2022-03-01 09:10:19.5 | 2022-03-01 09:10:25.0 | NULL              |
| 7070056534583032836 | move   | distribution load imbalance | tpcc           | orders          | o_w_id                         | 6619136 | 2022-03-01 09:10:13.6 | 2022-03-01 09:10:19.5 | NULL              |
| 7070056491813785604 | rerank | read imbalance              | tpcc           | customer        | __idx_customer__PRIMARY        | 39878656 | 2022-03-01 09:10:03.7 | 2022-03-01 09:10:03.7 | NULL              |
| 7070056491614640129 | rerank | distribution read imbalance | tpcc           | customer        | __idx_customer__PRIMARY        | 39616512 | 2022-03-01 09:10:03.6 | 2022-03-01 09:10:03.6 | NULL              |
| 7070056455896086530 | move   | load imbalance              | tpcc           | customer        | c_w_id                         | 4325376 | 2022-03-01 09:09:55.3 | 2022-03-01 09:10:01.3 | NULL              |
| 7070056431216841732 | move   | load imbalance              | tpcc           | orders          | o_w_id                         | 6881280 | 2022-03-01 09:09:49.5 | 2022-03-01 09:09:55.3 | NULL              |
| 7070056431216709635 | move   | distribution load imbalance | tpcc           | orders          | __idx_orders__PRIMARY          | 5013504 | 2022-03-01 09:09:49.5 | 2022-03-01 09:09:55.4 | NULL              |
| 7070056405749147652 | move   | distribution load imbalance | tpcc           | orders          | __idx_orders__PRIMARY          | 8912896 | 2022-03-01 09:09:43.6 | 2022-03-01 09:09:49.5 | NULL              |
| 7070056302358391809 | move   | load imbalance              | tpcc           | history         | __base_history                 | 7012352 | 2022-03-01 09:09:19.5 | 2022-03-01 09:09:19.6 | No source replica |
| 7070056276881671171 | move   | distribution load imbalance | tpcc           | customer        | __idx_customer__PRIMARY        | 51937280 | 2022-03-01 09:09:13.6 | 2022-03-01 09:09:20.5 | NULL              |
| 7070056276897830913 | move   | distribution load imbalance | tpcc           | history         | __base_history                 | 7012352 | 2022-03-01 09:09:13.6 | 2022-03-01 09:09:19.5 | NULL              |
| 7070056239674573826 | move   | load imbalance              | tpcc           | orders          | __idx_orders__PRIMARY          | 13959168 | 2022-03-01 09:09:05.0 | 2022-03-01 09:09:10.9 | NULL              |
| 7070056211155335172 | move   | load imbalance              | tpcc           | customer        | __idx_customer__PRIMARY        | 54919168 | 2022-03-01 09:08:58.3 | 2022-03-01 09:09:04.9 | NULL              |
| 7070056180744865796 | move   | distribution load imbalance | tpcc           | history         | __base_history                 | 9961472 | 2022-03-01 09:08:51.2 | 2022-03-01 09:08:58.3 | NULL              |
| 7070056148016963586 | move   | distribution load imbalance | tpcc           | customer        | __idx_customer__PRIMARY        | 42303488 | 2022-03-01 09:08:43.6 | 2022-03-01 09:08:51.2 | NULL              |
| 7070056064602365955 | move   | load imbalance              | tpcc           | customer        | __idx_customer__PRIMARY        | 51937280 | 2022-03-01 09:08:24.2 | 2022-03-01 09:08:24.2 | No source replica |
| 7070056062335256580 | rerank | read imbalance              | tpcc           | customer        | __idx_customer__PRIMARY        | 54919168 | 2022-03-01 09:08:23.7 | 2022-03-01 09:08:23.7 | NULL              |
| 7070056062109624322 | rerank | distribution read imbalance | tpcc           | customer        | __idx_customer__PRIMARY        | 64159744 | 2022-03-01 09:08:23.6 | 2022-03-01 09:08:23.6 | NULL              |
| 7070056035998943236 | move   | distribution load imbalance | tpcc           | customer        | __idx_customer__PRIMARY        | 59604992 | 2022-03-01 09:08:17.5 | 2022-03-01 09:08:24.2 | NULL              |
| 7070056019161892867 | move   | distribution load imbalance | tpcc           | stock           | __idx_stock__PRIMARY           | 97943552 | 2022-03-01 09:08:13.6 | 2022-03-01 09:08:22.8 | NULL              |
+---------------------+--------+-----------------------------+----------------+-----------------+--------------------------------+----------+-----------------------+-----------------------+-------------------+

바쁘게 일을 하네 ..

아까 확인해 봤던 c_w_id 인덱스를 다시 한번 확인해 보자

MySQL [system]> select * from table_replicas where `index` = 'c_w_id' ;
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| Database | Table    | Index | slice               | replica             | nodeid | rank | status | bytes   |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070056663513872387 |      3 |    0 |      1 | 4063232 |
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070001415015842817 |      1 |    2 |      1 |   98304 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070056456244686851 |      3 |    0 |      1 | 4063232 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070001415015845889 |      4 |    3 |      1 | 8945664 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070001415015840769 |      2 |    1 |      1 | 5701632 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070045848110571524 |      4 |    2 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070045791243413508 |      1 |    0 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070001415015844865 |      2 |    1 |      1 | 6225920 |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+

7070001415015849985 node3 node1
7070001415015847937 node3 node4
7070001415015851009 node2 node4
7070001415015848961 node1 node2

이상하다... node3에 메인 슬라이스가 2개나 위치하고 있다 .....
조금 시간을 두고 다시 쿼리

MySQL [system]> select * from table_replicas where `index` = 'c_w_id' ;
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| Database | Table    | Index | slice               | replica             | nodeid | rank | status | bytes   |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070056663513872387 |      3 |    0 |      1 | 4063232 |
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070001415015842817 |      1 |    2 |      1 |   98304 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070001415015845889 |      4 |    0 |      1 | 8945664 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070056456244686851 |      3 |    3 |      1 | 4063232 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070045848110571524 |      4 |    0 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070001415015840769 |      2 |    1 |      1 | 5701632 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070045791243413508 |      1 |    0 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070001415015844865 |      2 |    1 |      1 | 6225920 |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+

7070001415015849985 node3 node1
7070001415015847937 node4 node3
7070001415015851009 node4 node2
7070001415015848961 node1 node2

이번에는 node4에 메인 슬라이스가 2개 위치 ...
조금 시간을 두고 다시 쿼리

MySQL [system]> select * from table_replicas where `index` = 'c_w_id' ;
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| Database | Table    | Index | slice               | replica             | nodeid | rank | status | bytes   |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070056663513872387 |      3 |    0 |      1 | 4063232 |
| tpcc     | customer | c_w_id | 7070001415015849985 | 7070001415015842817 |      1 |    2 |      1 |   98304 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070001415015845889 |      4 |    0 |      1 | 8945664 |
| tpcc     | customer | c_w_id | 7070001415015847937 | 7070056456244686851 |      3 |    3 |      1 | 4063232 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070001415015840769 |      2 |    0 |      1 | 5701632 |
| tpcc     | customer | c_w_id | 7070001415015851009 | 7070045848110571524 |      4 |    1 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070045791243413508 |      1 |    0 |      1 | 2719744 |
| tpcc     | customer | c_w_id | 7070001415015848961 | 7070001415015844865 |      2 |    1 |      1 | 6225920 |
+----------+----------+--------+---------------------+---------------------+--------+------+--------+---------+

7070001415015849985 node3 node1
7070001415015847937 node4 node3
7070001415015851009 node2 node4
7070001415015848961 node1 node2

각각의 노드 마다 메인 슬라이스가 1개씩만 위치하는 것을 확인 할 수 있다.

-- 리발란서 동작 확인

MySQL [system]> select * from rebalancer_activity_targets ;
Empty set (0.00 sec)

MySQL [system]> select * from rebalancer_copy_activity ;
Empty set (0.01 sec)

-- 상태 확인

[xpand@xpand3 ~]$ clx space
nid | Hostname | Status |       Undo      |       Perm      |    Temp    |       Used      | DB Total | FS Free
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
1 |   xpand1 |    OK | 256.0K (0.00%) | 457.6M (3.00%) | 0 (0.00%) | 458.2M (3.01%) |    14.9G |    3.8G
2 |   xpand2 |    OK | 256.0K (0.00%) | 541.6M (3.56%) | 0 (0.00%) | 542.4M (3.56%) |    14.9G |    3.8G
3 |   xpand3 |    OK | 256.0K (0.00%) | 454.4M (2.98%) | 0 (0.00%) | 454.9M (2.99%) |    14.9G |    3.8G
4 |   xpand4 |    OK | 256.0K (0.00%) | 450.7M (2.96%) | 0 (0.00%) | 451.4M (2.96%) |    14.9G |    3.8G
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
                             1.0M (0.00%) |    1.9G (3.13%) | 0 (0.00%) |    1.9G (3.13%) |    59.5G |   15.2G

[MariaDB] Xpand + MaxScale #5

★깜장별★ — Fri, 4 Mar 2022 19:19:52 +0900

Xpand 노드 추가

기존에 MaxScale + 3개의 Xpand에 추가로 xpand4 node 추가 시나리오

xpand4

/ect/hosts

10.70.101.16   xpand1
10.70.101.17   xpand2
10.70.101.18   xpand3
10.70.101.19   xpand4
10.70.101.20   maxscale1

--방화벽 OFF (root)

systemctl stop firewalld
systemctl disable firewalld

-- 필요 패키지 설치 (root)

yum install bzip2 mdadm ntp ntpdate screen wget xz -y
yum-config-manager --enable rhui-REGION-rhel-server-optional (RHEL Only)

-- 엔진 설치 (root)

wget https://downloads.mariadb.com/MariaDB/mariadb_repo_setup
chmod +x mariadb_repo_setup
./mariadb_repo_setup --skip-server --skip-tools --apply
wget https://dlm.mariadb.com/6f1eb5f1-292b-4195-8295-d8ded5189e72/1601305/xpand/xpand-5.3.14/xpand-5.3.14.el7.tar.bz2
tar xvf xpand-5.3.14.el7.tar.bz2
cd xpand-5.3.14.el7
./xpdnode_install.py --wizard

-- 생성 된 유저 xpand, xpandm 유저에게 엔진 path 설정 (root)
echo "export PATH=$PATH:/opt/clustrix/bin" >> /home/xpandm/.bash_profile
echo "export PATH=$PATH:/opt/clustrix/bin" >> /home/xpand/.bash_profile
chown xpandm:xpandm /home/xpandm/.bash_profile
chown xpand:xpand /home/xpand/.bash_profile

-- xpand , xpandm 유저의 패스워드 설정 (root, 3개 노드 비번 동일하게 하여 관리 편의 성)

passwd xpand
passwd xpandm

-- xpand node 상태 확인

su - xpandm
clx status    <-- 여기서는 각자 자기 노드 1개 밖에 안 보임 ( 아직 클러스터로 만들지 않아서)
Cluster Name:    cl7d4939f00c4f39c1
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    1 nodes - 8 CPUs per Node
Current Node:    xpand4 - nid 1

nid | Hostname | Status |   IP Address | TPS |      Used      | Total
----+----------+--------+---------------+-----+----------------+-------
1 |   xpand4 |    OK | 10.70.101.19 | 12 | 19.7M (0.10%) | 19.8G
----+----------+--------+---------------+-----+----------------+-------
                           10.70.101.19 | 12 | 19.7M (0.10%) | 19.8G

정상 기동이 되지 않은 경우 clx dbstart 로 xpand 시작

-- DB Root 패스워드 변경

mysql --user=root --no-auto-rehash
SET PASSWORD FOR root@127.0.0.1 = PASSWORD("root1122");

xpand1

-- 클러스터 추가

su - xpand

mysql --user=root --password --no-auto-rehash
ALTER CLUSTER ADD '10.70.101.19';

use system
select nodeid , hostname , iface_ip , mysql_port from nodeinfo ;   -- 4개 서버 정보가 보임
+--------+----------+--------------+------------+
| nodeid | hostname | iface_ip     | mysql_port |
+--------+----------+--------------+------------+
|      1 | xpand1   | 10.70.101.16 |       3306 |
|      4 | xpand4   | 10.70.101.19 |       3306 |
|      2 | xpand2   | 10.70.101.17 |       3306 |
|      3 | xpand3   | 10.70.101.18 |       3306 |
+--------+----------+--------------+------------+

-- 클러스터 상태 확인

clx status (여기서 정상적으로 4개 노드가 보이면 됨)

Cluster Name:    cl90c9aff68a607030
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    4 nodes - 8 CPUs per Node
Current Node:    xpand1 - nid 1

nid | Hostname | Status |   IP Address | TPS |      Used      | Total
----+----------+--------+---------------+-----+----------------+-------
1 |   xpand1 |    OK | 10.70.101.16 |   0 | 23.3M (0.15%) | 14.9G
2 |   xpand2 |    OK | 10.70.101.17 |   0 | 22.1M (0.15%) | 14.9G
3 |   xpand3 |    OK | 10.70.101.18 |   0 | 21.6M (0.14%) | 14.9G
4 |   xpand4 |    OK | 10.70.101.19 |   0 | 10.6M (0.07%) | 14.9G
----+----------+--------+---------------+-----+----------------+-------
                                            0 | 77.7M (0.13%) | 59.5G

-- SSH key 복사 (root)

su - xpand
clx pubkeyinstall
clx cmd date (ssh key 정상 복사 유무테스트, 4개 서버의 현재 시간이 나와야 함)
exit

su - xpandm
clx pubkeyinstall
clx cmd date (ssh key 정상 복사 유무테스트, 4개 서버의 현재 시간이 나와야 함)

xpand4

-- 클러스터 연결 확인

su - xpand
[xpand@xpand4 ~]$ clx status
Cluster Name:    cl90c9aff68a607030   <-- cl7d4939f00c4f39c1 이름이 변경 됨을 확인
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    4 nodes - 8 CPUs per Node
Current Node:    xpand4 - nid 4

nid | Hostname | Status |   IP Address | TPS |      Used      | Total
----+----------+--------+---------------+-----+----------------+-------
1 |   xpand1 |    OK | 10.70.101.16 |   2 | 23.7M (0.16%) | 14.9G
2 |   xpand2 |    OK | 10.70.101.17 |   2 | 22.8M (0.15%) | 14.9G
3 |   xpand3 |    OK | 10.70.101.18 |   0 | 23.3M (0.15%) | 14.9G
4 |   xpand4 |    OK | 10.70.101.19 |   0 |   8.6M (0.06%) | 14.9G
----+----------+--------+---------------+-----+----------------+-------
                                            4 | 78.3M (0.13%) | 59.5G

mysql --user=root --password --no-auto-rehash

use system

select * from nodeinfo ;
select username , host from users ;
+------------------+--------------+
| username         | host         |
+------------------+--------------+
| clx_maint        | 127.0.0.1    |
| clx_view_definer | 127.0.0.1    |
| max              | 10.70.101.20 |
| mysql_slave      |              |
| root             | 127.0.0.1    |
| xpand            | localhost    |
| xpandm           | localhost    |
+------------------+--------------+

-- SSH 정상동작 확인 (root)

su - xpand
clx cmd date (ssh key 정상 복사 유무테스트, 4개 서버의 현재 시간이 나와야 함)
exit

su - xpandm
clx cmd date (ssh key 정상 복사 유무테스트, 4개 서버의 현재 시간이 나와야 함)

MaxScale에 노드 추가

maxscale1

-- xpand server4 추가

maxctrl create server xpand4 10.70.101.19
--> Error 발생
@@xpand4_monitor:node-4가 존재 한다고 나옴 ...

-- 해당 서버를 삭제 후 xpand4 추가

maxctrl destroy server @@xpand_monitor:node-4
maxctrl create server xpand4 10.70.101.19

-- 서버 리스트 확인
@@xpand_monitor:node-4는 삭제된 상태이고 xpand4는 Down으로 확인 됨
모니터를 추가 하면 되므로 계속 진행

-- 모니터 추가
기존에 사용하던 Monitor에 xpand4 추가
maxctrl link monitor xpand_monitor xpand4

모니터를 추가 하면 자동으로 service도 추가 되는데 안 될 경우,
maxctrl link service xpand_service xpand4 수행

-- 최종 확인

MaxScale 편집

기본으로 생성 되는 불 필요한 서버, 모니터, 서비스, 리스너 삭제

서버를 삭제 하려면 서버가 사용되고 있는 monitor와 service를 먼저 삭제 해야 하고,

service를 삭제 하려면 service를 사용하고 있는 리스너를 먼저 삭제 해야 한다.

그래서 모니터 --> 리스너 --> 서비스 --> 서버 순서로 삭제

-- 불필요한 서비스들 삭제

1. 모니터에서 server1과의 관계 삭제
maxctrl unlink monitor MariaDB-Monitor server1

2. 리스너 삭제
maxctrl destroy listener Read-Write-Service Read-Write-Listener
maxctrl destroy listener Read-Only-Service Read-Only-Listener

3. 서비스에서 server1 과의 관계 삭제
maxctrl unlink service Read-Only-Service server1
maxctrl unlink service Read-Write-Service server1

4. 서비스와 모니터, 서버 삭제
maxctrl destroy service Read-Only-Servic
maxctrl destroy service Read-Write-Service
maxctrl destroy monitor MariaDB-Monitor
maxctrl destroy server server1

[MariaDB] Xpand + MaxScale #4

★깜장별★ — Fri, 4 Mar 2022 19:07:21 +0900

Slice

Xpand는 기본적으로 node개수 만큼 테이블, PK, index를 나누어서 저장한다 (hash_dist_min_slices =0)

이때 동등한 크기를 가질 수 있도록 "사이즈"를 기준으로 나누어서 저장한다. (분산 쿼리가 용이 하도록)

슬라이스 개수는 node개수보다 작으면 안 된다. (분산처리의 의미가 없어 지므로...)

테이블의 데이터가 작고, 변경이 없는 테이블 (코드성)이면서 조인에 많이 사용 된다면, 해당 테이블의 데이터는 slice 하지 않고, 모든 노드에서 가지고 있을 수 있도록 생성하는 것이 좋다. (CREATE TABLE ...REPLICAS=ALLNODES)

하나의 슬라이스는 하나의 node에만 저장되며, 하나의 슬라이스가 커지게 되면(기본 8G), 해당 슬라이스를 리발란서가 다시 split 하게 된다. (rebalancer_split_threshold_kb = 8G 기본 값)

테이블 별로 슬라이스 max 크기를 지정해 줄 수도 있다.( split_threshold_kb)

ALTER TABLE tbl_name [SLICES = n] [ , INDEX index_name [SLICES = n] [split_threshold_kb = n]]

Name	Description
hash_dist_min_slices	Controls how data is sliced. If set to 0, (the default), Xpand will create new representations with the number of slices equal to the number of nodes currently in the cluster. At least one slice of the table or index is placed on each node. If set to a specific integer, that number of slices will be created for each new table and index instead.
rebalancer_split_threshold_kb	Default size at which the rebalancer splits slices.
task_rebalancer_reprotect_interval_ms	Defines how frequently the Rebalancer will assess if additional slices are needed. Specify 0 to disable slice splitting.

-- 초기 설치 진행 후

[root@xpand3 clustrix]# clx status
Cluster Name:    cl36c9aa23ff872a40
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    3 nodes - 8 CPUs per Node
Current Node:    xpand3 - nid 2

nid | Hostname | Status |   IP Address | TPS |      Used      | Total
----+----------+--------+---------------+-----+----------------+-------
1 |   xpand1 |    OK | 10.70.101.51 |   0 | 20.8M (0.06%) | 31.9G
2 |   xpand3 |    OK | 10.70.101.53 |   0 | 19.9M (0.06%) | 31.9G
3 |   xpand2 |    OK | 10.70.101.52 |   0 | 20.2M (0.06%) | 31.9G
----+----------+--------+---------------+-----+----------------+-------
                                            0 | 60.9M (0.06%) | 95.6G

[root@xpand3 clustrix]# clx space
nid | Hostname | Status |       Undo      |      Perm      |    Temp    |      Used      | DB Total | FS Free
----+----------+--------+-----------------+----------------+------------+----------------+----------+--------
1 |   xpand1 |    OK | 256.0K (0.00%) | 24.7M (0.08%) | 0 (0.00%) | 25.2M (0.08%) |    31.9G |    5.5G
2 |   xpand3 |    OK | 256.0K (0.00%) | 24.0M (0.07%) | 0 (0.00%) | 24.3M (0.07%) |    31.9G |    5.5G
3 |   xpand2 |    OK | 256.0K (0.00%) | 24.4M (0.07%) | 0 (0.00%) | 24.8M (0.08%) |    31.9G |    5.5G
----+----------+--------+-----------------+----------------+------------+----------------+----------+--------
                           768.0K (0.00%) | 73.1M (0.07%) | 0 (0.00%) | 74.3M (0.08%) |    95.6G |   16.5G

데이터를 넣지 않은 초기 상태임에도 DB Total Size가 각 노드 마다 31.9G를 차지 함.
(테이블 스페이스 처럼 미리 생성된 공간이라 보면 됨)

-- tpcc 데이터 생성 후

MySQL [(none)]> use system
Database changed

MySQL [system]> select * from TABLE_SIZES where database = 'tpcc' ;
-- 여기서 나오는 size (bytes)는 슬라이스 및 Replica의 합이다.

+----------+------------+-----------+
| Database | Table      | Bytes     |
+----------+------------+-----------+
| tpcc     | customer   | 461733888 |
| tpcc     | new_order |   8060928 |
| tpcc     | order_line | 522682368 |
| tpcc     | item       | 22216704 |
| tpcc     | orders     | 85557248 |
| tpcc     | district   |    393216 |
| tpcc     | history    | 58785792 |
| tpcc     | stock      | 686718976 |
| tpcc     | warehouse |    393216 |
+----------+------------+-----------+
9 rows in set (0.04 sec)

MySQL [system]> select * from TABLE_slices where database = 'tpcc' order by 'table' , 'index' ;
+----------+------------+---------------------------+---------------------+
| Database | Table      | Index                     | slice               |
+----------+------------+---------------------------+---------------------+
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858997249 |
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858998273 |
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858999297 |
| tpcc     | customer   | c_w_id                    | 7067205827863643137 |
| tpcc     | customer   | c_w_id                    | 7067205827863644161 |
| tpcc     | customer   | c_w_id                    | 7067205827863645185 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979982849 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979983873 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979984897 |
| tpcc     | history    | __base_history            | 7067205828073042945 |
| tpcc     | history    | __base_history            | 7067205828073043969 |
| tpcc     | history    | __base_history            | 7067205828073044993 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170768385 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170769409 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170770433 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259182593 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259183617 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259184641 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366210049 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366211073 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366212097 |
| tpcc     | orders     | o_w_id                    | 7067205828366220289 |
| tpcc     | orders     | o_w_id                    | 7067205828366221313 |
| tpcc     | orders     | o_w_id                    | 7067205828366222337 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477888513 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477889537 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477890561 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584907777 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584908801 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584909825 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670395393 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670396417 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670397441 |
+----------+------------+---------------------------+---------------------+
33 rows in set (0.01 sec)

MySQL [system]> select * from table_replicas where slice in ( select slice from table_slices where database = 'tpcc') order by 'table', nodeid , rank ;
+----------+------------+---------------------------+---------------------+---------------------+--------+------+--------+-----------+
| Database | Table      | Index                     | slice               | replica             | nodeid | rank | status | bytes     |
+----------+------------+---------------------------+---------------------+---------------------+--------+------+--------+-----------+
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858999297 | 7067205827858991105 |      1 |    0 |      1 | 56492032 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979983873 | 7067205827979978753 |      1 |    0 |      1 |     65536 |
| tpcc     | customer   | c_w_id                    | 7067205827863644161 | 7067205827863639041 |      1 |    0 |      1 |   5865472 |
| tpcc     | history    | __base_history            | 7067205828073042945 | 7067205828073040897 |      1 |    0 |      1 | 10321920 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366211073 | 7067205828366205953 |      1 |    0 |      1 |   3964928 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259184641 | 7067205828259176449 |      1 |    0 |      1 |   1179648 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170769409 | 7067205828170764289 |      1 |    0 |      1 |   3702784 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477890561 | 7067205828477882369 |      1 |    0 |      1 | 77496320 |
| tpcc     | orders     | o_w_id                    | 7067205828366221313 | 7067205828366216193 |      1 |    0 |      1 |   5734400 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584907777 | 7067205828584905729 |      1 |    0 |      1 | 163250176 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670396417 | 7067205828670391297 |      1 |    0 |      1 |     65536 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979982849 | 7067205827979981825 |      1 |    1 |      1 |     65536 |
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858998273 | 7067205827858994177 |      1 |    1 |      1 | 35749888 |
| tpcc     | customer   | c_w_id                    | 7067205827863643137 | 7067205827863642113 |      1 |    1 |      1 | 14909440 |
| tpcc     | history    | __base_history            | 7067205828073044993 | 7067205828073037825 |      1 |    1 |      1 | 10649600 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259183617 | 7067205828259179521 |      1 |    1 |      1 |    851968 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366210049 | 7067205828366209025 |      1 |    1 |      1 |   9764864 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170768385 | 7067205828170767361 |      1 |    1 |      1 |   3702784 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584909825 | 7067205828584902657 |      1 |    1 |      1 | 114196480 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477889537 | 7067205828477885441 |      1 |    1 |      1 | 48005120 |
| tpcc     | orders     | o_w_id                    | 7067205828366220289 | 7067205828366219265 |      1 |    1 |      1 | 11829248 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670395393 | 7067205828670394369 |      1 |    1 |      1 |     65536 |
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858997249 | 7067205827858995201 |      2 |    0 |      1 | 105906176 |
| tpcc     | customer   | c_w_id                    | 7067205827863643137 | 7067205827863641089 |      2 |    0 |      1 | 14909440 |
| tpcc     | history    | __base_history            | 7067205828073044993 | 7067205828073036801 |      2 |    0 |      1 |   9338880 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979984897 | 7067205827979976705 |      2 |    0 |      1 |     65536 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366212097 | 7067205828366203905 |      2 |    0 |      1 |   5898240 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259182593 | 7067205828259180545 |      2 |    0 |      1 |   1998848 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170768385 | 7067205828170766337 |      2 |    0 |      1 |   3702784 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584909825 | 7067205828584901633 |      2 |    0 |      1 | 97976320 |
| tpcc     | orders     | o_w_id                    | 7067205828366222337 | 7067205828366214145 |      2 |    0 |      1 |   5439488 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477888513 | 7067205828477886465 |      2 |    0 |      1 | 135823360 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670397441 | 7067205828670389249 |      2 |    0 |      1 |     65536 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979983873 | 7067205827979979777 |      2 |    1 |      1 |     65536 |
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858999297 | 7067205827858992129 |      2 |    1 |      1 | 56492032 |
| tpcc     | customer   | c_w_id                    | 7067205827863645185 | 7067205827863638017 |      2 |    1 |      1 |   8650752 |
| tpcc     | history    | __base_history            | 7067205828073043969 | 7067205828073039873 |      2 |    1 |      1 |   9371648 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170770433 | 7067205828170763265 |      2 |    1 |      1 |   3702784 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259184641 | 7067205828259177473 |      2 |    1 |      1 |   1179648 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366211073 | 7067205828366206977 |      2 |    1 |      1 |   3964928 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584908801 | 7067205828584904705 |      2 |    1 |      1 | 74022912 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477890561 | 7067205828477883393 |      2 |    1 |      1 | 77496320 |
| tpcc     | orders     | o_w_id                    | 7067205828366221313 | 7067205828366217217 |      2 |    1 |      1 |   5734400 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670396417 | 7067205828670392321 |      2 |    1 |      1 |     65536 |
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858998273 | 7067205827858993153 |      3 |    0 |      1 | 35749888 |
| tpcc     | customer   | c_w_id                    | 7067205827863645185 | 7067205827863636993 |      3 |    0 |      1 |   8650752 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979982849 | 7067205827979980801 |      3 |    0 |      1 |     65536 |
| tpcc     | history    | __base_history            | 7067205828073043969 | 7067205828073038849 |      3 |    0 |      1 |   9568256 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170770433 | 7067205828170762241 |      3 |    0 |      1 |   3702784 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366210049 | 7067205828366208001 |      3 |    0 |      1 |   9764864 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259183617 | 7067205828259178497 |      3 |    0 |      1 |    851968 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584908801 | 7067205828584903681 |      3 |    0 |      1 | 74022912 |
| tpcc     | orders     | o_w_id                    | 7067205828366220289 | 7067205828366218241 |      3 |    0 |      1 | 12124160 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670395393 | 7067205828670393345 |      3 |    0 |      1 |     65536 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477889537 | 7067205828477884417 |      3 |    0 |      1 | 48005120 |
| tpcc     | customer   | __idx_customer__PRIMARY   | 7067205827858997249 | 7067205827858996225 |      3 |    1 |      1 | 112492544 |
| tpcc     | customer   | c_w_id                    | 7067205827863644161 | 7067205827863640065 |      3 |    1 |      1 |   5865472 |
| tpcc     | district   | __idx_district__PRIMARY   | 7067205827979984897 | 7067205827979977729 |      3 |    1 |      1 |     65536 |
| tpcc     | history    | __base_history            | 7067205828073042945 | 7067205828073041921 |      3 |    1 |      1 |   9535488 |
| tpcc     | new_order | __idx_new_order__PRIMARY | 7067205828259182593 | 7067205828259181569 |      3 |    1 |      1 |   1998848 |
| tpcc     | orders     | __idx_orders__PRIMARY     | 7067205828366212097 | 7067205828366204929 |      3 |    1 |      1 |   5898240 |
| tpcc     | item       | __idx_item__PRIMARY       | 7067205828170769409 | 7067205828170765313 |      3 |    1 |      1 |   3702784 |
| tpcc     | orders     | o_w_id                    | 7067205828366222337 | 7067205828366215169 |      3 |    1 |      1 |   5439488 |
| tpcc     | order_line | __idx_order_line__PRIMARY | 7067205828477888513 | 7067205828477887489 |      3 |    1 |      1 | 135856128 |
| tpcc     | stock      | __idx_stock__PRIMARY      | 7067205828584907777 | 7067205828584906753 |      3 |    1 |      1 | 163250176 |
| tpcc     | warehouse | __idx_warehouse__PRIMARY | 7067205828670397441 | 7067205828670390273 |      3 |    1 |      1 |     65536 |
+----------+------------+---------------------------+---------------------+---------------------+--------+------+--------+-----------+
66 rows in set (0.03 sec)

[root@xpand3 ~]# clx status
Cluster Name:    cl36c9aa23ff872a40
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    3 nodes - 8 CPUs per Node
Current Node:    xpand3 - nid 2

nid | Hostname | Status |   IP Address | TPS |       Used      | Total
----+----------+--------+---------------+-----+-----------------+-------
1 |   xpand1 |    OK | 10.70.101.51 |   0 | 651.9M (2.00%) | 31.9G
2 |   xpand3 |    OK | 10.70.101.53 |   0 | 571.5M (1.75%) | 31.9G
3 |   xpand2 |    OK | 10.70.101.52 |   0 | 571.6M (1.75%) | 31.9G
----+----------+--------+---------------+-----+-----------------+-------
                                            0 |    1.8G (1.83%) | 95.6G

[root@xpand3 ~]# clx space
nid | Hostname | Status |       Undo      |       Perm      |    Temp    |       Used      | DB Total | FS Free
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
1 |   xpand1 |    OK | 256.0K (0.00%) | 651.2M (2.00%) | 0 (0.00%) | 652.0M (2.00%) |    31.9G |    5.3G
2 |   xpand3 |    OK | 256.0K (0.00%) | 571.0M (1.75%) | 0 (0.00%) | 571.8M (1.75%) |    31.9G |    5.5G
3 |   xpand2 |    OK | 256.0K (0.00%) | 571.2M (1.75%) | 0 (0.00%) | 571.9M (1.75%) |    31.9G |    5.5G
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
                           768.0K (0.00%) |    1.8G (1.83%) | 0 (0.00%) |    1.8G (1.83%) |    95.6G |   16.3G

Replicas

고 가용성을 위하여 하나의 슬라이스는 1개 이상의 물리적 replicas를 가지게 된다. (즉 기본 2개)

동일한 노드에 동인한 슬라이스의 Replica가 존재 하지 않는다.

Write 발생 시에는 동일 슬라이스의 Replicas(2개)에 동시에 Write가 발생 되지만, Read시에는 우선 순위가 있는 하나의 슬라이스를 읽게 된다.

max_failures 글로벌 변수의 값이 기본적으로 생성되는 replica의 개수 이다.

show variables like 'max_failures' ;
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| max_failures | 1 |
+---------------+-------+

테이블 생성 시 REPLICAS 옵션으로 따로 지정 가능 (CREATE TABLE ... REPLICAS=n / ALTER TABLE .... REPLICAS=n)

작은 테이블이면서 변경 사항이 거의 없고, 조인에 자주 사용되는 테이블이라면 REPLICAS = ALLNODES 옵션을 주고 모든 노드에 데이터가 있을 수 있게 해 주자.

작은 테이블 :

10M 보다 작은 테이블, Write가 1K 미만으로 발생, 다른 테이블과 조인하는데 많이 사용

Rebalancer

노드간 슬라이스의 최적 분포 유지

큰 슬라이스를 작은 슬라이스들로 분할 (rebalancer_split_threshold_kb, split_threshold_kb)

노드 추가 또는 제거 시 슬라이스 이동 담당

노드 사용률이 과도할 때 슬라이스 이동 (워크로드 핫스팟 제거)

노드장애가 발생해도 Xpand 내에서 복제 본 자동 생성 및 유지

가용 영역(AZ) 인식 데이터 분산 복제로 특정 AZ 전체 장애에도 서비스 유지 가능

파라미터 :

rebalancer_split_threshold_kb : Default size at which the rebalancer splits slices
rebalancer_global_task_limit : specifies the number of concurrent rebalancer actions, applicable to all rebalancer actions.
task_rebalancer_%_interval_ms : defines the interval time of when a particular rebalancer task is initiated.
rebalancer_rebalance_task_limit: controls the number of concurrent rebalancing tasks.
rebalancer_vdev_task_limit : limits the number of concurrent rebalancer actions that touch a single device.

현재 Rebalancer가 동작 중일 경우, rebalancer_activity_targets , rebalancer_copy_activity 테이블에서 조회 가능

지난 Rebalancer 동작은 rebalancer_activity_log 테이블에서 조회 가능

System Database

일반 mysql과 다르게 초기 설치 후에 DB에 접속 후 확인해 보면 information_schema 와 test 데이터 베이스 밖에 보이지 않는다.

기본적인 데이터 베이스는 system 데이터베이스 이며, hidden이 true로 되어 있다.

MySQL [(none)]> use system
Database changed

MySQL [system]> select * from databases ;
+---------------------+---------------------+--------------------+--------+------+
| db                  | user                | name               | hidden | cscl |
+---------------------+---------------------+--------------------+--------+------+
|                4097 |                4099 | system             |      1 | 518 |
|                4098 |                4099 | _replication       |      1 | 518 |
|                4104 |                4099 | information_schema |      0 | 518 |
|                4105 |                4099 | temporary_tables   |      1 | 518 |
|                4111 |                4099 | _cursors           |      1 | 518 |
| 7067052845912992769 |                4099 | test               |      0 | 518 |
| 7067052937791471617 | 7067052852943940609 | clustrix_statd     |      1 | 518 |
| 7067052948688550913 | 7067052852943940609 | clustrix_dbi       |      1 | 518 |
+---------------------+---------------------+--------------------+--------+------+

MySQL [system]> select * from users ;
+---------------------+------------------+--------------+-------------------------------------------+----------+-----------+-----------------------+
| user                | username         | host         | password                                  | priority | profiling | plugin                |
+---------------------+------------------+--------------+-------------------------------------------+----------+-----------+-----------------------+
|                4099 | root             | 127.0.0.1    | *E6CC90B878B948C35E92B003C792C46C58C4AF40 |        0 |         0 | mysql_native_password |
|                4100 | mysql_slave      |              |                                           |        0 |         0 | mysql_native_password |
|                4110 | clx_maint        | 127.0.0.1    | *845EE408D4B27C5C5A9751D7786FDA7E89B2C02E |        0 |         0 | mysql_native_password |
|                4112 | clx_view_definer | 127.0.0.1    |                                           |        0 |         0 | mysql_native_password |
| 7067052852943940609 | xpand            | localhost    |                                           |        0 |         0 | mysql_native_password |
| 7067052853915312129 | xpandm           | localhost    |                                           |        0 |         0 | mysql_native_password |
| 7067137009741132801 | mxs              | 10.70.101.55 | *E6CC90B878B948C35E92B003C792C46C58C4AF40 |        0 |         0 | mysql_native_password |
| 7067205341296116737 | app              | %            | *E6CC90B878B948C35E92B003C792C46C58C4AF40 |        0 |         0 | mysql_native_password |
+---------------------+------------------+--------------+-------------------------------------------+----------+-----------+-----------------------+

system 데이터 베이스가 유용한 정보를 모두 담고 있으므로, 사용하는 모든 쿼리는 system 데이터베이스에서 조회 하면 된다.

[MariaDB] Xpand + MaxScale #3

★깜장별★ — Fri, 4 Mar 2022 18:35:33 +0900

MaxScale

기본 명령어들

기동/종료

systemctl [start | stop | restart | status] maxscale

systemctl [enable | disable] maxscale

journalctl -u maxscale

운영

설치

-- /ect/hosts (root)

10.70.101.16   xpand1
10.70.101.17   xpand2
10.70.101.18   xpand3
10.70.101.19   xpand4

10.70.101.20   maxscale1

--방화벽 OFF (root)

systemctl stop firewalld
systemctl disable firewalld

-- 필요 패키지 설치 (root) <-- Xpand와 그냥 동일하게 맞춰 주었음

yum install bzip2 mdadm ntp ntpdate screen wget xz -y
yum-config-manager --enable rhui-REGION-rhel-server-optional (RHEL Only)

-- 엔진 설치 (root)

wget https://downloads.mariadb.com/MariaDB/mariadb_repo_setup
echo "b9e90cde27affc2a44f9fc60e302ccfcacf71f4ae02071f30d570e6048c28597 mariadb_repo_setup" \ | sha256sum -c -
chmod +x mariadb_repo_setup
sudo ./mariadb_repo_setup \
   --skip-server \
   --skip-tools \
   --mariadb-maxscale-version="2.5"
yum install maxscale -y

-- 기본 설정(root)

/etc/maxscale.cnf
[maxscale]
threads = auto
admin_host = 0.0.0.0
admin_secure_gui = false

-- maxscale 재 시작

systemctl restart maxscale

-- 기본 서비스 확인

-- Xpand Server 추가

maxctrl create server xpand1 10.70.101.16
maxctrl create server xpand2 10.70.101.17
maxctrl create server xpand3 10.70.101.18

-- Xpand Monitor 추가 (maxscale DB유저 입력)

maxctrl create monitor xpand_monitor xpandmon \
   --servers xpand1 xpand2 xpand3 \
   --monitor-user=max \
   --monitor-password=max1122

-- readwritesplit라는 이름의 라우터 추가 (부하 분산용으로 3개 서버를 설정)

maxctrl create service xpand_service readwritesplit \
   user=max \
   password=max1122 \
   --servers xpand1 xpand2 xpand3

-- 라우터의 리슨 포트 생성
(maxscale에 3306으로 접속하게 되면 자동으로 xpand1 , xpand2 , xpand3으로 커넥션 개수를 기준으로 라우팅 함)

maxctrl create listener xpand_service xpand_listener 3306 \
--protocol=MariaDBClient

-- 설정 확인

-- client 접속

10.70.101.20 3306 포트를 이용하여 접근하면 된다.

모니터링 웹

8989 포트를 이용하여 MaxScale 모니터링 웹에 접속 가능

기본 제공 :

admin / mariadb

[MariaDB] Xpand + MaxScale #2

★깜장별★ — Fri, 4 Mar 2022 18:16:54 +0900

Xpand

Xpand를 설치 하게 되면 다음과 같은 기본 유저들이 생성 된다.

O/S 유저

Xpand

설치 시에 자동으로 xpand , xpandm 유저가 생성된다.

SSH를 통한 node간에 접속이 있어야 하므로, 관리 편의 상 동일 O/S 패스워드 사용 권장

xpand : mysql 기동 유저

xpandm : 관리용 유저

DB 유저

기본으로 생성 되는 유저

root@127.0.0.1 : database administration

xpand@localhost : xpand OS user account. Xpand process Managing.

xpandm@localhost : xpandm OS user account. run "clx" command

Xpand Node 설치

최소 사양인 3대 노드를 설치 한다.

-- 공통 작업 (3개 노드 모두 진행)

-- /ect/hosts 설정 (root)

10.70.101.16   xpand1
10.70.101.17   xpand2
10.70.101.18   xpand3
10.70.101.19   xpand4
10.70.101.20   maxscale1

-- 엔진 설치 (root)

wget https://downloads.mariadb.com/MariaDB/mariadb_repo_setup
chmod +x mariadb_repo_setup
./mariadb_repo_setup --skip-server --skip-tools --apply
wget https://dlm.mariadb.com/6f1eb5f1-292b-4195-8295-d8ded5189e72/1601305/xpand/xpand-5.3.14/xpand-5.3.14.el7.tar.bz2
tar xvf xpand-5.3.14.el7.tar.bz2
cd xpand-5.3.14.el7
./xpdnode_install.py --wizard

-- 생성 된 유저 xpand, xpandm 유저에게 엔진 path 설정(root)

echo "export PATH=$PATH:/opt/clustrix/bin" >> /home/xpandm/.bash_profile
echo "export PATH=$PATH:/opt/clustrix/bin" >> /home/xpand/.bash_profile
chown xpandm:xpandm /home/xpandm/.bash_profile
chown xpand:xpand /home/xpand/.bash_profile

-- xpand , xpandm 유저의 패스워드 설정 (root, 3개 노드 비번 동일하게 하여 관리 편의 성)

passwd xpand
passwd xpandm

-- xpand node 상태 확인

su - xpandm
clx status    <-- 여기서는 각자 자기 노드 1개 밖에 안 보임 ( 아직 클러스터로 만들지 않아서)
Cluster Name:    cl90c9aff68a607030
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    1 nodes - 8 CPUs per Node
Current Node:    xpand1 - nid 1

nid | Hostname | Status |   IP Address | TPS |      Used      | Total
----+----------+--------+---------------+-----+----------------+-------
1 |   xpand1 |    OK | 10.70.101.16 |   0 | 19.7M (0.10%) | 19.8G
----+----------+--------+---------------+-----+----------------+-------
                           10.70.101.16 |   0 | 19.7M (0.10%) | 19.8G

정상 기동이 되지 않은 경우 clx dbstart 로 xpand 시작

-- DB Root 패스워드 변경
mysql --user=root --no-auto-rehash
SET PASSWORD FOR root@127.0.0.1 = PASSWORD("root1122");

라이센스 입력 (1개 노드에서만 적용, 여기서는 xpand1에서 수행)

메일로 받은 트라이얼 라이센스나 정식 라이센스 입력

-- xpand 라이센스 적용 (하나의 node에서만 한다.)

mysql --user=root --password --no-auto-rehash

SET GLOBAL license = '{"expiration": "2022-04-06 00:00:00", "company": "Goodusdata", "email": "xxxxxx@yyyyy.com", "person": "********", "signature": "302c02145eb31944ed5d5657a07a4de5fa85ed136e2d5902"}' ;

클러스터 설정( 라이센스 입력한 노드에서만 수행)

-- 클러스터 설정 (라이센스를 적용한 node에서만 한다.)

mysql --user=root --password --no-auto-rehash

ALTER CLUSTER ADD '10.70.101.17', '10.70.101.18';

use system
select nodeid , hostname , iface_ip , mysql_port from nodeinfo ;   -- 3개 서버 정보가 보임

+--------+----------+--------------+------------+
| nodeid | hostname | iface_ip     | mysql_port |
+--------+----------+--------------+------------+
|      1 | xpand1   | 10.70.101.16 |       3306 |
|      3 | xpand3   | 10.70.101.18 |       3306 |
|      2 | xpand2   | 10.70.101.17 |       3306 |
+--------+----------+--------------+------------+

-- 클러스터 상태 확인

clx status (여기서 정상적으로 3개 노드가 보이면 됨)

clx status
Cluster Name:    cl90c9aff68a607030
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    3 nodes - 8 CPUs per Node
Current Node:    xpand1 - nid 1

nid | Hostname | Status |   IP Address | TPS |      Used      | Total
----+----------+--------+---------------+-----+----------------+-------
1 |   xpand1 |    OK | 10.70.101.16 |   0 | 20.2M (0.13%) | 14.9G
2 |   xpand2 |    OK | 10.70.101.17 |   0 | 14.8M (0.10%) | 14.9G
3 |   xpand3 |    OK | 10.70.101.18 |   0 | 14.7M (0.10%) | 14.9G
----+----------+--------+---------------+-----+----------------+-------
                                            0 | 49.6M (0.11%) | 44.6G

SSH키 복사

sudo grep -i -E "^PasswordAuthentication" /etc/ssh/sshd_config 명령어 수행 결과에 따라서 다르게 수행

"PasswordAuthentication yes" 일 경우 ssh key가 자동 복사 되므로 node 1에서만 수행

"PasswordAuthentication no" 일 경우 ssh key가 자동으로 복사 되지 않으므로 모든 노드에서 SSH Key 복사 필요

"PasswordAuthentication yes" 일 경우

su - xpand
clx pubkeyinstall
clx cmd date (ssh key 정상 복사 유무테스트, 3개 서버의 현재 시간이 나와야 함)
exit

su - xpandm
clx pubkeyinstall
clx cmd date (ssh key 정상 복사 유무테스트, 3개 서버의 현재 시간이 나와야 함)
exit

"PasswordAuthentication no" 일 경우

su - xpand
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub 해서 값 확인
3개 노드면 3개 노드 값을 모두 cpoy해서 3개 노드 모두 저장해 준다.
vi ~..ssh/autorized_keys 파일에 키 값 저장
chmod 600 ~/.ssh/authorized_keys
clx cmd date (ssh key 정상 복사 유무테스트, 3개 서버의 현재 시간이 나와야 함)

su - xpandm
ssh-keygen -t rsa
cat ~/.ssh/id_rsa.pub 해서 값 확인
3개 노드면 3개 노드 값을 모두 cpoy해서 3개 노드 모두 저장해 준다.
vi ~..ssh/autorized_keys 파일에 키 값 저장
chmod 600 ~/.ssh/authorized_keys
clx cmd date (ssh key 정상 복사 유무테스트, 3개 서버의 현재 시간이 나와야 함)

MaxScale 모니터링용 DB계정 생성

클러스터가 묶인 상태이므로 아무 서버에서 한 번만 수행해 주면 된다.

(여기서는 maxscale 서버 아이피는 10.70.101.20 이다)

-- maxscale용 DB 유저 생성

mysql --user=root --password --no-auto-rehash

CREATE USER 'max'@'10.70.101.20' IDENTIFIED BY 'max1122';
GRANT SELECT ON system.membership TO 'max'@'10.70.101.20';
GRANT SELECT ON system.nodeinfo TO 'max'@'10.70.101.20';
GRANT SELECT ON system.softfailed_nodes TO 'max'@'10.70.101.20';
GRANT SELECT ON system.user_acl TO 'max'@'10.70.101.20';
GRANT SELECT ON system.users TO 'max'@'10.70.101.20';

Xpand Web 접속 확인

Xpand마다 기본적으로 모니터링 웹을 제공한다. (기본 8080)

기본 값 :

Email: noreply@mariadb.com
Password: mariadb
Target: localhost

기본 명령어들

[xpandm@xpand1 clustrix]$ clx status

Cluster Name:    cl36c9aa23ff872a40
Cluster Version: 5.3.14
Cluster Status:   OK
Cluster Size:    4 nodes - 8 CPUs per Node
Current Node:    xpand1 - nid 1

nid | Hostname | Status |   IP Address | TPS |       Used      | Total
----+----------+--------+---------------+-----+-----------------+--------
1 |   xpand1 |    OK | 10.70.101.51 |   0 | 402.1M (1.23%) |   31.9G
2 |   xpand3 |    OK | 10.70.101.53 |   0 | 429.1M (1.31%) |   31.9G
3 |   xpand2 |    OK | 10.70.101.52 |   0 | 377.5M (1.16%) |   31.9G
4 |   xpand4 |    OK | 10.70.101.54 |   0 | 449.9M (1.38%) |   31.9G
----+----------+--------+---------------+-----+-----------------+--------
                                            0 |    1.6G (1.27%) | 127.5G

[xpandm@xpand1 clustrix]$ clx space

nid | Hostname | Status |       Undo      |       Perm      |    Temp    |       Used      | DB Total | FS Free
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
1 |   xpand1 |    OK | 256.0K (0.00%) | 401.3M (1.23%) | 0 (0.00%) | 402.1M (1.23%) |    31.9G |    5.3G <-- df 했을 때 O/S의 남은 공간
2 |   xpand3 |    OK | 256.0K (0.00%) | 428.3M (1.31%) | 0 (0.00%) | 429.1M (1.31%) |    31.9G |    5.5G
3 |   xpand2 |    OK | 256.0K (0.00%) | 376.9M (1.15%) | 0 (0.00%) | 377.5M (1.16%) |    31.9G |    5.5G
4 |   xpand4 |    OK | 256.0K (0.00%) | 449.5M (1.38%) | 0 (0.00%) | 449.9M (1.38%) |    31.9G |    5.5G
----+----------+--------+-----------------+-----------------+------------+-----------------+----------+--------
                             1.0M (0.00%) |    1.6G (1.27%) | 0 (0.00%) |    1.6G (1.27%) |   127.5G |   21.8G

[xpandm@xpand1 clustrix]$ df -h
Filesystem               Size Used Avail Use% Mounted on
devtmpfs                  32G     0   32G   0% /dev
tmpfs                     32G 129M   32G   1% /dev/shm
tmpfs                     32G 9.0M   32G   1% /run
tmpfs                     32G     0   32G   0% /sys/fs/cgroup
/dev/mapper/centos-root   81G 2.6G   74G   4% /
/dev/sdb2                 50G   55M   47G   1% /log
/dev/sdb1                 50G   42G 5.4G 89% /data
/dev/sda1                 10G 187M 9.9G   2% /boot
tmpfs                    6.3G     0 6.3G   0% /run/user/1000
tmpfs                    6.3G     0 6.3G   0% /run/user/0

[xpandm@xpand1 clustrix]$ clx disk

nid | Hostname | Status | Read IOPS | Write IOPS | Sync IOPS | Total IOPS
----+----------+--------+-----------+------------+-----------+-----------
1 |   xpand1 |    OK |      15.8 |        0.0 |       0.2 |       16.0
2 |   xpand3 |    OK |      16.0 |        0.0 |       0.0 |       16.0
3 |   xpand2 |    OK |      15.8 |        0.0 |       0.2 |       16.0
4 |   xpand4 |    OK |      16.0 |        0.0 |       0.2 |       16.2
----+----------+--------+-----------+------------+-----------+-----------
                               63.6 |        0.0 |       0.6 |       64.2

[MariaDB] Xpand + MaxScale #1

★깜장별★ — Fri, 4 Mar 2022 17:41:36 +0900

테스트를 해 봐야 할 일이 생겨서 테스트 했던거 포스팅 합니다.

하나로 올리기에는 길어서 나눠서 올립니다.

Xpand는 샤딩 개념이 들어간 데이터베이스 이다. 기존의 cluster들은 동일한 데이터를 각각의 데이터 베이스가 동일하게 가지고 있으면서 동기화를 하는 개념이었다면, Xpand는 shared nothing 개념으로 데이터를 슬라이스라는 작은 단위로 쪼개서 서로 다른 노드들이 가지게 하는 것이다.

기존의 샤딩을 지원하는 DB들은 샤드키를 잘 생성해야 데이터가 모든 노드에 균등하게 들어가게 할 수 있었다.

그러나 Xpand는 샤드키가 아닌, 사이즈(크기)를 기준으로 자동으로 데이터를 노드간에 나눠 가지게 된다.

발란서가 그 역활을 수행하며, 다른 샤드 DB들과 마찬가지로 SSH 연결을 통하여 데이터를 주고 받는다.

최소 구성으로 1개의 MaxScale 서버와 3개의 Xpand Node로 구성되어 진다.

Xpand Node에서 데이터가 저장이 되는 위치 ext4 파일 포멧으로 만들어져야 하며, 개별 마운트포인트를 이용하여 마운트 시켜야 한다. ( 테스트 시에 루투 / 를 ext4로 해 주었더니 개별 마운트 포인트가 아니라 에러가 발생 ...)

아래는 최소 사양으로 구성된 Xpand 구성도

Xpand를 설치하고 확인해 보면 기본 스토리지 엔진은 Clustrix엔진으로 되어 있다.

기존 MariaDB처럼 mysqld가 기동이 되는 것이 아니라 clxnode라는 프로그램이 기동이 되어서 MySQL 역활을 수행하게 된다. (nanny --> clxnode 기동)

[root@xpand1 ~]# ps -u xpand uf

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND

xpand 1213 1.7 0.3 319384 258836 ? SLl 09:11 0:49 /opt/clustrix/bin/nanny -config /opt/clustrix/etc/nanny.conf -logfile /log/nanny.log

xpand 1390 9.4 0.3 61752816 197708 ? SLl 09:11 4:29 \_ /opt/clustrix/bin/clxnode -hugetlb -memlog -memlog-clx-path /log/clustrix.log -memlog-debug-path /log/debug.log -memlog-query-path /log/query.log -memlog-user-path /log/user.log -data-path /data/clustrix -memdump-path /log -be-addr 10.70.101.51 -s

xpand 1391 0.0 0.0 303628 33040 ? Sl 09:11 0:02 \_ puma 3.0.0 (tcp://0.0.0.0,tcp://0.0.0.0:8080) [clxgui] <-- web monitoring

xpand 2801 0.0 0.1 1324112 70440 ? Sl 09:11 0:02 | \_ puma: cluster worker 0: 1391 [clxgui]

xpand 1392 0.0 0.0 295592 33504 ? Sl 09:11 0:02 \_ puma 3.7.1 (tcp://0.0.0.0,tcp://0.0.0.0:7888) [clxdbi]

xpand 1409 0.0 0.0 11824 1188 ? S 09:11 0:00 | \_ bash /opt/clustrix/clxdbi/bin/run

xpand 1410 0.0 0.0 11824 1160 ? S 09:11 0:00 | \_ bash /opt/clustrix/clxdbi/bin/run

xpand 2950 0.1 0.1 813624 99596 ? Sl 09:12 0:05 | \_ puma: cluster worker 0: 1392 [clxdbi]

xpand 3054 0.0 0.0 99580 11128 ? S 09:40 0:00 \_ python /opt/clustrix/bin/alerter.py /opt/clustrix/var/tmp/alertsfifo

xpand 3056 0.0 0.0 188436 12704 ? S 09:41 0:00 \_ python /opt/clustrix/bin/statd.py

Xpand는 mairadb enterprise 제품으로 테스트를 위해서 45일간 쓸 수 있는 트라이얼 라이선스를 발급 받아서 사용할 수 있다. (홈페이지에 신청을 하게 되면, 다운 받는 위치와 트라이얼 라이센스를 제공한다.)

MaxScale

1대 이상 필요.

클라이언트로 부터의 쿼리 요청을 받으며, Xpand Node로의 커넥션 분산을 처리 한다.

Route queries(Read/Write분산) , Read Connection(readconnroute) , Read/Write Connection(readwritespit)

xpand node의 health & availability 모니터링 (xpandmon) 모니터링 웹 화면 제공

CPU : 8+ / Memory 16G+

Xpand Node

3대 이상의 xpand Node 필요

분산 데이터 저장 , 개별 노드마다 모니터링 웹 화면 제공

CPU : 8-32 / Memory 64G+

O/S : Cent7 (x86_64) , Red Hat Enterprise Linux 7(x86_64)

Data, Log 저장 경로 : 개별 마운트(/data, /log) 된 디스크가 필요 하며, ext4 포멧으로 만들어져야 한다.

/data/clustrix 20G+ SSD ext4

/data/clustrix안에 device1 파일에 유저 데이터 저장

Oracle Data to Mongo

★깜장별★ — Tue, 11 Aug 2020 10:27:06 +0900

Oracle to Mongo 시나리오 테스트

2020년 8월 10일 월요일

09:49

Oracle Sample schema Download : https://www.oracletutorial.com/getting-started/oracle-sample-database/

테이블 생성 시 에러 발생 발생 할 경우 generate 제거하고 그냥 Primary Key만 살려 두고 만들면 됨.

ERD

Table Info

Order 부분을 묶어서 하나의 Collection으로 만드는 예

다음 쿼리 결과를 하나의 Collection으로 묶을 예정

SELECT o.customer_id , o.order_id , oi.item_id , o.status , o.order_date ,product_name , quantity , p.list_price , o.salesman_id

FROM orders o ,

order_items oi ,

products p

WHERE o.order_id = oi.order_id

AND oi.product_id = p.product_id

ORDER BY o.customer_id , o.order_id , item_id ;

1 60 1 Shipped 2017/06/30 금요일 G.Skill Trident X 74 649.99 62

1 60 2 Shipped 2017/06/30 금요일 G.Skill Ripjaws V Series 138 704.99 62

1 60 3 Shipped 2017/06/30 금요일 G.Skill Ripjaws 4 Series 144 1073.99 62

1 60 4 Shipped 2017/06/30 금요일 Intel Xeon E5-2643 V2 (OEM/Tray) 125 2200 62

1 60 5 Shipped 2017/06/30 금요일 G.Skill Ripjaws V Series 114 677.99 62

1 60 6 Shipped 2017/06/30 금요일 Intel Xeon E5-2695 V4 79 2269.99 62

1 60 7 Shipped 2017/06/30 금요일 Asus X99-E-10G WS 146 649 62

1 77 1 Shipped 2017/01/02 월요일 Kingston 65 671.38 60

1 77 2 Shipped 2017/01/02 월요일 Seagate ST31000340NS – FFP 136 43.99 60

1 77 3 Shipped 2017/01/02 월요일 Intel Xeon E5-2660 V4 143 1388.89 60

1 77 4 Shipped 2017/01/02 월요일 Intel Xeon E5-2697 V4 99 2554.99 60

1 77 5 Shipped 2017/01/02 월요일 MSI X99A GODLIKE GAMING CARBON 148 549.59 60

1 94 1 Shipped 2017/10/27 금요일 Seagate ST2000DX002 38 90.99 62

1 94 2 Shipped 2017/10/27 금요일 PNY VCQK5200-PB 146 1449.98 62

1 94 3 Shipped 2017/10/27 금요일 Intel Xeon E5-2660 V4 86 1388.89 62

1 94 4 Shipped 2017/10/27 금요일 Intel Xeon E5-1650 V3 133 564.89 62

1 94 5 Shipped 2017/10/27 금요일 ATI FirePro R5000 143 999.99 62

1 94 6 Shipped 2017/10/27 금요일 AMD 100-505989 111 2699.99 62

1 94 7 Shipped 2017/10/27 금요일 ASRock EP2C612 WS 37 358.49 62

1 94 8 Shipped 2017/10/27 금요일 PNY SSD7CS1311-120-RB 73 57.98 62

1 94 9 Shipped 2017/10/27 금요일 Gigabyte GV-N108TAORUSX W-11GD 33 824.98 62

예상 추출 데이터 형식

하나의 주문에 대해서 상세 내역을 배열로 넣을 예정

{ customer_id : 1 , order_id : 60 , status : "Shipped" , order_date : "2017/06/30 금요일" , sales_id : 62 ,

order_detail : [ { product_name : "G.Skill Trident X" , quntity : 74 , unit_price : 649.99 } ,

{ product_name : "G.Skill Ripjaws V Series" , quntity : 138 , unit_price : 704.99 } ,

{ product_name : "G.Skill Ripjaws 4 Series" , quntity : 144 , unit_price : 1073.99 } ,

{ product_name : "Intel Xeon E5-2643 V2 (OEM/Tray)" , quntity : 125 , unit_price : 2200 } ,

{ product_name : "G.Skill Ripjaws V Series" , quntity : 114 , unit_price : 677.99 } ,

{ product_name : "Intel Xeon E5-2695 V4" , quntity : 79 , unit_price : 2269.99 } ,

{ product_name : "Asus X99-E-10G WS" , quntity : 146 , unit_price : 649 } ] } ,

{ customer_id : 1 , order_id : 77 , status : "Shipped" , order_date : "2017/01/02 월요일" , sales_id : 60 ,

order_detail : [ { product_name : "Kingston" , quntity : 65 , unit_price : 671.38 } ,

{ product_name : "Seagate ST31000340NS - FFP" , quntity : 136 , unit_price : 43.99 } ,

{ product_name : "Intel Xeon E5-2660 V4" , quntity : 143 , unit_price : 1388.89 } ,

{ product_name : "Intel Xeon E5-2697 V4" , quntity : 99 , unit_price : 2554.99 } ,

{ product_name : "MSI X99A GODLIKE GAMING CARBON" , quntity : 148 , unit_price : 549.59 } ] }

Oracle 11g

JSON 함수를 지원하지 않기 때문에 PL/SQL로 데이터 추출 (Insert 구문으로 추출 예정)

데이터가 없는 부분은 Null 처리 ( mongo에서는 null 소문자로 써야 인식)

/tmp/order.sql

set timing off

set feed off

set serveroutput on

DECLARE

v_total_rn number := 1 ;

v_sub_rn number := 1 ;

v_item_sum number := 0 ;

v_db varchar2(100) := 'use ot' ;

v_collection varchar2(4000) := 'db.order.insert( ' ;

BEGIN

DBMS_OUTPUT.PUT_LINE( v_db ) ;

FOR v_main IN ( SELECT count(*) over () total_rn , o.order_id , ' { customer_id : ' || to_char(o.customer_id) || ' , order_id : ' || to_char(o.order_id) || ' , status : "' || nvl(o.status,'null') || '" , order_date : "' || nvl(to_char(o.order_date),'null') || '" , sales_id : ' || NVL(to_char(o.salesman_id), 'null') || ' , order_detail : ' as str FROM orders o ) LOOP

v_sub_rn := 1 ;

v_item_sum := 0 ;

DBMS_OUTPUT.PUT_LINE (v_collection ||v_main.str || ' [ ' ) ;

FOR v_sub IN ( SELECT max(oi.item_id) over (partition by oi.order_id) rn , oi.quantity * oi.unit_price as price , ' ' || ' { product_name : "' || p.product_name || '" , quantity : ' || to_char(oi.quantity) || ' , unit_price : ' || to_char(oi.unit_price) || ' } ' as s_str FROM order_items oi , products p WHERE oi.order_id = v_main.order_id AND oi.product_id = p.product_id ) LOOP

v_item_sum := v_item_sum + v_sub.price ;

IF v_sub_rn = v_sub.rn THEN

DBMS_OUTPUT.PUT_LINE( v_sub.s_str || ' ]' ) ;

ELSE

DBMS_OUTPUT.PUT_LINE( v_sub.s_str || ' , ' ) ;

END IF ;

v_sub_rn := v_sub_rn + 1 ;

END LOOP ;

v_total_rn := v_total_rn + 1 ;

DBMS_OUTPUT.PUT_LINE ( ' , order_sum_price : ' || to_char(v_item_sum) || ' }) ' ) ;

END LOOP ;

END ;

sqlplus -s ot/ot < /tmp/order.sql > /tmp/order.json

use ot

db.order.insert( { customer_id : 1 , order_id : 105 , status : "Pending" , order_date : "17-NOV-16" , sales_id : 54 , order_detail : [

{ product_name : "G.Skill Ripjaws V Series" , quantity : 80 , unit_price : 645.99 } ,

{ product_name : "Supermicro MBD-X10DAX" , quantity : 116 , unit_price : 443.72 } ,

{ product_name : "MSI X99A GODLIKE GAMING" , quantity : 45 , unit_price : 399.99 } ,

{ product_name : "Intel Xeon E5-1680 V3 (OEM/Tray)" , quantity : 67 , unit_price : 1751.99 } ,

{ product_name : "Asus GTX780TI-3GD5" , quantity : 79 , unit_price : 899.99 } ,

{ product_name : "EVGA 11G-P4-6598-KR" , quantity : 77 , unit_price : 809.99 } ]

, order_sum_price : 372002.04 })

db.order.insert( { customer_id : 2 , order_id : 44 , status : "Pending" , order_date : "20-FEB-17" , sales_id : 55 , order_detail : [

{ product_name : "MSI X99A GODLIKE GAMING CARBON" , quantity : 37 , unit_price : 549.59 } ,

{ product_name : "Intel Xeon E5-2667 V3 (OEM/Tray)" , quantity : 100 , unit_price : 2009.46 } ,

{ product_name : "PNY VCQP4000-PB" , quantity : 148 , unit_price : 829.89 } ,

{ product_name : "MSI GeForce GTX 1080 Ti GAMING X 11G" , quantity : 118 , unit_price : 759.99 } ]

, order_sum_price : 433783.37 })

db.order.insert( { customer_id : 3 , order_id : 101 , status : "Pending" , order_date : "03-JAN-17" , sales_id : 55 , order_detail : [

{ product_name : "Corsair Vengeance LPX" , quantity : 47 , unit_price : 699.01 } ,

{ product_name : "MSI X299 TOMAHAWK ARCTIC" , quantity : 36 , unit_price : 281.98 } ,

{ product_name : "Samsung MZ-V6E250" , quantity : 134 , unit_price : 127.88 } ,

{ product_name : "Kingston SA400S37/120G" , quantity : 52 , unit_price : 54.99 } ,

{ product_name : "Intel Xeon E5-2687W V4" , quantity : 123 , unit_price : 2042.69 } ,

{ product_name : "Corsair Dominator Platinum" , quantity : 138 , unit_price : 719.99 } ,

{ product_name : "G.Skill Trident Z" , quantity : 134 , unit_price : 704.99 } ]

, order_sum_price : 508078.3 })

db.order.insert( { customer_id : 6 , order_id : 6 , status : "Shipped" , order_date : "09-APR-15" , saleman_id : null , order_detail : [

{ product_name : "G.Skill Ripjaws V Series" , quantity : 119 , unit_price : 680.99 } ,

{ product_name : "G.Skill Ripjaws V Series" , quantity : 41 , unit_price : 645.99 } ,

{ product_name : "PNY SSD7CS1311-120-RB" , quantity : 145 , unit_price : 57.98 } ,

{ product_name : "Western Digital WD2500AVVS" , quantity : 90 , unit_price : 15.55 } ,

{ product_name : "Intel Xeon E5-2630 V4" , quantity : 67 , unit_price : 647.99 } ,

{ product_name : "PNY VCQP5000-PB" , quantity : 74 , unit_price : 2015.11 } ,

{ product_name : "PNY VCGGTX780T3XPB-OC" , quantity : 150 , unit_price : 749.99 } ]

, order_sum_price : 422361.97 })

mongo --host mongo:9999 -u root -p root1122 --authenticationDatabase admin < /tmp/order.json

Oracle12c

제공해 주는 JSON 함수를 이용하여 추출 (12c R2부터 지원)

/tmp/order12.sql

set pagesize 0

set heading off

set timing off

set feed off

set line 32767

SELECT 'use ot' FROM dual

UNION ALL

SELECT 'db.order12.insert(' || JSON_OBJECT(

'customer_id' value o.customer_id

, 'order_id' value o.order_id

, 'status' value o.status

, 'order_date' value o.order_date

, 'sales_id' value o.salesman_id

, 'order_detail' value ( SELECT JSON_ARRAYAGG( JSON_OBJECT(

'product_name' value p.product_name

, 'quantity' value oi.quantity

, 'unit_price' value oi.unit_price ))

FROM order_items oi , products p

WHERE oi.order_id = o.order_id AND oi.product_id = p.product_id

)
, 'order_sum_price' value ( select sum(quantity*unit_price) from order_items oi where oi.order_id = o.order_id)

) || ' ) '

FROM orders o ;

sqlplus -s ot/ot < /tmp/order12.sql > /tmp/order12.json

use ot

db.order12.insert({"customer_id":1,"order_id":105,"status":"Pending","order_date":"2016-11-17T00:00:00","sales_id":54,"order_detail":[{"product_name":"Asus GTX780TI-3GD5","quantity":79,"unit_price":899.99},{"product_name":"EVGA 11G-P4-6598-KR","quantity":77,"unit_price":809.99},{"product_name":"G.Skill Ripjaws V Series","quantity":80,"unit_price":645.99},{"product_name":"Supermicro MBD-X10DAX","quantity":116,"unit_price":443.72},{"product_name":"MSI X99A GODLIKE GAMING","quantity":45,"unit_price":399.99},{"product_name":"Intel Xeon E5-1680 V3 (OEM/Tray)","quantity":67,"unit_price":1751.99}],"order_sum_price":372002.04} )

db.order12.insert({"customer_id":2,"order_id":44,"status":"Pending","order_date":"2017-02-20T00:00:00","sales_id":55,"order_detail":[{"product_name":"PNY VCQP4000-PB","quantity":148,"unit_price":829.89},{"product_name":"MSI GeForce GTX 1080 Ti GAMING X 11G","quantity":118,"unit_price":759.99},{"product_name":"MSI X99A GODLIKE GAMING CARBON","quantity":37,"unit_price":549.59},{"product_name":"Intel Xeon E5-2667 V3 (OEM/Tray)","quantity":100,"unit_price":2009.46}],"order_sum_price":433783.37} )

db.order12.insert({"customer_id":3,"order_id":101,"status":"Pending","order_date":"2017-01-03T00:00:00","sales_id":55,"order_detail":[{"product_name":"Corsair Dominator Platinum","quantity":138,"unit_price":719.99},{"product_name":"G.Skill Trident Z","quantity":134,"unit_price":704.99},{"product_name":"Corsair Vengeance LPX","quantity":47,"unit_price":699.01},{"product_name":"MSI X299 TOMAHAWK ARCTIC","quantity":36,"unit_price":281.98},{"product_name":"Samsung MZ-V6E250","quantity":134,"unit_price":127.88},{"product_name":"Kingston SA400S37/120G","quantity":52,"unit_price":54.99},{"product_name":"Intel Xeon E5-2687W V4","quantity":123,"unit_price":2042.69}],"order_sum_price":508078.3} )

db.order12.insert({"customer_id":6,"order_id":6,"status":"Shipped","order_date":"2015-04-09T00:00:00","sales_id":null,"order_detail":[{"product_name":"PNY VCGGTX780T3XPB-OC","quantity":150,"unit_price":749.99},{"product_name":"G.Skill Ripjaws V Series","quantity":119,"unit_price":680.99},{"product_name":"G.Skill Ripjaws V Series","quantity":41,"unit_price":645.99},{"product_name":"PNY SSD7CS1311-120-RB","quantity":145,"unit_price":57.98},{"product_name":"Western Digital WD2500AVVS","quantity":90,"unit_price":15.55},{"product_name":"Intel Xeon E5-2630 V4","quantity":67,"unit_price":647.99},{"product_name":"PNY VCQP5000-PB","quantity":74,"unit_price":2015.11}],"order_sum_price":422361.97} )

mongo --host mongo:9999 -u root -p root1122 --authenticationDatabase admin < /tmp/order12.json

Mongo Backup/Restore Scripts

★깜장별★ — Thu, 6 Aug 2020 16:53:55 +0900

Dump

Meta Data(JSON)와 Data(BSON)를 동시에 받는다.

전체를 받을 수 있고, 데이터 베이스, 콜렉션 단위로 받을 수도 있다.

백업 경로만 지정해 주면 데이터베이스 명으로 디렉터리를 만들고 Dump를 수행 함

Dump

Dump ALL Databases (dumpall.sh)

#!/bin/bash

server='localhost'

port=9999

user='root'

pwd='root1122'

backupdir=/tmp/backup/all/dump

/usr/bin/mongodump -h $server:$port -u $user -p $pwd --authenticationDatabase admin -o $backupdir

Dump ALL Databases (dumpall.sh)

#!/bin/bash

server='localhost'

port=9999

user='root'

pwd='root1122'

backupdir=/tmp/backup/all/dump

DBS=$(mongo $server:$port -u $user -p $pwd --authenticationDatabase admin --quiet --eval "db.adminCommand('listDatabases').databases.forEach(function (d) {

mdb = db.getSiblingDB(d.name);

print(mdb);

})")

for DB in $DBS; do

if [ $DB != "local" ]; then

echo "Exporting $DB ..."

/usr/bin/mongodump -h $server:$port -u $user -p $pwd --authenticationDatabase admin -d $DB -o $backupdir

done

Restore

Restore all Database (restoreall.sh)

#!/bin/bash

server='localhost'

port=9999

user='root'

pwd='root1122'

backupdir=/tmp/backup/all/dump

/usr/bin/mongorestore -h $server:$port -u $user -p $pwd --authenticationDatabase admin $backupdir

특정 데이터베이스만 Restore 하려면

/usr/bin/mongorestore -h $server:$port -u $user -p $pwd --authenticationDatabase admin -d 3db /tmp/backup/all/dump/3db

특정 컬렉션만 Restore 하려면

/usr/bin/mongorestore -h $server:$port -u $user -p $pwd --authenticationDatabase admin -d tpcc /tmp/backup/all/dump/tpcc/WAREHOUSE.bson

Export

데이터만 받으면서 Collection 레벨로만 받을 수 있다.

그래서 인덱스 생성 구문을 따로 추출해서 작업해 줘야 한다.

Export

Export All Database (exportall.sh)

#!/bin/bash

server='localhost'

port=9999

user='root'

pwd='root1122'

backupdir=/tmp/backup/all/export

DBS=$(mongo $server:$port -u $user -p $pwd --authenticationDatabase admin --quiet --eval "db.adminCommand('listDatabases').databases.forEach(function (d) {

mdb = db.getSiblingDB(d.name);

print(mdb);

})")

for DB in $DBS; do

COLLECTIONS=$(mongo $server:$port/$DB -u $user -p $pwd --authenticationDatabase admin --quiet --eval "db.getCollectionNames().join('\n')" )

for collection in $COLLECTIONS; do

if [ $DB != "admin" ]; then

echo "Exporting $DB/$collection ..."

/usr/bin/mongoexport -h $server:$port -u $user -p $pwd --authenticationDatabase admin -d $DB -c $collection -o $backupdir/$DB.$collection.json

done

Import

Import All Database (importall.sh)

#!/bin/bash

server='localhost'

port=8888

user='root'

pwd='root1122'

backupdir=/tmp/backup/all/export

FILES=$(ls $backupdir)

for file in $FILES; do

DB=`echo $file | cut -d'.' -f1`

coll=`echo $file | cut -d'.' -f2`

ext=`echo $file | cut -d'.' -f3`

if [ $ext != "json" ]; then

coll=`echo $file | cut -d'.' -f2`.`echo $file | cut -d'.' -f3`

ext=`echo $file | cut -d'.' -f4`

if [ $ext != "json" ]; then

coll=`echo $file | cut -d'.' -f2`.`echo $file | cut -d'.' -f3`.`echo $file | cut -d'.' -f4`

ext=`echo "$file" | cut -d'.' -f5`

if [ $ext != "json" ]; then

coll=`echo $file | cut -d'.' -f2`.`echo $file | cut -d'.' -f3`.`echo $file | cut -d'.' -f4`.`echo $file | cut -d'.' -f5`

echo "$DB $coll"

/usr/bin/mongoimport --host $server:$port -u $user -p $pwd --authenticationDatabase=admin --db $DB --collection $coll --file $backupdir/$file

done

MongoDB Check Script

★깜장별★ — Wed, 5 Aug 2020 14:22:06 +0900

Database, Collection, Index 정보 조회 (기본 인덱스인 _id_ 정보는 제외)

Database Info

dbcnt = 1

dbTotalcnt = db.getMongo().getDBNames().length

db.adminCommand({ "listDatabases": 1}).databases.forEach(function(d) {

mdb = db.getSiblingDB(d.name);

dbobj = mdb.stats(1024*1024);

if ( dbcnt == 1 ) {

print( '----------------------------------------------------------------------------------------------------------------------------------------------------') ;

print( ' Summary ') ;

print( '----------------------------------------------------------------------------------------------------------------------------------------------------') ;

}

print( ' DB : ' , dbobj.db.padEnd(20,' ') ,

' | Collections : ' , dbobj.collections ,

' | Views : ' , dbobj.views ,

' | Indexes : ' , dbobj.indexes ,

' | DataSize(M) : ' , dbobj.dataSize.toFixed(7) ,

' | IndexSize(M) : ' , dbobj.indexSize.toFixed(7));

if ( dbcnt == dbTotalcnt) {

print( '----------------------------------------------------------------------------------------------------------------------------------------------------') ;

}

dbcnt += 1 ;

});

출력 결과

----------------------------------------------------------------------------------------------------------------------------------------------------

Summary

----------------------------------------------------------------------------------------------------------------------------------------------------

----------------------------------------------------------------------------------------------------------------------------------------------------

Collection Info

db.adminCommand({ "listDatabases": 1}).databases.forEach(function(d) {

mdb = db.getSiblingDB(d.name);

print( ' ') ;

print( '--------------------' ) ;

print( ' ' , mdb ) ;

print( '--------------------' ) ;

mdb.getCollectionNames().forEach(function(col) {

colstat = mdb[col].stats(1024*1024) ;

colsize = colstat.size ;

colcnt = colstat.count ;

colindexcnt = colstat.nindexes ;

print( ' ' );

print( ' ' , col , '\n Size(M) : ' , colsize , ' documentcount : ' , colcnt , '\n IndexCount : ' , colindexcnt );

mdb[col].getIndexes().forEach(function(idx) {

iname = idx.name ;

ikey = tojson(idx.key) ;

if ( iname !== '_id_') {

print( ' - name : ' , iname , ' keys : ' , ikey);

}

});

출력결과

--------------------

100YWeatherSmall

--------------------

data

Size(M) : 327 documentcount : 250000

IndexCount : 1

--------------------

3db

--------------------

tab1

Size(M) : 0 documentcount : 1

IndexCount : 1

--------------------

admin

--------------------

system.profile

Size(M) : 0 documentcount : 1

IndexCount : 0

system.users

Size(M) : 0 documentcount : 4

IndexCount : 2

- name : user_1_db_1 keys : { "user" : 1, "db" : 1 }

system.version

Size(M) : 0 documentcount : 2

IndexCount : 1

--------------------

citibike

--------------------

trips

Size(M) : 834 documentcount : 1990273

IndexCount : 1

--------------------

city

--------------------

neighborhoods

Size(M) : 3 documentcount : 195

IndexCount : 1

--------------------

config

--------------------

system.sessions

Size(M) : 0 documentcount : 1

IndexCount : 2

- name : lsidTTLIndex keys : { "lastUse" : 1 }

--------------------

local

--------------------

startup_log

Size(M) : 0 documentcount : 51

IndexCount : 1

--------------------

m201

--------------------

people

Size(M) : 18 documentcount : 50474

IndexCount : 3

- name : ssn_1 keys : { "ssn" : 1 }

- name : ssn_1_birthday_1 keys : { "ssn" : 1, "birthday" : 1 }

restaurants

Size(M) : 173 documentcount : 1000000

IndexCount : 1

--------------------

m312

--------------------

sampleWorkload

Size(M) : 0 documentcount : 262

IndexCount : 2

- name : firstNumber_1 keys : { "firstNumber" : 1 }

system.profile

Size(M) : 0 documentcount : 791

IndexCount : 0

--------------------

results

--------------------

scores

Size(M) : 0 documentcount : 1127

IndexCount : 1

surveys

Size(M) : 0 documentcount : 424

IndexCount : 1

--------------------

scott

--------------------

dept

Size(M) : 0 documentcount : 4

IndexCount : 1

emp

Size(M) : 0 documentcount : 14

IndexCount : 1

empjoindept

Size(M) : 0 documentcount : 14

IndexCount : 1

salgrade

Size(M) : 0 documentcount : 5

IndexCount : 1

--------------------

ships

--------------------

shipwrecks

Size(M) : 3 documentcount : 11095

IndexCount : 1

--------------------

test

--------------------

arraytest

Size(M) : 0 documentcount : 2

IndexCount : 1

arraytest1

Size(M) : 0 documentcount : 4

IndexCount : 1

blog

Size(M) : 0 documentcount : 2

IndexCount : 1

colCapped

Size(M) : 0 documentcount : 4

IndexCount : 1

exists

Size(M) : 0 documentcount : 4

IndexCount : 1

intest

Size(M) : 0 documentcount : 3

IndexCount : 1

myMovies

Size(M) : 0 documentcount : 4

IndexCount : 1

retest

Size(M) : 0 documentcount : 2

IndexCount : 1

signs

Size(M) : 0 documentcount : 0

IndexCount : 2

- name : sign_text_1 keys : { "sign_text" : 1 }

sorttest

Size(M) : 0 documentcount : 5

IndexCount : 1

sorttest1

Size(M) : 0 documentcount : 3

IndexCount : 1

sorttest2

Size(M) : 0 documentcount : 4

IndexCount : 1

test

Size(M) : 7 documentcount : 74907

IndexCount : 3

- name : name_1_member_1 keys : { "name" : 1, "member" : 1 }

- name : member2_1 keys : { "member2" : 1 }

test1

Size(M) : 0 documentcount : 4

IndexCount : 1

text

Size(M) : 0 documentcount : 1

IndexCount : 2

- name : desc_text keys : { "_fts" : "text", "_ftsx" : 1 }

upsert

Size(M) : 0 documentcount : 4

IndexCount : 1

--------------------

testdb

--------------------

test1

Size(M) : 0 documentcount : 1

IndexCount : 1

testdbtab1

Size(M) : 0 documentcount : 3

IndexCount : 1

--------------------

tpcc

--------------------

CUSTOMER

Size(M) : 97 documentcount : 120000

IndexCount : 4

- name : C_W_ID_1_C_D_ID_1_C_ID_1 keys : { "C_W_ID" : 1, "C_D_ID" : 1, "C_ID" : 1 }

- name : C_W_ID_1_C_D_ID_1_C_LAST_1 keys : { "C_W_ID" : 1, "C_D_ID" : 1, "C_LAST" : 1 }

- name : C_CITY_1 keys : { "C_CITY" : 1 }

DISTRICT

Size(M) : 0 documentcount : 40

IndexCount : 2

- name : D_W_ID_1_D_ID_1_D_NEXT_O_ID_1_D_TAX_1 keys : { "D_W_ID" : 1, "D_ID" : 1, "D_NEXT_O_ID" : 1, "D_TAX" : 1 }

HISTORY

Size(M) : 17 documentcount : 120000

IndexCount : 1

ITEM

Size(M) : 14 documentcount : 100000

IndexCount : 2

- name : I_W_ID_1_I_ID_1 keys : { "I_W_ID" : 1, "I_ID" : 1 }

NEW_ORDER

Size(M) : 2 documentcount : 36000

IndexCount : 2

- name : NO_W_ID_1_NO_D_ID_1_NO_O_ID_1 keys : { "NO_W_ID" : 1, "NO_D_ID" : 1, "NO_O_ID" : 1 }

ORDERS

Size(M) : 178 documentcount : 120000

IndexCount : 3

- name : O_W_ID_1_O_D_ID_1_O_ID_1_O_C_ID_1 keys : { "O_W_ID" : 1, "O_D_ID" : 1, "O_ID" : 1, "O_C_ID" : 1 }

- name : O_C_ID_1_O_D_ID_1_O_W_ID_1_O_ID_-1_O_CARRIER_ID_1_O_ENTRY_ID_1 keys : {

"O_C_ID" : 1,

"O_D_ID" : 1,

"O_W_ID" : 1,

"O_ID" : -1,

"O_CARRIER_ID" : 1,

"O_ENTRY_ID" : 1

}

STOCK

Size(M) : 213 documentcount : 400000

IndexCount : 3

- name : S_W_ID_1_S_I_ID_1_S_QUANTITY_1 keys : { "S_W_ID" : 1, "S_I_ID" : 1, "S_QUANTITY" : 1 }

- name : S_I_ID_1 keys : { "S_I_ID" : 1 }

WAREHOUSE

Size(M) : 0 documentcount : 4

IndexCount : 2

- name : W_ID_1_W_TAX_1 keys : { "W_ID" : 1, "W_TAX" : 1 }

--------------------

video

--------------------

movieDetails

Size(M) : 1 documentcount : 2295

IndexCount : 1

reviews

Size(M) : 0 documentcount : 20

IndexCount : 1

Index Create Statement

indexcnt = 1

beforeDB = ''

db.adminCommand({ "listDatabases": 1}).databases.forEach(function(d) {

mdb = db.getSiblingDB(d.name);

if ( indexcnt == 1 ) {

print( '-------------------------------------------' );

print( ' Index Create Scrtipts (exclude _id_ index)' );

print( '-------------------------------------------' );

}

nowDB = mdb ;

if ( mdb == 'admin' ) { return true; }

if ( mdb == 'local' ) { return true; }

if ( mdb == 'config' ) { return true; }

mdb.getCollectionNames().forEach(function(col) {

mdb[col].getIndexes().forEach(function(idx) {

iname = idx.name ;

ikey = tojson(idx.key) ;

if ( iname !== '_id_') {

if ( beforeDB !== nowDB ) print(' DB : ' , mdb ) ;

print( ' use ' + mdb , '\n db.' + col + '.createIndex( ' + ikey + ' ) ');

beforeDB = mdb ;

}

});

indexcnt += 1 ;

});

출력 결과

-------------------------------------------

Index Create Scrtipts (exclude _id_ index)

-------------------------------------------

DB : m201

use m201

db.people.createIndex( { "ssn" : 1 } )

use m201

db.people.createIndex( { "ssn" : 1, "birthday" : 1 } )

DB : m312

use m312

db.sampleWorkload.createIndex( { "firstNumber" : 1 } )

DB : test

use test

db.signs.createIndex( { "sign_text" : 1 } )

use test

db.test.createIndex( { "name" : 1, "member" : 1 } )

use test

db.test.createIndex( { "member2" : 1 } )

use test

db.text.createIndex( { "_fts" : "text", "_ftsx" : 1 } )

DB : tpcc

use tpcc

db.CUSTOMER.createIndex( { "C_W_ID" : 1, "C_D_ID" : 1, "C_ID" : 1 } )

use tpcc

db.CUSTOMER.createIndex( { "C_W_ID" : 1, "C_D_ID" : 1, "C_LAST" : 1 } )

use tpcc

db.CUSTOMER.createIndex( { "C_CITY" : 1 } )

use tpcc

db.DISTRICT.createIndex( { "D_W_ID" : 1, "D_ID" : 1, "D_NEXT_O_ID" : 1, "D_TAX" : 1 } )

use tpcc

db.ITEM.createIndex( { "I_W_ID" : 1, "I_ID" : 1 } )

use tpcc

db.NEW_ORDER.createIndex( { "NO_W_ID" : 1, "NO_D_ID" : 1, "NO_O_ID" : 1 } )

use tpcc

db.ORDERS.createIndex( { "O_W_ID" : 1, "O_D_ID" : 1, "O_ID" : 1, "O_C_ID" : 1 } )

use tpcc

db.ORDERS.createIndex( {

"O_C_ID" : 1,

"O_D_ID" : 1,

"O_W_ID" : 1,

"O_ID" : -1,

"O_CARRIER_ID" : 1,

"O_ENTRY_ID" : 1

} )

use tpcc

db.STOCK.createIndex( { "S_W_ID" : 1, "S_I_ID" : 1, "S_QUANTITY" : 1 } )

use tpcc

db.STOCK.createIndex( { "S_I_ID" : 1 } )

use tpcc

db.WAREHOUSE.createIndex( { "W_ID" : 1, "W_TAX" : 1 } )

mysqlbackup 및 복구

★깜장별★ — Wed, 24 Jun 2020 15:25:24 +0900

1. 현재의 상태 확인

mysql> use test;

mysql> show tables ;

+----------------+

| Tables_in_test |

+----------------+

| tab1 |

+----------------+

mysql> select * from tab1 ;

+------+--------+

| id | name |

+------+--------+

| 1 | mysql |

| 2 | mysql |

| 3 | mysql |

| 4 | oracle |

+------+--------+

2. backup

[root@MySQL ~]# /engine/mysql-8.0/bin/mysqlbackup --defaults-file=/etc/my.cnf8.0 --user=root --password=root1122 --socket=/data/my8.0.19/mysql.sock --backup-dir=/tmp/backup/backup_2020-06-23 --compress backup

MySQL Enterprise Backup Ver 8.0.19-commercial for linux-glibc2.12 on x86_64 (MySQL Enterprise - Commercial)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Starting with following command line ...

/engine/mysql-8.0/bin/mysqlbackup --defaults-file=/etc/my.cnf8.0

--user=root --password=xxxxxxxx --socket=/data/my8.0.19/mysql.sock

--backup-dir=/tmp/backup/backup_2020-06-23 --compress backup

IMPORTANT: Please check that mysqlbackup run completes successfully.

At the end of a successful 'backup' run mysqlbackup

prints "mysqlbackup completed OK!".

200623 16:04:46 MAIN INFO: Starting to log actions.

200623 16:04:46 MAIN INFO: No SSL options specified.

200623 16:04:46 MAIN INFO: MySQL server version is '8.0.19-commercial'

200623 16:04:46 MAIN INFO: MySQL server compile os version is 'linux-glibc2.12'

200623 16:04:46 MAIN INFO: Got some server configuration information from running server.

200623 16:04:46 MAIN INFO: No SSL options specified.

200623 16:04:46 MAIN INFO: Backup directory created: '/tmp/backup/backup_2020-06-23'

200623 16:04:46 MAIN INFO: MySQL server version_comment is 'MySQL Enterprise Server - Commercial'

200623 16:04:46 MAIN INFO: Mysqlbackup component not installed.

200623 16:04:46 MAIN INFO: Server is not a community server.

200623 16:04:46 MAIN INFO: KEF target path:'/tmp/backup/backup_2020-06-23/meta/keyring_kef'

200623 16:04:46 MAIN INFO: TDE Keyring service initialized.

200623 16:04:46 MAIN INFO: MEB logfile created at /tmp/backup/backup_2020-06-23/meta/MEB_2020-06-23.16-04-46_compress_backup.log

--------------------------------------------------------------------

Server Repository Options:

--------------------------------------------------------------------

datadir = /data/my8.0.19/

innodb_data_home_dir =

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /data/my8.0.19/

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_undo_directory = /data/my8.0.19/

innodb_undo_tablespaces = 2

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = 16384

innodb_checksum_algorithm = crc32

--------------------------------------------------------------------

Backup Config Options:

--------------------------------------------------------------------

datadir = /tmp/backup/backup_2020-06-23/datadir

innodb_data_home_dir = /tmp/backup/backup_2020-06-23/datadir

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /tmp/backup/backup_2020-06-23/datadir

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_undo_directory = /tmp/backup/backup_2020-06-23/datadir

innodb_undo_tablespaces = 2

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = 16384

innodb_checksum_algorithm = crc32

200623 16:04:46 MAIN INFO: Unique generated backup id for this is 15928958867325335

200623 16:04:46 MAIN INFO: Uses LZ4 r109 for data compression.

200623 16:04:46 MAIN INFO: Copying the server config file '/data/my8.0.19/auto.cnf'

200623 16:04:46 MAIN INFO: Creating 20 buffers each of size 16794072.

200623 16:04:46 MAIN INFO: Failed to start redo log archiving...

200623 16:04:46 MAIN INFO: Found checkpoint at lsn 3694448807.

200623 16:04:46 MAIN INFO: Starting log scan from lsn = 3694448640 at offset = 20381696 and checkpoint = 3694448807 in file /data/my8.0.19/ib_logfile1.

200623 16:04:46 MAIN INFO: Compress Backup operation starts with following threads

1 read-threads 6 process-threads 1 write-threads

200623 16:04:46 MAIN INFO: Starting to copy all innodb files...

200623 16:04:46 MAIN INFO: No SSL options specified.

200623 16:04:47 RDR1 INFO: Copying /data/my8.0.19/ibdata1.

200623 16:04:47 RDR1 INFO: Starting to copy all undo files...

200623 16:04:47 RLP1 INFO: Starting to parse redo log at lsn = 3694448777, whereas checkpoint_lsn = 3694448807.

200623 16:04:47 RDR1 INFO: Copying /data/my8.0.19/undo_002.

200623 16:04:47 RDR1 INFO: Copying /data/my8.0.19/undo_001.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sys/sys_config.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/actor.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/address.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/category.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/city.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/country.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/customer.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/film.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/film_actor.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/film_category.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/film_text.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_being_deleted.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_being_deleted_cache.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_config.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_deleted.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_deleted_cache.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_00000000000000ab_index_1.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_00000000000000ab_index_2.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_00000000000000ab_index_3.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_00000000000000ab_index_4.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_00000000000000ab_index_5.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/fts_000000000000042c_00000000000000ab_index_6.ibd.

200623 16:04:48 RDR1 INFO: Copying /data/my8.0.19/sakila/inventory.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sakila/language.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sakila/payment.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sakila/rental.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sakila/staff.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sakila/store.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/world/city.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/world/country.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/world/countrylanguage.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/employees/employees.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/employees/departments.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/employees/dept_manager.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/employees/dept_emp.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/employees/titles.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/employees/salaries.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest9.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest11.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest4.ibd.

200623 16:04:49 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest8.ibd.

200623 16:04:50 RDR1 Progress in MB: 200

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest10.ibd.

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest6.ibd.

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest3.ibd.

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest1.ibd.

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest5.ibd.

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest14.ibd.

200623 16:04:50 RDR1 Progress in MB: 400

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest7.ibd.

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest2.ibd.

200623 16:04:50 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest12.ibd.

200623 16:04:51 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest15.ibd.

200623 16:04:51 RDR1 INFO: Copying /data/my8.0.19/sysbench/sbtest13.ibd.

200623 16:04:51 RDR1 Progress in MB: 600

200623 16:04:51 RDR1 INFO: Copying /data/my8.0.19/test/tab1.ibd.

200623 16:04:51 RDR1 INFO: Copying /data/my8.0.19/mysql/backup_progress.ibd.

200623 16:04:51 RDR1 INFO: Copying /data/my8.0.19/mysql.ibd.

200623 16:04:51 RDR1 INFO: Completing the copy of innodb files.

200623 16:04:51 RDR1 INFO: Requesting a dump of the InnoDB buffer pool

200623 16:04:51 RDR1 INFO: Waiting for the dump of the InnoDB buffer pool to complete

200623 16:04:51 RDR1 INFO: The dump of the InnoDB buffer pool completed

200623 16:04:51 RDR1 INFO: Binary Log Basename: '/data/my8.0.19/binlog'

200623 16:04:51 RDR1 INFO: Binary Log Index: '/data/my8.0.19/binlog.index'

200623 16:04:51 RDR1 INFO: Relay Channel: 'group_replication_applier'

200623 16:04:51 RDR1 INFO: Relay Log Basename: '/data/my8.0.19/MySQL-relay-bin-group_replication_applier'

200623 16:04:51 RDR1 INFO: Relay Log Index: '/data/my8.0.19/MySQL-relay-bin-group_replication_applier.index'

200623 16:04:51 RDR1 INFO: Relay Channel: 'group_replication_recovery'

200623 16:04:51 RDR1 INFO: Relay Log Basename: '/data/my8.0.19/MySQL-relay-bin-group_replication_applier-group_replication_recovery'

200623 16:04:51 RDR1 INFO: Relay Log Index: '/data/my8.0.19/MySQL-relay-bin-group_replication_recovery.index'

200623 16:04:51 RDR1 INFO: Starting to copy Binlog files.

200623 16:04:51 RDR1 INFO: Starting to lock instance for backup...

200623 16:04:51 RDR1 INFO: The server instance is locked for backup.

200623 16:04:51 RDR1 INFO: Requesting flush of redo log reading after LSN 3694540407.

200623 16:04:51 RDR1 INFO: Requesting flush of redo log processing after LSN 3694540407.

200623 16:04:51 RDR1 INFO: Waiting to read redo log LSN 3694540407, have 3694540307.

200623 16:04:51 RDR1 INFO: Completed flush of redo log reading after LSN 3694540407.

200623 16:04:51 RDR1 INFO: Waiting to process redo log LSN 3694540407, have 3694540307.

200623 16:04:51 RDR1 INFO: Completed flush of redo log processing after LSN 3694540407.

200623 16:04:53 RDR1 INFO: Starting to read-lock tables...

200623 16:04:53 RDR1 INFO: No tables to read-lock.

200623 16:04:53 RDR1 INFO: Opening backup source directory '/data/my8.0.19'

200623 16:04:53 RDR1 INFO: Starting to copy non-innodb files in subdirs of '/data/my8.0.19'

200623 16:04:53 RDR1 INFO: Copying the database directory 'employees'

200623 16:04:53 RDR1 INFO: Copying the database directory 'mysql'

200623 16:04:53 RDR1 INFO: Copying the database directory 'performance_schema'

200623 16:04:53 RDR1 INFO: Copying the database directory 'sakila'

200623 16:04:53 RDR1 INFO: Copying the database directory 'sys'

200623 16:04:53 RDR1 INFO: Copying the database directory 'sysbench'

200623 16:04:53 RDR1 INFO: Copying the database directory 'test'

200623 16:04:53 RDR1 INFO: Copying the database directory 'world'

200623 16:04:53 RDR1 INFO: Completing the copy of all non-innodb files.

200623 16:04:53 RDR1 INFO: Requesting consistency information...

200623 16:04:53 RDR1 INFO: Locked the consistency point for 11666 microseconds.

200623 16:04:53 RDR1 INFO: Consistency point server_uuid '995c9064-5776-11ea-93fc-080027323367'.

200623 16:04:53 RDR1 INFO: Consistency point gtid_executed ''.

200623 16:04:53 RDR1 INFO: Consistency point binary_log_file 'binlog.000014'.

200623 16:04:53 RDR1 INFO: Consistency point binary_log_position 155.

200623 16:04:53 RDR1 INFO: Consistency point InnoDB lsn 3694545284.

200623 16:04:53 RDR1 INFO: Consistency point InnoDB lsn_checkpoint 3694540417.

200623 16:04:53 RDR1 INFO: Requesting completion of redo log copy after LSN 3694545284.

200623 16:04:53 RLR1 INFO: Redo log reader waited 135 times for a total of 770.00 ms for logs to generate.

200623 16:04:53 RLW1 INFO: A copied database page was modified at 3694540211. (This is the highest lsn found on a page)

200623 16:04:53 RLW1 INFO: Scanned log up to lsn 3694545284.

200623 16:04:53 RLW1 INFO: Was able to parse the log up to lsn 3694545284.

200623 16:04:53 RDR1 INFO: Copying /data/my8.0.19/binlog.000014.

200623 16:04:53 RDR1 INFO: Completed the copy of binlog files...

200623 16:04:53 RDR1 INFO: The server instance is unlocked after 1.939 seconds.

200623 16:04:53 RDR1 INFO: Reading all global variables from the server.

200623 16:04:53 RDR1 INFO: Completed reading of all 555 global variables from the server.

200623 16:04:53 RDR1 INFO: Writing server defaults files 'server-my.cnf' and 'server-all.cnf' for server '8.0.19-commercial' in '/tmp/backup/backup_2020-06-23'.

200623 16:04:53 MAIN INFO: 634 MB of data files compressed to 326619 kbytes (compression 49.69%).

200623 16:04:53 MAIN INFO: Compress Backup operation completed successfully.

200623 16:04:53 MAIN INFO: Backup created in directory '/tmp/backup/backup_2020-06-23'

200623 16:04:53 MAIN INFO: MySQL binlog position: filename binlog.000014, position 155

-------------------------------------------------------------

Parameters Summary

-------------------------------------------------------------

Start LSN : 3694448640

End LSN : 3694545284

-------------------------------------------------------------

mysqlbackup completed OK!

백업 시점의 MySQL 로그 확인

[root@MySQL meta]# cat /tmp/backup/backup_2020-06-23/meta/backup_variables.txt

# This file is auto generated by mysqlbackup.

[backup_variables]

apply_log_done=0

binlog_index=binlog.index

binlog_position=binlog.000014:155

consistency_time_utc=1592895893150975

end_lsn=3694545284

end_time_utc=1592895893300158

has_tde_tables=0

is_compressed=1

is_incremental=0

is_incremental_with_redo_log_only=0

is_onlyinnodb=0

is_partial=0

is_skip_binlog=0

is_skip_relaylog=0

is_skip_unused_pages=0

last_checkpoint=3694540417

log_bin_index_name=binlog

log_bin_name=binlog

meb_version=8.0.19

mysql_version=8.0.19-commercial

mysql_version_comment=MySQL Enterprise Server - Commercial

start_lsn=3694448640

start_time_utc=159289588665723

3. DB에 새로운 트랜잭션 발생

Insert문 수행

mysql> insert into tab1 select * from tab1 ;

.......

mysql> insert into tab1 select * from tab1 ;

mysql> select count(*) from tab1 ;

+----------+

| count(*) |

+----------+

| 1048576 |

+----------+

1 row in set (0.07 sec)

4. DB 장애 발생 (시나리오)

DB 종료

최신의 데이터까지 완전 복구를 하기 위해서 binlog.000014이후의 binary log 파일을 다른 곳으로 copy

현재 시나리오에서는 로그가 binlog.000014만 있으므로 해당 파일만 copy

cp /data/my8.0.19/binlog.000014 /tmp/backup/

5. 백업 파일을 이용한 복구

BACKUP 받은 파일에 백업 받는 시점에 발생했던 트랜잭션 로그 적용

[root@MySQL backup]# /engine/mysql-8.0/bin/mysqlbackup --defaults-file=/etc/my.cnf8.0 --backup-dir=/tmp/backup/backup_2020-06-23 --uncompress apply-log

MySQL Enterprise Backup Ver 8.0.19-commercial for linux-glibc2.12 on x86_64 (MySQL Enterprise - Commercial)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Starting with following command line ...

/engine/mysql-8.0/bin/mysqlbackup --defaults-file=/etc/my.cnf8.0

--backup-dir=/tmp/backup/backup_2020-06-23 --uncompress apply-log

IMPORTANT: Please check that mysqlbackup run completes successfully.

At the end of a successful 'apply-log' run mysqlbackup

prints "mysqlbackup completed OK!".

200623 16:17:50 MAIN INFO: Starting to log actions.

200623 16:17:50 MAIN INFO: MySQL server version is '8.0.19-commercial'

200623 16:17:50 MAIN INFO: Restoring ...8.0.19-commercial version

200623 16:17:50 MAIN INFO: Server is not a community server.

200623 16:17:50 MAIN INFO: KEF source path:'/tmp/backup/backup_2020-06-23/meta'

200623 16:17:50 MAIN INFO: KEF target path:'/tmp/backup/backup_2020-06-23/meta'

200623 16:17:50 MAIN INFO: TDE Keyring service initialized.

200623 16:17:50 MAIN INFO: MEB logfile created at /tmp/backup/backup_2020-06-23/meta/MEB_2020-06-23.16-17-50_uncompress_apply_log.log

--------------------------------------------------------------------

Backup Config Options:

--------------------------------------------------------------------

datadir = /tmp/backup/backup_2020-06-23/datadir

innodb_data_home_dir = /tmp/backup/backup_2020-06-23/datadir

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /tmp/backup/backup_2020-06-23/datadir

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_undo_directory = /tmp/backup/backup_2020-06-23/datadir

innodb_undo_tablespaces = 2

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = 16384

innodb_checksum_algorithm = crc32

200623 16:17:50 MAIN INFO: Creating 20 buffers each of size 16794072.

200623 16:17:50 MAIN INFO: Uncompress operation starts with following threads

1 read-threads 6 process-threads 1 write-threads

200623 16:17:50 MAIN INFO: Connection to server is not available for query 'SHOW ENGINE INNODB STATUS'

200623 16:17:50 MAIN INFO: Cannot get redo log flush status

200623 16:17:50 MAIN INFO: Starting to copy all innodb files...

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/ibdata1.ibz.

200623 16:17:50 RDR1 INFO: Starting to copy all undo files...

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/undo_002.uz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/undo_001.uz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/mysql.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/employees/departments.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/employees/dept_emp.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/employees/dept_manager.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/employees/employees.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/employees/salaries.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/employees/titles.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/mysql/backup_progress.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/actor.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/address.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/category.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/city.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/country.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/customer.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/film.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/film_actor.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/film_category.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/film_text.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_1.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_2.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_3.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_4.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_5.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_6.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_being_deleted.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_being_deleted_cache.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_config.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_deleted.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_deleted_cache.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/inventory.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/language.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/payment.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/rental.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/staff.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sakila/store.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sys/sys_config.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest1.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest10.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest11.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest12.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest13.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest14.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest15.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest2.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest3.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest4.ibz.

200623 16:17:50 RDR1 Progress in MB: 200

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest5.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest6.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest7.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest8.ibz.

200623 16:17:50 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest9.ibz.

200623 16:17:51 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/test/tab1.ibz.

200623 16:17:51 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/world/city.ibz.

200623 16:17:51 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/world/country.ibz.

200623 16:17:51 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/world/countrylanguage.ibz.

200623 16:17:51 RDR1 INFO: Binary Log Basename: ''

200623 16:17:51 RDR1 INFO: Binary Log Index: '/tmp/backup/backup_2020-06-23/datadir/binlog.index'

200623 16:17:51 RDR1 INFO: Starting to copy Binlog files.

200623 16:17:51 RDR1 INFO: Uncompressing /tmp/backup/backup_2020-06-23/datadir/binlog.000014.bz.

200623 16:17:51 RDR1 INFO: Completed the copy of binlog files...

200623 16:17:51 WTR1 INFO: Compressed files removed successfully

200623 16:17:51 MAIN INFO: Uncompress operation completed successfully.

200623 16:17:51 MAIN INFO: Creating 14 buffers each of size 65680.

200623 16:17:51 MAIN INFO: Apply-log operation starts with following threads

1 read-threads 1 process-threads 6 apply-threads

200623 16:17:51 MAIN INFO: Using up to 400 MB of memory.

200623 16:17:51 MAIN INFO: ibbackup_logfile's creation parameters:

start lsn 3694448640, end lsn 3694545284,

start checkpoint 3694448807.

200623 16:17:51 MAIN INFO: Loading the space id : 0, space name : /tmp/backup/backup_2020-06-23/datadir/ibdata1.

200623 16:17:51 MAIN INFO: Apply log: Loading the undo space id : 4294967278, space name : /tmp/backup/backup_2020-06-23/datadir/undo_002.

200623 16:17:51 MAIN INFO: Loading the space id : 4294967278, space name : /tmp/backup/backup_2020-06-23/datadir/undo_002.

200623 16:17:51 MAIN INFO: Apply log: Loading the undo space id : 4294967279, space name : /tmp/backup/backup_2020-06-23/datadir/undo_001.

200623 16:17:51 MAIN INFO: Loading the space id : 4294967279, space name : /tmp/backup/backup_2020-06-23/datadir/undo_001.

200623 16:17:51 MAIN INFO: Loading the space id : 33, space name : /tmp/backup/backup_2020-06-23/datadir/employees/departments.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 35, space name : /tmp/backup/backup_2020-06-23/datadir/employees/dept_emp.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 34, space name : /tmp/backup/backup_2020-06-23/datadir/employees/dept_manager.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 32, space name : /tmp/backup/backup_2020-06-23/datadir/employees/employees.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 37, space name : /tmp/backup/backup_2020-06-23/datadir/employees/salaries.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 36, space name : /tmp/backup/backup_2020-06-23/datadir/employees/titles.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 84, space name : /tmp/backup/backup_2020-06-23/datadir/mysql/backup_progress.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 2, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/actor.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 3, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/address.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 4, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/category.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 5, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/city.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 6, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/country.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 7, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/customer.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 8, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/film.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 9, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/film_actor.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 10, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/film_category.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 11, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/film_text.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 17, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_1.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 18, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_2.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 19, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_3.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 20, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_4.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 21, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_5.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 22, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_6.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 12, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_being_deleted.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 13, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_being_deleted_cache.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 14, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_config.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 15, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_deleted.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 16, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_deleted_cache.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 23, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/inventory.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 24, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/language.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 25, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/payment.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 26, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/rental.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 27, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/staff.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 28, space name : /tmp/backup/backup_2020-06-23/datadir/sakila/store.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 1, space name : /tmp/backup/backup_2020-06-23/datadir/sys/sys_config.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 73, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest1.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 70, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest10.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 67, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest11.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 78, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest12.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 80, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest13.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 75, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest14.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 79, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest15.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 77, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest2.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 72, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest3.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 68, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest4.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 74, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest5.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 71, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest6.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 76, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest7.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 69, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest8.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 66, space name : /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest9.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 83, space name : /tmp/backup/backup_2020-06-23/datadir/test/tab1.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 29, space name : /tmp/backup/backup_2020-06-23/datadir/world/city.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 30, space name : /tmp/backup/backup_2020-06-23/datadir/world/country.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 31, space name : /tmp/backup/backup_2020-06-23/datadir/world/countrylanguage.ibd.

200623 16:17:51 MAIN INFO: Loading the space id : 4294967294, space name : /tmp/backup/backup_2020-06-23/datadir/mysql.ibd.

200623 16:17:51 PCR1 INFO: Starting to parse redo log at lsn = 3694448777, whereas checkpoint_lsn = 3694448807.

200623 16:17:51 PCR1 INFO: Doing recovery: scanned up to log sequence number 3694545284.

200623 16:17:51 PCR1 INFO: Starting to apply a batch of log records to the database....

InnoDB: Progress in percent: 1 2 4 5 6

200623 16:17:51 PCR1 INFO: Updating last checkpoint to 3694540417 in redo log/

200623 16:17:51 PCR1 INFO: Setting 'ib_logfile0' file size to 50331648

200623 16:17:51 PCR1 INFO: Setting 'ib_logfile1' file size to 50331648

200623 16:17:51 PCR1 INFO: Log file header:

format = 3

pad1 = 0

start lsn = 3694540288

checkpoint lsn = 3694540417

checksum = 3688586621

creator = MEB 8.0.19

200623 16:17:51 PCR1 INFO: We were able to parse ibbackup_logfile up to

lsn 3694545284.

200623 16:17:51 PCR1 INFO: Last MySQL binlog file position 0 155, file name binlog.000014

200623 16:17:51 PCR1 INFO: The first data file is '/tmp/backup/backup_2020-06-23/datadir/ibdata1'

and the new created log files are at '/tmp/backup/backup_2020-06-23/datadir'

200623 16:17:51 MAIN INFO: Apply-log operation completed successfully.

200623 16:17:51 MAIN INFO: Full backup prepared for recovery successfully.

mysqlbackup completed OK!

datadir 디렉터리 내용 삭제

[root@MySQL backup]# cd /data/my8.0.19

[root@MySQL my8.0.19]# ls

auto.cnf binlog.index ca.pem client-key.pem error.log ibdata1 ib_logfile1 mysql performance_schema public_key.pem server-cert.pem sys test undo_002

binlog.000014 ca-key.pem client-cert.pem employees ib_buffer_pool ib_logfile0 #innodb_temp mysql.ibd private_key.pem sakila server-key.pem sysbench undo_001 world

[root@MySQL my8.0.19]# rm -rf *

백업 파일 copy back

[root@MySQL my8.0.19]# /engine/mysql-8.0/bin/mysqlbackup --defaults-file=/etc/my.cnf8.0 --backup-dir=/tmp/backup/backup_2020-06-23 copy-back

MySQL Enterprise Backup Ver 8.0.19-commercial for linux-glibc2.12 on x86_64 (MySQL Enterprise - Commercial)

Oracle is a registered trademark of Oracle Corporation and/or its

affiliates. Other names may be trademarks of their respective

owners.

Starting with following command line ...

/engine/mysql-8.0/bin/mysqlbackup --defaults-file=/etc/my.cnf8.0

--backup-dir=/tmp/backup/backup_2020-06-23 copy-back

IMPORTANT: Please check that mysqlbackup run completes successfully.

At the end of a successful 'copy-back' run mysqlbackup

prints "mysqlbackup completed OK!".

200623 16:21:24 MAIN INFO: Starting to log actions.

200623 16:21:24 MAIN INFO: MySQL server version is '8.0.19-commercial'

200623 16:21:24 MAIN INFO: Restoring ...8.0.19-commercial version

200623 16:21:24 MAIN WARNING: If you restore to a server of a different version, the innodb_data_file_path parameter might have a different default. In that case you need to add 'innodb_data_file_path=ibdata1:12M:autoextend' to the target server configuration.

200623 16:21:24 MAIN WARNING: If you restore to a server of a different version, the innodb_log_files_in_group parameter might have a different default. In that case you need to add 'innodb_log_files_in_group=2' to the target server configuration.

200623 16:21:24 MAIN WARNING: If you restore to a server of a different version, the innodb_log_file_size parameter might have a different default. In that case you need to add 'innodb_log_file_size=50331648' to the target server configuration.

200623 16:21:24 MAIN INFO: Server is not a community server.

200623 16:21:24 MAIN INFO: KEF source path:'/tmp/backup/backup_2020-06-23/meta'

200623 16:21:24 MAIN INFO: KEF target path:'/data/my8.0.19'

200623 16:21:24 MAIN INFO: TDE Keyring service initialized.

200623 16:21:24 MAIN INFO: MEB logfile created at /tmp/backup/backup_2020-06-23/meta/MEB_2020-06-23.16-21-24_copy_back.log

--------------------------------------------------------------------

Server Repository Options:

--------------------------------------------------------------------

datadir = /data/my8.0.19

innodb_data_home_dir = /data/my8.0.19

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /data/my8.0.19

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_undo_directory = /data/my8.0.19

innodb_undo_tablespaces = 2

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = Null

innodb_checksum_algorithm = crc32

--------------------------------------------------------------------

Backup Config Options:

--------------------------------------------------------------------

datadir = /tmp/backup/backup_2020-06-23/datadir

innodb_data_home_dir = /tmp/backup/backup_2020-06-23/datadir

innodb_data_file_path = ibdata1:12M:autoextend

innodb_log_group_home_dir = /tmp/backup/backup_2020-06-23/datadir

innodb_log_files_in_group = 2

innodb_log_file_size = 50331648

innodb_undo_directory = /tmp/backup/backup_2020-06-23/datadir

innodb_undo_tablespaces = 2

innodb_buffer_pool_filename = ib_buffer_pool

innodb_page_size = 16384

innodb_checksum_algorithm = crc32

200623 16:21:24 MAIN INFO: Creating 14 buffers each of size 16777216.

200623 16:21:24 MAIN INFO: Copy-back operation starts with following threads

1 read-threads 1 write-threads

200623 16:21:24 MAIN INFO: Connection to server is not available for query 'SHOW ENGINE INNODB STATUS'

200623 16:21:24 MAIN INFO: Cannot get redo log flush status

200623 16:21:24 MAIN INFO: Starting to copy all innodb files...

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/ibdata1.

200623 16:21:24 RDR1 INFO: Starting to copy all undo files...

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/undo_002.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/undo_001.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/mysql.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/employees/departments.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/employees/dept_emp.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/employees/dept_manager.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/employees/employees.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/employees/salaries.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/employees/titles.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/mysql/backup_progress.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/actor.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/address.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/category.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/city.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/country.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/customer.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/film.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/film_actor.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/film_category.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/film_text.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_1.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_2.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_3.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_4.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_5.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_00000000000000ab_index_6.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_being_deleted.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_being_deleted_cache.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_config.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_deleted.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/fts_000000000000042c_deleted_cache.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/inventory.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/language.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/payment.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/rental.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/staff.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sakila/store.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sys/sys_config.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest1.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest10.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest11.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest12.ibd.

200623 16:21:24 RDR1 Progress in MB: 200

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest13.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest14.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest15.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest2.ibd.

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest3.ibd.

200623 16:21:24 RDR1 Progress in MB: 400

200623 16:21:24 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest4.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest5.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest6.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest7.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest8.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/sysbench/sbtest9.ibd.

200623 16:21:25 RDR1 Progress in MB: 600

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/test/tab1.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/world/city.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/world/country.ibd.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/world/countrylanguage.ibd.

200623 16:21:25 RDR1 INFO: Binary Log Basename: ''

200623 16:21:25 RDR1 INFO: Binary Log Index: '/tmp/backup/backup_2020-06-23/datadir/binlog.index'

200623 16:21:25 RDR1 INFO: Starting to copy Binlog files.

200623 16:21:25 RDR1 INFO: Copying /tmp/backup/backup_2020-06-23/datadir/binlog.000014.

200623 16:21:25 RDR1 INFO: Copying the database directory 'employees'

200623 16:21:25 RDR1 INFO: Copying the database directory 'mysql'

200623 16:21:25 RDR1 INFO: Copying the database directory 'performance_schema'

200623 16:21:25 RDR1 INFO: Copying the database directory 'sakila'

200623 16:21:25 RDR1 INFO: Copying the database directory 'sys'

200623 16:21:25 RDR1 INFO: Copying the database directory 'sysbench'

200623 16:21:25 RDR1 INFO: Copying the database directory 'test'

200623 16:21:25 RDR1 INFO: Copying the database directory 'world'

200623 16:21:25 RDR1 INFO: Completing the copy of all non-innodb files.

200623 16:21:25 RDR1 INFO: Completed the copy of binlog files...

200623 16:21:25 RDR1 INFO: Copying the log file 'ib_logfile0'

200623 16:21:25 RDR1 INFO: Copying the log file 'ib_logfile1'

200623 16:21:25 MAIN INFO: No Keyring file to process.

200623 16:21:25 MAIN INFO: Copying the server config file '/tmp/backup/backup_2020-06-23/datadir/backup-auto.cnf'

200623 16:21:25 MAIN INFO: Copy-back operation completed successfully.

200623 16:21:25 MAIN INFO: Finished copying backup files to '/data/my8.0.19'

mysqlbackup completed OK! with 3 warnings

6. datadir 디렉터리 권한 변경

[root@MySQL my8.0.19]# cd /data/my8.0.19

[root@MySQL my8.0.19]# ls -al

total 168000

drwxr-xr-x 10 mysql mysql 4096 Jun 23 16:21 .

drwxr-xr-x 7 mysql mysql 86 Mar 18 11:09 ..

-rw-r----- 1 root root 56 Jun 23 16:21 backup-auto.cnf

-rw-r--r-- 1 root root 721 Jun 23 16:21 backup_variables.txt

-rw-r----- 1 root root 155 Jun 23 16:21 binlog.000014

-rw-r----- 1 root root 16 Jun 23 16:21 binlog.index

drwxr-x--- 2 root root 132 Jun 23 16:21 employees

-rw-r----- 1 root root 3853 Jun 23 16:21 ib_buffer_pool

-rw-r----- 1 root root 12582912 Jun 23 16:21 ibdata1

-rw-r----- 1 root root 50331648 Jun 23 16:21 ib_logfile0

-rw-r----- 1 root root 50331648 Jun 23 16:21 ib_logfile1

drwxr-x--- 2 root root 170 Jun 23 16:21 mysql

-rw-r----- 1 root root 25165824 Jun 23 16:21 mysql.ibd

drwxr-x--- 2 root root 8192 Jun 23 16:21 performance_schema

drwxr-x--- 2 root root 4096 Jun 23 16:21 sakila

-rw-r----- 1 root root 17179 Jun 23 16:21 server-all.cnf

-rw-r----- 1 root root 813 Jun 23 16:21 server-my.cnf

drwxr-x--- 2 root root 28 Jun 23 16:21 sys

drwxr-x--- 2 root root 297 Jun 23 16:21 sysbench

drwxr-x--- 2 root root 22 Jun 23 16:21 test

-rw-r----- 1 root root 16777216 Jun 23 16:21 undo_001

-rw-r----- 1 root root 16777216 Jun 23 16:21 undo_002

drwxr-x--- 2 root root 68 Jun 23 16:21 world

권한이 모두 root로 되어 있으니 mysql 유저로 변경

chown -R mysql:mysql /data/my8.0.19

7. DB기동 후 데이터 확인

DB 기동 후 test DB 확인

mysql> use test ;

mysql> show tables ;

+----------------+

| Tables_in_test |

+----------------+

| tab1 |

+----------------+

mysql> select * from tab1 ;

+------+--------+

| id | name |

+------+--------+

| 1 | mysql |

| 2 | mysql |

| 3 | mysql |

| 4 | oracle |

+------+--------+

백업 받는 시점의 데이터로만 복구 됨.

8. 완전 복구 수행

완전복구를 하기 위해서 copy해 놓았던 binlog.000014 파일을 이용해서 복구 진행 binlog.000014에서 포지션은 155 부터만 적용하면 되고 만약에 binlog.000015가 있으면 전체 내용을 복구 하면 되므로 start-position은 필요 없음

SQL로 추출

/engine/mysql-8.0/bin/mysqlbinlog --start-position=155 /tmp/backup/binlog.000014 > /tmp/backup/backup.sql

추출하면서 바로 적용

/engine/mysql-8.0/bin/mysqlbinlog --start-position=155 /tmp/backup/binlog.000014 | /engine/mysql-8.0/bin/mysql -h 127.0.0.1 -P 3380 -u root -p

9. 복구 데이터 확인

데이터 확인

mysql> select count(*) from tab1 ;

+----------+

| count(*) |

+----------+

| 1048576 |

+----------+