IT STUDY LOG
Sprint - 성능 테스트를 위한 서비스 모니터링(advanced) 본문
#학습 목표
- Prometheus와 Grafana는 오픈소스로 제공되는 모니터링 서비스로서 많은 개발자들에게 큰 인기를 얻고 있음.
- 이와 마찬가지로 k6 역시 오픈소스로 제공되는 툴로서 소프트웨어 엔지니어링 커뮤니티에서 아주 큰 인기를 얻고 있으며, 특히 개발자 친화적으로 설계가 되어 있고 스크립트 언어로 javascript를 사용하므로 개발자들이 성능 테스트를 쉽게 작성하고 유지 가능
=> 이러한 두 서비스의 장점을 살려서 성능 테스트를 모니터링
- 기본적인 K6 성능테스트 스크립트를 작성
- K6 output 및 output 을 위한 k6 extention을 이해
- K6 ouput을 위한 도구로 프로메테우스를 사용하면서 프로메테우스를 익힘
- k6로부터 나온 output을 그라파나로 시각화
#과제 항목별 진행 상황
✏️ Set up & Preparation
Step 1 : k6확장 리포지토리를 clone
# ssh 접속으로 clone
$ git clone git@github.com:grafana/xk6-output-prometheus-remote.git
# HTTPS 접속으로 clone
$ git clone https://github.com/grafana/xk6-output-prometheus-remote.git
Step 2 : 성능테스트를 위한 tests 폴더를 clone해 온 폴더 안에 생성
$ cd xk6-output-prometheus-remote
$ mkdir tests
Step 3 : tests 폴더 안에 k6 성능 테스트를 위한 basic_k6_test.js 파일을 생성
// basic_k6_test.js
import http from 'k6/http';
import { describe } from 'https://jslib.k6.io/expect/0.0.4/index.js';
import { check, sleep } from 'k6';
export default function () {
describe('Basic k6 test', () => {
let res = http.get('https://test.k6.io');
check(res, {
'is status 200': (r) => r.status === 200
});
sleep(1);
});
}
Step 4 : 상위 폴더로 돌아가 Dockerfile 확인
# Multi-stage build to generate custom k6 with extension
FROM golang:1.20-alpine as builder # 별칭을 builder로
WORKDIR $GOPATH/src/go.k6.io/k6
COPY . .
RUN apk --no-cache add git=~2 # apk : alpine의 패키지 매니저
RUN CGO_ENABLED=0 go install go.k6.io/xk6/cmd/xk6@latest \
&& CGO_ENABLED=0 xk6 build \
--with github.com/grafana/xk6-output-prometheus-remote=. \
--output /tmp/k6
# Create image for running k6 with output for Prometheus Remote Write
FROM alpine:3.17
# hadolint ignore=DL3018
RUN apk add --no-cache ca-certificates && \
adduser -D -u 12345 -g 12345 k6
COPY --from=builder /tmp/k6 /usr/bin/k6
USER 12345
WORKDIR /home/k6
ENTRYPOINT ["k6"]
- 도커 파일 내용 참고 : 도커허브, k6 공식사이트
Step 5 : docker-compose.yaml 파일 작성
version: '3.8'
volumes: # 로컬에 생성할 volumes
prometheus-data:
grafana-data-storage:
networks:
k6:
grafana:
prometheus:
services:
prometheus:
image: prom/prometheus:v2.42.0
command:
- --web.enable-remote-write-receiver
- --enable-feature=native-histograms
- --config.file=/etc/prometheus/prometheus.yml
- --enable-feature=remote-write-receiver --config.file=/etc/prometheus/prometheus.yaml
networks:
- k6
- grafana
- prometheus
ports:
- "9090:9090"
depends_on:
- grafana
volumes:
- ./prometheus.yml:/etc/prometheus/prometheus.yaml
- prometheus-data:/prometheus
grafana:
image: grafana/grafana:9.4.7
networks:
- grafana
- prometheus
ports:
- "3000:3000"
environment:
- GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
- GF_AUTH_ANONYMOUS_ENABLED=true
- GF_AUTH_BASIC_ENABLED=false
volumes:
- ./grafana:/etc/grafana/provisioning/
- grafana-data-storage:/var/lib/grafana
k6:
build: .
networks:
- k6
ports:
- "6565:6565"
environment:
- K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write
- K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true
- K6_OUT=xk6-prometheus-rw
depends_on:
- prometheus
- grafana
volumes:
- ./samples:/scripts
- .:/app:delegated
# k6 기동시 작성했던 basic_k6_test.js 파일이 실행되도록 커맨드 실행
command: run --vus 150 --duration 30s -o xk6-prometheus-rw /app/tests/basic_k6_test.js
Step 6 : prometheus.yaml 파일 작성
# prometheus.yml
global:
scrape_interval: 30s
scrape_timeout: 10s
scrape_configs:
- job_name: services
metrics_path: /metrics
static_configs:
- targets:
- 'prometheus:9090'
✏️ 프로메테우스와 그라파나에서 성능 테스트 내역 확인
Step 1 : 도커 실행
$ docker compose up
# 백그라운드 실행시
$ docker compose up -d
# 일부만 실행할 시
$ docker compose up prometheus grafana
$ docker compose up k6
- k6 테스트 수행 로그
더보기
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | /\ |‾‾| /‾‾/ /‾‾/
xk6-output-prometheus-remote-k6-1 | /\ / \ | |/ / / /
xk6-output-prometheus-remote-k6-1 | / \/ \ | ( / ‾‾\
xk6-output-prometheus-remote-k6-1 | / \ | |\ \ | (‾) |
xk6-output-prometheus-remote-k6-1 | / __________ \ |__| \__\ \_____/ .io
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | execution: local
xk6-output-prometheus-remote-k6-1 | script: /app/tests/basic_k6_test.js
xk6-output-prometheus-remote-k6-1 | output: Prometheus remote write (http://prometheus:9090/api/v1/write)
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | scenarios: (100.00%) 1 scenario, 150 max VUs, 1m0s max duration (incl. graceful stop):
xk6-output-prometheus-remote-k6-1 | * default: 150 looping VUs for 30s (gracefulStop: 30s)
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m00.1s), 150/150 VUs, 0 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 0% ] 150 VUs 00.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m01.1s), 150/150 VUs, 0 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 4% ] 150 VUs 01.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m02.1s), 150/150 VUs, 53 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 7% ] 150 VUs 02.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m03.1s), 150/150 VUs, 153 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 10% ] 150 VUs 03.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m04.1s), 150/150 VUs, 300 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 14% ] 150 VUs 04.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m05.1s), 150/150 VUs, 450 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 17% ] 150 VUs 05.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m06.1s), 150/150 VUs, 558 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 20% ] 150 VUs 06.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m07.1s), 150/150 VUs, 658 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 24% ] 150 VUs 07.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m08.1s), 150/150 VUs, 781 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 27% ] 150 VUs 08.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m09.1s), 150/150 VUs, 900 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 30% ] 150 VUs 09.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m10.1s), 150/150 VUs, 1050 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 34% ] 150 VUs 10.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m11.1s), 150/150 VUs, 1161 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 37% ] 150 VUs 11.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m12.1s), 150/150 VUs, 1273 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 40% ] 150 VUs 12.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m13.1s), 150/150 VUs, 1398 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 44% ] 150 VUs 13.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m14.1s), 150/150 VUs, 1511 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 47% ] 150 VUs 14.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m15.1s), 150/150 VUs, 1650 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 50% ] 150 VUs 15.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m16.1s), 150/150 VUs, 1786 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 54% ] 150 VUs 16.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m17.1s), 150/150 VUs, 1876 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 57% ] 150 VUs 17.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m18.1s), 150/150 VUs, 2002 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 60% ] 150 VUs 18.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m19.1s), 150/150 VUs, 2124 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 64% ] 150 VUs 19.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m20.1s), 150/150 VUs, 2253 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 67% ] 150 VUs 20.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m21.1s), 150/150 VUs, 2365 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 70% ] 150 VUs 21.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m22.1s), 150/150 VUs, 2485 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 74% ] 150 VUs 22.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m23.1s), 150/150 VUs, 2606 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 77% ] 150 VUs 23.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m24.1s), 150/150 VUs, 2733 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 80% ] 150 VUs 24.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m25.1s), 150/150 VUs, 2863 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 84% ] 150 VUs 25.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m26.1s), 150/150 VUs, 2985 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 87% ] 150 VUs 26.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m27.1s), 150/150 VUs, 3109 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 90% ] 150 VUs 27.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m28.1s), 150/150 VUs, 3223 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 94% ] 150 VUs 28.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m29.1s), 150/150 VUs, 3344 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default [ 97% ] 150 VUs 29.1s/30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m30.1s), 143/150 VUs, 3470 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default ↓ [ 100% ] 150 VUs 30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m31.1s), 011/150 VUs, 3602 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default ↓ [ 100% ] 150 VUs 30s
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | █ Basic k6 test
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | ✓ is status 200
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | checks.........................: 100.00% ✓ 3613 ✗ 0
xk6-output-prometheus-remote-k6-1 | data_received..................: 42 MB 1.4 MB/s
xk6-output-prometheus-remote-k6-1 | data_sent......................: 405 kB 13 kB/s
xk6-output-prometheus-remote-k6-1 | group_duration.................: avg=1.27s min=1.19s med=1.21s max=2.33s p(90)=1.39s p(95)=1.4s
xk6-output-prometheus-remote-k6-1 | http_req_blocked...............: avg=28.46ms min=2.91µs med=7.56µs max=792.34ms p(90)=17.89µs p(95)=69.27µs
xk6-output-prometheus-remote-k6-1 | http_req_connecting............: avg=9.39ms min=0s med=0s max=253.98ms p(90)=0s p(95)=0s
xk6-output-prometheus-remote-k6-1 | http_req_duration..............: avg=241.87ms min=178.39ms med=209.14ms max=576.4ms p(90)=389.24ms p(95)=402.86ms
xk6-output-prometheus-remote-k6-1 | { expected_response:true }...: avg=241.87ms min=178.39ms med=209.14ms max=576.4ms p(90)=389.24ms p(95)=402.86ms
xk6-output-prometheus-remote-k6-1 | http_req_failed................: 0.00% ✓ 0 ✗ 3613
xk6-output-prometheus-remote-k6-1 | http_req_receiving.............: avg=14.83ms min=-16607384ns med=194.68µs max=222.45ms p(90)=11.02ms p(95)=187.99ms
xk6-output-prometheus-remote-k6-1 | http_req_sending...............: avg=414.76µs min=10.95µs med=33.21µs max=58.02ms p(90)=115.31µs p(95)=217.52µs
xk6-output-prometheus-remote-k6-1 | http_req_tls_handshaking.......: avg=15.15ms min=0s med=0s max=459.9ms p(90)=0s p(95)=0s
xk6-output-prometheus-remote-k6-1 | http_req_waiting...............: avg=226.62ms min=177.51ms med=207.14ms max=565.9ms p(90)=274.12ms p(95)=359.74ms
xk6-output-prometheus-remote-k6-1 | http_reqs......................: 3613 115.416168/s
xk6-output-prometheus-remote-k6-1 | iteration_duration.............: avg=1.27s min=1.19s med=1.21s max=2.33s p(90)=1.39s p(95)=1.4s
xk6-output-prometheus-remote-k6-1 | iterations.....................: 3613 115.416168/s
xk6-output-prometheus-remote-k6-1 | vus............................: 11 min=11 max=150
xk6-output-prometheus-remote-k6-1 | vus_max........................: 150 min=150 max=150
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 |
xk6-output-prometheus-remote-k6-1 | running (0m31.3s), 000/150 VUs, 3613 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1 | default ✓ [ 100% ] 150 VUs 30s
xk6-output-prometheus-remote-k6-1 exited with code 0
Step 2 : 프로메테우스 연결 - http://localhost:9090
- 프로메테우스에서 타깃을 조사해 보면 k6에 관한 메트릭이 표시되며 add Panal 키와 Execute 버튼으로 내용을 추가 가능
Step 3 : 그라파나 접속 http://localhost:3000
- 최초 접속 시 id/pw는 모두 admin
1) configuration(톱니바퀴 모양) 찾아서 데이터 소스 찾아 prometheus 추가
2) 반드시 URL을 http://prometheus:9090으로 작성
Step 4 : 그라파나 대시보드 확인
- 이미 커뮤니티에 작성된 대시보드 export 또한 가능
# TROUBLE SHOOTING LOG
📝 문제 1 : docker compose up 시 exit code 0과 함께 k6가 exited
1. 현상
xk6-output-prometheus-remote-k6-1 exited with code 0
2. 원인
- docker-compose.yaml 파일의 k6에 test 스크립트 수행 command를 입력하지 않아 발생
3. 해결 방안
- docker-compose.yaml 파일에 작성한 test 스크립트를 볼륨으로 추가하고, command 수행하도록 수정
k6:
build: . # 현재 Dockerfile을 build하겠다는 의미
networks:
- k6
ports:
- "6565:6565"
environment:
- K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write
- K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true
- K6_OUT=xk6-prometheus-rw
depends_on:
- prometheus
- grafana
volumes:
- ./samples:/scripts
- .:/app:delegated #
command: run --vus 150 --duration 30s -o xk6-prometheus-rw /app/tests/basic_k6_test.js
# references
https://k6.io/docs/results-output/real-time/prometheus-remote-write/
'devops bootcamp 4 > pair/team log' 카테고리의 다른 글
Sprint - aws EC2와 K6를 이용한 성능테스트 (0) | 2023.06.07 |
---|---|
Sprint - 모니터링 시스템 구축 (0) | 2023.06.05 |
Sprint - Auto Scaling + CloudWatch를 이용한 알림 (0) | 2023.06.02 |
Sprint - 새 버전이 망가졌어요 (0) | 2023.05.19 |
Sprint - Terraform x AWS (0) | 2023.05.17 |
Comments