IT STUDY LOG

Sprint - 성능 테스트를 위한 서비스 모니터링(advanced) 본문

devops bootcamp 4/pair／team log

Sprint - 성능 테스트를 위한 서비스 모니터링(advanced)

roheerumi 2023. 6. 8. 09:41

#학습 목표

- Prometheus와 Grafana는 오픈소스로 제공되는 모니터링 서비스로서 많은 개발자들에게 큰 인기를 얻고 있음.
- 이와 마찬가지로 k6 역시 오픈소스로 제공되는 툴로서 소프트웨어 엔지니어링 커뮤니티에서 아주 큰 인기를 얻고 있으며, 특히 개발자 친화적으로 설계가 되어 있고 스크립트 언어로 javascript를 사용하므로 개발자들이 성능 테스트를 쉽게 작성하고 유지 가능
=> 이러한 두 서비스의 장점을 살려서 성능 테스트를 모니터링
  • 기본적인 K6 성능테스트 스크립트를 작성
  • K6 output 및 output 을 위한 k6 extention을 이해
  • K6 ouput을 위한 도구로 프로메테우스를 사용하면서 프로메테우스를 익힘
  • k6로부터 나온 output을 그라파나로 시각화 

 

#과제 항목별 진행 상황

✏️  Set up & Preparation

Step 1 : k6확장 리포지토리를 clone

# ssh 접속으로 clone
$ git clone git@github.com:grafana/xk6-output-prometheus-remote.git

# HTTPS 접속으로 clone
$ git clone https://github.com/grafana/xk6-output-prometheus-remote.git

Step 2 : 성능테스트를 위한 tests 폴더를 clone해 온 폴더 안에 생성

$ cd xk6-output-prometheus-remote 
$ mkdir tests

Step 3 : tests 폴더 안에 k6 성능 테스트를 위한 basic_k6_test.js 파일을 생성

// basic_k6_test.js

import http from 'k6/http';
import { describe } from 'https://jslib.k6.io/expect/0.0.4/index.js';
import { check, sleep } from 'k6';

export default function () {
    describe('Basic k6 test', () => {
        let res = http.get('https://test.k6.io');
        check(res, {
            'is status 200': (r) => r.status === 200
        });
        sleep(1);
    });
}

Step 4 : 상위 폴더로 돌아가 Dockerfile 확인

# Multi-stage build to generate custom k6 with extension
FROM golang:1.20-alpine as builder # 별칭을 builder로
WORKDIR $GOPATH/src/go.k6.io/k6
COPY . .
RUN apk --no-cache add git=~2 # apk : alpine의 패키지 매니저
RUN CGO_ENABLED=0 go install go.k6.io/xk6/cmd/xk6@latest  \
    && CGO_ENABLED=0 xk6 build \
    --with github.com/grafana/xk6-output-prometheus-remote=. \
    --output /tmp/k6

# Create image for running k6 with output for Prometheus Remote Write
FROM alpine:3.17

# hadolint ignore=DL3018
RUN apk add --no-cache ca-certificates && \
    adduser -D -u 12345 -g 12345 k6
COPY --from=builder /tmp/k6 /usr/bin/k6

USER 12345
WORKDIR /home/k6

ENTRYPOINT ["k6"]

- 도커 파일 내용 참고 :  도커허브, k6 공식사이트

Step 5 : docker-compose.yaml 파일 작성

version: '3.8'
volumes: # 로컬에 생성할 volumes 
  prometheus-data: 
  grafana-data-storage:

networks:
  k6:
  grafana:
  prometheus:

services:
  prometheus:
    image: prom/prometheus:v2.42.0
    command:
      - --web.enable-remote-write-receiver
      - --enable-feature=native-histograms
      - --config.file=/etc/prometheus/prometheus.yml
      - --enable-feature=remote-write-receiver --config.file=/etc/prometheus/prometheus.yaml
    networks:
      - k6
      - grafana
      - prometheus
    ports:
      - "9090:9090"
    depends_on:
      - grafana
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yaml
      - prometheus-data:/prometheus

  grafana:
    image: grafana/grafana:9.4.7
    networks:
      - grafana
      - prometheus
    ports:
      - "3000:3000"
    environment:
      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
      - GF_AUTH_ANONYMOUS_ENABLED=true
      - GF_AUTH_BASIC_ENABLED=false
    volumes:
      - ./grafana:/etc/grafana/provisioning/
      - grafana-data-storage:/var/lib/grafana

  k6:
    build: .
    networks:
      - k6
    ports:
      - "6565:6565"
    environment:
      - K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write
      - K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true
      - K6_OUT=xk6-prometheus-rw
    depends_on:
      - prometheus
      - grafana
    volumes:
      - ./samples:/scripts
      - .:/app:delegated
    # k6 기동시 작성했던 basic_k6_test.js 파일이 실행되도록 커맨드 실행  
    command: run --vus 150 --duration 30s -o xk6-prometheus-rw /app/tests/basic_k6_test.js

Step 6 : prometheus.yaml 파일 작성

# prometheus.yml
global:
  scrape_interval: 30s
  scrape_timeout: 10s
scrape_configs:
  - job_name: services
    metrics_path: /metrics
    static_configs:
      - targets:
          - 'prometheus:9090'

 

✏️  프로메테우스와 그라파나에서 성능 테스트 내역 확인

Step 1 : 도커 실행

$ docker compose up

# 백그라운드 실행시
$ docker compose up -d 

# 일부만 실행할 시
$ docker compose up prometheus grafana
$ docker compose up k6

- k6 테스트 수행 로그

더보기 
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          |           /\      |‾‾| /‾‾/   /‾‾/   
xk6-output-prometheus-remote-k6-1          |      /\  /  \     |  |/  /   /  /    
xk6-output-prometheus-remote-k6-1          |     /  \/    \    |     (   /   ‾‾\  
xk6-output-prometheus-remote-k6-1          |    /          \   |  |\  \ |  (‾)  | 
xk6-output-prometheus-remote-k6-1          |   / __________ \  |__| \__\ \_____/ .io
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          |   execution: local
xk6-output-prometheus-remote-k6-1          |      script: /app/tests/basic_k6_test.js
xk6-output-prometheus-remote-k6-1          |      output: Prometheus remote write (http://prometheus:9090/api/v1/write)
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          |   scenarios: (100.00%) 1 scenario, 150 max VUs, 1m0s max duration (incl. graceful stop):
xk6-output-prometheus-remote-k6-1          |            * default: 150 looping VUs for 30s (gracefulStop: 30s)
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m00.1s), 150/150 VUs, 0 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [   0% ] 150 VUs  00.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m01.1s), 150/150 VUs, 0 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [   4% ] 150 VUs  01.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m02.1s), 150/150 VUs, 53 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [   7% ] 150 VUs  02.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m03.1s), 150/150 VUs, 153 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  10% ] 150 VUs  03.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m04.1s), 150/150 VUs, 300 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  14% ] 150 VUs  04.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m05.1s), 150/150 VUs, 450 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  17% ] 150 VUs  05.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m06.1s), 150/150 VUs, 558 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  20% ] 150 VUs  06.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m07.1s), 150/150 VUs, 658 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  24% ] 150 VUs  07.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m08.1s), 150/150 VUs, 781 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  27% ] 150 VUs  08.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m09.1s), 150/150 VUs, 900 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  30% ] 150 VUs  09.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m10.1s), 150/150 VUs, 1050 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  34% ] 150 VUs  10.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m11.1s), 150/150 VUs, 1161 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  37% ] 150 VUs  11.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m12.1s), 150/150 VUs, 1273 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  40% ] 150 VUs  12.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m13.1s), 150/150 VUs, 1398 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  44% ] 150 VUs  13.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m14.1s), 150/150 VUs, 1511 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  47% ] 150 VUs  14.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m15.1s), 150/150 VUs, 1650 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  50% ] 150 VUs  15.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m16.1s), 150/150 VUs, 1786 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  54% ] 150 VUs  16.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m17.1s), 150/150 VUs, 1876 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  57% ] 150 VUs  17.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m18.1s), 150/150 VUs, 2002 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  60% ] 150 VUs  18.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m19.1s), 150/150 VUs, 2124 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  64% ] 150 VUs  19.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m20.1s), 150/150 VUs, 2253 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  67% ] 150 VUs  20.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m21.1s), 150/150 VUs, 2365 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  70% ] 150 VUs  21.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m22.1s), 150/150 VUs, 2485 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  74% ] 150 VUs  22.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m23.1s), 150/150 VUs, 2606 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  77% ] 150 VUs  23.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m24.1s), 150/150 VUs, 2733 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  80% ] 150 VUs  24.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m25.1s), 150/150 VUs, 2863 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  84% ] 150 VUs  25.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m26.1s), 150/150 VUs, 2985 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  87% ] 150 VUs  26.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m27.1s), 150/150 VUs, 3109 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  90% ] 150 VUs  27.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m28.1s), 150/150 VUs, 3223 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  94% ] 150 VUs  28.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m29.1s), 150/150 VUs, 3344 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default   [  97% ] 150 VUs  29.1s/30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m30.1s), 143/150 VUs, 3470 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default ↓ [ 100% ] 150 VUs  30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m31.1s), 011/150 VUs, 3602 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default ↓ [ 100% ] 150 VUs  30s
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          |      █ Basic k6 test
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          |        ✓ is status 200
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          |      checks.........................: 100.00% ✓ 3613       ✗ 0    
xk6-output-prometheus-remote-k6-1          |      data_received..................: 42 MB   1.4 MB/s
xk6-output-prometheus-remote-k6-1          |      data_sent......................: 405 kB  13 kB/s
xk6-output-prometheus-remote-k6-1          |      group_duration.................: avg=1.27s    min=1.19s       med=1.21s    max=2.33s    p(90)=1.39s    p(95)=1.4s    
xk6-output-prometheus-remote-k6-1          |      http_req_blocked...............: avg=28.46ms  min=2.91µs      med=7.56µs   max=792.34ms p(90)=17.89µs  p(95)=69.27µs 
xk6-output-prometheus-remote-k6-1          |      http_req_connecting............: avg=9.39ms   min=0s          med=0s       max=253.98ms p(90)=0s       p(95)=0s      
xk6-output-prometheus-remote-k6-1          |      http_req_duration..............: avg=241.87ms min=178.39ms    med=209.14ms max=576.4ms  p(90)=389.24ms p(95)=402.86ms
xk6-output-prometheus-remote-k6-1          |        { expected_response:true }...: avg=241.87ms min=178.39ms    med=209.14ms max=576.4ms  p(90)=389.24ms p(95)=402.86ms
xk6-output-prometheus-remote-k6-1          |      http_req_failed................: 0.00%   ✓ 0          ✗ 3613 
xk6-output-prometheus-remote-k6-1          |      http_req_receiving.............: avg=14.83ms  min=-16607384ns med=194.68µs max=222.45ms p(90)=11.02ms  p(95)=187.99ms
xk6-output-prometheus-remote-k6-1          |      http_req_sending...............: avg=414.76µs min=10.95µs     med=33.21µs  max=58.02ms  p(90)=115.31µs p(95)=217.52µs
xk6-output-prometheus-remote-k6-1          |      http_req_tls_handshaking.......: avg=15.15ms  min=0s          med=0s       max=459.9ms  p(90)=0s       p(95)=0s      
xk6-output-prometheus-remote-k6-1          |      http_req_waiting...............: avg=226.62ms min=177.51ms    med=207.14ms max=565.9ms  p(90)=274.12ms p(95)=359.74ms
xk6-output-prometheus-remote-k6-1          |      http_reqs......................: 3613    115.416168/s
xk6-output-prometheus-remote-k6-1          |      iteration_duration.............: avg=1.27s    min=1.19s       med=1.21s    max=2.33s    p(90)=1.39s    p(95)=1.4s    
xk6-output-prometheus-remote-k6-1          |      iterations.....................: 3613    115.416168/s
xk6-output-prometheus-remote-k6-1          |      vus............................: 11      min=11       max=150
xk6-output-prometheus-remote-k6-1          |      vus_max........................: 150     min=150      max=150
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | 
xk6-output-prometheus-remote-k6-1          | running (0m31.3s), 000/150 VUs, 3613 complete and 0 interrupted iterations
xk6-output-prometheus-remote-k6-1          | default ✓ [ 100% ] 150 VUs  30s
xk6-output-prometheus-remote-k6-1 exited with code 0

 

Step 2 :  프로메테우스 연결 - http://localhost:9090

- 프로메테우스에서 타깃을 조사해 보면 k6에 관한 메트릭이 표시되며 add Panal 키와 Execute 버튼으로 내용을 추가 가능

Step 3 :  그라파나 접속 http://localhost:3000

- 최초 접속 시 id/pw는 모두 admin

1) configuration(톱니바퀴 모양) 찾아서 데이터 소스 찾아 prometheus 추가

2) 반드시 URL을 http://prometheus:9090으로 작성

Step 4 :  그라파나 대시보드 확인

- 이미 커뮤니티에 작성된 대시보드 export 또한 가능

 

# TROUBLE SHOOTING LOG

📝 문제 1 : docker compose up 시 exit code 0과 함께 k6가 exited 

1. 현상

xk6-output-prometheus-remote-k6-1 exited with code 0

2. 원인

- docker-compose.yaml 파일의 k6에 test 스크립트 수행 command를 입력하지 않아 발생

3. 해결 방안

- docker-compose.yaml 파일에 작성한 test 스크립트를 볼륨으로 추가하고, command 수행하도록 수정

  k6:
    build: . # 현재 Dockerfile을 build하겠다는 의미
    networks:
      - k6
    ports:
      - "6565:6565"
    environment:
      - K6_PROMETHEUS_RW_SERVER_URL=http://prometheus:9090/api/v1/write
      - K6_PROMETHEUS_RW_TREND_AS_NATIVE_HISTOGRAM=true
      - K6_OUT=xk6-prometheus-rw
    depends_on:
      - prometheus
      - grafana
    volumes:
      - ./samples:/scripts
      - .:/app:delegated # 
    command: run --vus 150 --duration 30s -o xk6-prometheus-rw /app/tests/basic_k6_test.js

 

# references

https://k6.io/docs/results-output/real-time/prometheus-remote-write/

 

Prometheus remote write

Use the Prometheus remote write output to send test results to any Prometheus remote write endpoint.

k6.io

https://github.com/grafana/xk6-output-prometheus-remote

 

GitHub - grafana/xk6-output-prometheus-remote: k6 extension to output real-time test metrics using Prometheus Remote Write.

k6 extension to output real-time test metrics using Prometheus Remote Write. - GitHub - grafana/xk6-output-prometheus-remote: k6 extension to output real-time test metrics using Prometheus Remote W...

github.com

https://stackoverflow.com/questions/43844639/how-do-i-add-cached-or-delegated-into-a-docker-compose-yml-volumes-list

 

How do I add :cached or :delegated into a docker-compose.yml volumes list?

Title says it all. I've got a few volumes set up as readonly (:ro) but want to test :cached and :delegated for helping with file i/o performance, but couldn't figure out how to set this up in a com...

stackoverflow.com

https://docs.docker.com/compose/

 

Docker Compose overview

 

docs.docker.com

저작자표시 비영리 변경금지

'devops bootcamp 4 > pair／team log' 카테고리의 다른 글

Sprint - aws EC2와 K6를 이용한 성능테스트  (0) 2023.06.07
Sprint - 모니터링 시스템 구축  (0) 2023.06.05
Sprint - Auto Scaling + CloudWatch를 이용한 알림  (0) 2023.06.02
Sprint - 새 버전이 망가졌어요  (0) 2023.05.19
Sprint - Terraform x AWS  (0) 2023.05.17
'devops bootcamp 4/pair／team log' Related Articles more
Comments