IT STUDY LOG
[SECTION3] <마이크로서비스> DAY 1 LOG 본문
# achivement goals
- Serveless를 이용한 AWS 리소스 생성
- 메시지 Queue가 사용되는 구조 이해
# PROJECT LOG
Tutorial - Step 1
Step 1: Serverless를 이용한 Lambda 생성 - serverless framework를 이용하여 간단한 Lambda 함수를 생성하고 배포
1. Serverless 튜토리얼을 통한 프로젝트 생성
1-1. nvm, npm, node 환경 구성
$ nvm ls
-> v20.2.0
default -> node (-> v20.2.0)
iojs -> N/A (default)
unstable -> N/A (default)
node -> stable (-> v20.2.0) (default)
stable -> 20.2 (-> v20.2.0) (default)
lts/* -> lts/hydrogen (-> N/A)
lts/argon -> v4.9.1 (-> N/A)
lts/boron -> v6.17.1 (-> N/A)
lts/carbon -> v8.17.0 (-> N/A)
lts/dubnium -> v10.24.1 (-> N/A)
lts/erbium -> v12.22.12 (-> N/A)
lts/fermium -> v14.21.3 (-> N/A)
lts/gallium -> v16.20.0 (-> N/A)
lts/hydrogen -> v18.16.0 (-> N/A)
tutorial-step-2 $ git:(main) ✗ nvm --help | grep use
nvm use [<version>] Modify PATH to use <version>. Uses .nvmrc if available and version is omitted.
The following optional arguments, if provided, must appear directly after `nvm use`:
nvm use 8.0 Use the latest available 8.0.x release
nvm use node Use the latest version
nvm use --lts Use the latest LTS version
tutorial-step-2 $ git:(main) ✗ nvm use lts/hydrogen
N/A: version "lts/hydrogen -> v18.16.0" is not yet installed.
You need to run `nvm install lts/hydrogen` to install and use it.
tutorial-step-2 $ git:(main) ✗ nvm install lts/hydrogen
Downloading and installing node v18.16.0...
Downloading https://nodejs.org/dist/v18.16.0/node-v18.16.0-darwin-x64.tar.xz...
##################################################################################################################################################### 100.0%
Computing checksum with shasum -a 256
Checksums matched!
Now using node v18.16.0 (npm v9.5.1)
$ nvm use lts/hydrogen
Now using node v18.16.0 (npm v9.5.1)
$ npm install --save-dev aws-sdk1-2. serverless framework 설치
$ npm install -g serverless
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated querystring@0.2.1: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated querystring@0.2.0: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated superagent@7.1.6: Please downgrade to v7.1.5 if you need IE/ActiveXObject support OR upgrade to v8.0.0 as we no longer support IE and published an incorrect patch version (see https://github.com/visionmedia/superagent/issues/1731)
added 418 packages in 26s
68 packages are looking for funding
run `npm fund` for details
npm notice
npm notice New patch version of npm available! 9.6.6 -> 9.6.7
npm notice Changelog: https://github.com/npm/cli/releases/tag/v9.6.7
npm notice Run npm install -g npm@9.6.7 to update!
npm notice
section2 $ npm install -g npm@9.6.7
changed 28 packages in 3s
27 packages are looking for funding
run `npm fund` for details
section2 $ serverless
Creating a new serverless project
? What do you want to make? AWS - Node.js - HTTP API
? What do you want to call this project? aws-node-http-api-project
✔ Project successfully created in aws-node-http-api-project folder1-3. serverless framework를 이용해 lambda 프로젝트 생성
$ serverless
Creating a new serverless project
? What do you want to make? AWS - Node.js - HTTP API
? What do you want to call this project? aws-node-http-api-prj
✔ Project successfully created in aws-node-http-api-prj folder
? Do you want to deploy now? No
What next?
Run these commands in the project directory:
serverless deploy Deploy changes
serverless info View deployed endpoints and resources
serverless invoke Invoke deployed functions
serverless --help Discover more commands2. Serverless.yml 레퍼런스를 참고하여 리전 변경
# serverless.yaml
service: tutorial-step-2
frameworkVersion: '3'
provider:
name: aws
runtime: nodejs14.x
region: ap-northeast-2
# ... 하략 ...3. 서버 구현을 위해 handler.js 편집
module.exports.hello = async (event) => {
let inputValue, outputValue
console.log(event.body)
if (event.body) {
let body = JSON.parse(event.body)
// 추가 도전과제: body가 { input: 숫자 } 가 맞는지 검증하고, 검증에 실패하면 응답코드 400 및 에러 메시지를 반환
if (typeof body.input !== "number") {
return {
statusCode: 400,
body: JSON.stringify(
{
message: `입력값이 숫자가 아닙니다.`
},
null, // replacer
2 // 문자열 간격 조정
),
};
}
inputValue = parseInt(body.input)
outputValue = inputValue + 1
}
const message = `메시지를 받았습니다. 입력값: ${inputValue}, 결과: ${outputValue}`
return {
statusCode: 200,
body: JSON.stringify(
{
message
},
null,
2
),
};
};4. serverless deploy를 통한 배포
$ sls deploy5. CURL을 통한 테스트: 입력값의 +1 반환 함수 실행을 확인 가능
# 정상적인 요청
$ curl -X POST https://.execute-api.ap-northeast-2.amazonaws.com/ \
--header 'Content-type: application/json' \
--data-raw '{ "input": 1 }'
{
"message": "메시지를 받았습니다. 입력값: 1, 결과: 2"
}
# 비정상적인 요청
$ curl -X POST https://.execute-api.ap-northeast-2.amazonaws.com/ \
--header 'Content-type: application/json' \
--data-raw '{ "input": "스트링" }'
{
"message": "입력값이 숫자가 아닙니다."
}
Tutorial - Step 2
Step 2: Serverless를 이용한 Lambda - SQS - Lambda 구조 생성 - serverless framework를 이용하여 메시지 큐(SQS)를 이용한 producer/consumer 구조를 생성하고 배포
1. Serverless CLI를 통한 프로젝트 생성 (AWS - Node.js - SQS Worker)
$ serverless
Creating a new serverless project
? What do you want to make? AWS - Node.js - SQS Worker
? What do you want to call this project? tutorial-step-2
✔ Project successfully created in tutorial-step-2 folder
? Do you want to deploy now? No
What next?
Run these commands in the project directory:
serverless deploy Deploy changes
serverless info View deployed endpoints and resources
serverless invoke Invoke deployed functions
serverless --help Discover more commands2. Serverless.yml 레퍼런스를 참고하여 리전 변경
# serverless.yaml
service: tutorial-step-2
frameworkVersion: '3'
provider:
name: aws
runtime: nodejs14.x
region: ap-northeast-2
constructs:
jobs:
type: queue
worker:
handler: handler.consumer
functions:
producer:
handler: handler.producer
events:
- httpApi:
method: post
path: /produce
environment:
QUEUE_URL: ${construct:jobs.queueUrl}
plugins:
- serverless-lift3. handler.js 편집 (컨슈머 구현)
// /section2/project3-tutorial/tutorial-step-2/handler.js
const { SQSClient, SendMessageCommand } = require("@aws-sdk/client-sqs");
const sqs = new SQSClient();
const producer = async (event) => {
let statusCode = 200;
let message;
if (!event.body) {
return {
statusCode: 400,
body: JSON.stringify({
message: "No body was found",
}),
};
}
try {
await sqs.send(new SendMessageCommand({
QueueUrl: process.env.QUEUE_URL,
MessageBody: event.body,
MessageAttributes: {
AttributeName: {
StringValue: "Attribute Value",
DataType: "String",
},
},
}));
message = "Message accepted!";
} catch (error) {
console.log(error);
message = error;
statusCode = 500;
}
return {
statusCode,
body: JSON.stringify({
message,
}),
};
};
const consumer = async (event) => {
for (const record of event.Records) {
console.log("Message Body: ", record.body);
let body = JSON.parse(event.body)
let inputValue, outputValue
// +1을 수행하는 코드
if (event.body) {
if (typeof body.input !== "number") {
return {
statusCode: 400,
body: JSON.stringify(
{
message: `입력값이 숫자가 아닙니다.`
},
null, // replacer
2 // 문자열 간격 조정
),
};
}
inputValue = parseInt(body.input)
outputValue = inputValue + 1
}
const message = `메시지를 받았습니다. 입력값: ${inputValue}, 결과: ${outputValue}`
console.log(message)
return {
statusCode: 200,
body: JSON.stringify(
{
message
},
null,
2
),
};
}
};
module.exports = {
producer,
consumer,
};4. serverless deploy를 통한 배포
$ serverless deploy
Running "serverless" from node_modules
Deploying tutorial-step-2 to stage dev (ap-northeast-2)
✔ Service deployed to stack tutorial-step-2-dev (269s)
endpoint: POST - https://.execute-api.ap-northeast-2.amazonaws.com/produce
functions:
producer: tutorial-step-2-dev-producer (119 kB)
jobsWorker: tutorial-step-2-dev-jobsWorker (119 kB)
jobs: https://sqs.ap-northeast-2.amazonaws.com//tutorial-step-2-dev-jobs
5. 프로듀서 실행 → CloudWatch를 통해 컨슈머가 메시지를 소비하는 것을 확인
$ curl -X POST https://.execute-api.ap-northeast-2.amazonaws.com/produce \
--header 'Content-type: application/json' \
--data-raw '{ "input": 1 }'
{"message":"Message accepted!"}%
6. 프로듀서를 여러 번 반복해서 실행 (쉘 스크립트의 반복문을 이용)
$ for i in {1..50}
do
curl -X POST https://.execute-api.ap-northeast-2.amazonaws.com/produce \
--header 'Content-type: application/json' \
--data-raw '{ "input": 1 }'
done
{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}{"message":"Message accepted!"}%
Tutorial - Step 3
Step 3-1: DLQ 연결 및 K6 성능테스트
1. 15초 시간을 지연시켜 요청 발생
1-1. lambda jobsWorker 함수의 타임아웃과 SQS 큐의 visibility time 확인
1-2. cloudwatch에서 타임아웃 발생 로그 확인
3. SQS 대기열에서 소비되지 못한 메시지가 dlq로 이동한 것 확인
4. consumer에 실행 시간을 늘리는 코드 추가 후 sls deploy
// project_study/section2/project3-tutorial/tutorial-step-2/handler.js
function delay(time) {
return new Promise(resolve => setTimeout(resolve, time));
}
const consumer = async (event) => {
await delay(15000)5. k6 성능 테스트 도구 활용해 프로듀서를 반복적으로 실행 및 테스트 진행
$ brew install k6$ ./run.sh
/\ |‾‾| /‾‾/ /‾‾/
/\ / \ | |/ / / /
/ \/ \ | ( / ‾‾\
/ \ | |\ \ | (‾) |
/ __________ \ |__| \__\ \_____/ .io
execution: local
script: ./single-request.k6.js
output: -
scenarios: (100.00%) 1 scenario, 1 max VUs, 10m30s max duration (incl. graceful stop):
* default: 100 iterations shared among 1 VUs (maxDuration: 10m0s, gracefulStop: 30s)
# .. 상략 ... #
INFO[0010] {"input":69} source=console
INFO[0010] status code : 200, body.message : {"message":"Message accepted!"} source=console
INFO[0010] {"input":70} source=console
INFO[0010] status code : 200, body.message : {"message":"Message accepted!"} source=console
INFO[0010] {"input":71} source=console
INFO[0010] status code : 503, body.message : {"message":"Service Unavailable"} source=console
INFO[0010] {"input":72} source=console
INFO[0010] status code : 503, body.message : {"message":"Service Unavailable"} source=console
INFO[0010] {"input":73} source=console
INFO[0010] status code : 503, body.message : {"message":"Service Unavailable"} source=console
# .. 중략 ... #
INFO[0013] status code : 503, body.message : {"message":"Service Unavailable"} source=console
INFO[0013] {"input":95} source=console
INFO[0013] status code : 200, body.message : {"message":"Message accepted!"} source=console
INFO[0013] {"input":96} source=console
INFO[0013] status code : 503, body.message : {"message":"Service Unavailable"} source=console
INFO[0014] {"input":97} source=console
INFO[0014] status code : 503, body.message : {"message":"Service Unavailable"} source=console
INFO[0014] {"input":98} source=console
INFO[0014] status code : 503, body.message : {"message":"Service Unavailable"} source=console
INFO[0014] {"input":99} source=console
INFO[0014] status code : 503, body.message : {"message":"Service Unavailable"} source=console
INFO[0014] {"input":100} source=console
INFO[0014] status code : 503, body.message : {"message":"Service Unavailable"} source=console
✗ status is 200
↳ 76% — ✓ 76 / ✗ 24
✗ response body
↳ 76% — ✓ 76 / ✗ 24
checks.........................: 76.00% ✓ 152 ✗ 48
data_received..................: 25 kB 1.8 kB/s
data_sent......................: 10 kB 750 B/s
http_req_blocked...............: avg=589.34µs min=0s med=1µs max=58.82ms p(90)=2µs p(95)=2µs
http_req_connecting............: avg=88.93µs min=0s med=0s max=8.89ms p(90)=0s p(95)=0s
http_req_duration..............: avg=33.32ms min=17.33ms med=33.06ms max=65.92ms p(90)=42.74ms p(95)=51.74ms
{ expected_response:true }...: avg=36.96ms min=29.8ms med=34.64ms max=65.92ms p(90)=45.05ms p(95)=54.88ms
http_req_failed................: 24.00% ✓ 24 ✗ 76
http_req_receiving.............: avg=94.02µs min=62µs med=79µs max=850µs p(90)=110.1µs p(95)=132.05µs
http_req_sending...............: avg=146µs min=96µs med=135µs max=406µs p(90)=180.5µs p(95)=213.49µs
http_req_tls_handshaking.......: avg=367.89µs min=0s med=0s max=36.78ms p(90)=0s p(95)=0s
http_req_waiting...............: avg=33.08ms min=17.05ms med=32.85ms max=65.71ms p(90)=42.53ms p(95)=51.35ms
✓ http_reqs......................: 100 7.310731/s
iteration_duration.............: avg=136.75ms min=120.24ms med=135.66ms max=225.93ms p(90)=146.95ms p(95)=154.66ms
iterations.....................: 100 7.310731/s
vus............................: 1 min=1 max=1
vus_max........................: 1 min=1 max=1
running (00m13.7s), 0/1 VUs, 100 complete and 0 interrupted iterations
default ✓ [======================================] 1 VUs 00m13.7s/10m0s 100/100 shared iters
tutorial-step-3 $ git:(main) $6. 의도적으로 visibility time을 변경해 sqs, lambda에서 중복 소비되도록 테스트
# ISSUE LOG
[ISSUE 1] 프로듀서 호출 시 서버 에러 발생
현상
2023-05-24T04:05:05.859Z undefined ERROR Uncaught Exception
{
"errorType": "Runtime.ImportModuleError",
"errorMessage": "Error: Cannot find module '@aws-sdk/client-sqs'\nRequire stack:\n- /var/task/handler.js\n- /var/runtime/UserFunction.js\n- /var/runtime/Runtime.js\n- /var/runtime/index.js",
"stack": [
"Runtime.ImportModuleError: Error: Cannot find module '@aws-sdk/client-sqs'",
"Require stack:",
"- /var/task/handler.js",
"- /var/runtime/UserFunction.js",
"- /var/runtime/Runtime.js",
"- /var/runtime/index.js",
" at _loadUserApp (/var/runtime/UserFunction.js:225:13)",
" at Object.module.exports.load (/var/runtime/UserFunction.js:300:17)",
" at Object.<anonymous> (/var/runtime/index.js:43:34)",
" at Module._compile (internal/modules/cjs/loader.js:1114:14)",
" at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)",
" at Module.load (internal/modules/cjs/loader.js:979:32)",
" at Function.Module._load (internal/modules/cjs/loader.js:819:12)",
" at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:75:12)",
" at internal/main/run_main_module.js:17:47"
]
}원인
aws-sdk는 v2, v3이 존재 (aws-sdk 모듈의 경우 v2, @aws-sdk 모듈의 경우 v3)하며 v3의 경우 node.js 14 이상의 버전을 지원하므로 발생하는 문제
해결 방안
node.js 런타임을 18로 변경
Default AWS Region
Hi, Im fairly new to Serverless, so apologies if this is a dumb question. I’ve got the AWS cli set up to use a specific region (eu-west-2) as my default region. When I deploy with Serverless though, it defaults to us-east-1 if I don’t specify the regio
forum.serverless.com
https://docs.aws.amazon.com/AWSJavaScriptSDK/v3/latest/index.html
AWS SDK for JavaScript v3
New API Documentation - Developer Preview Available We are excited to announce the developer preview of our new API documentation for AWS SDK for JavaScript v3. Please follow instructions on the landing page to leave us your feedback. AWS SDK for JavaScrip
docs.aws.amazon.com
Amazon SQS용 CloudWatch 메트릭에 액세스 - Amazon Simple Queue Service
이 페이지에 작업이 필요하다는 점을 알려 주셔서 감사합니다. 실망시켜 드려 죄송합니다. 잠깐 시간을 내어 설명서를 향상시킬 수 있는 방법에 대해 말씀해 주십시오.
docs.aws.amazon.com
GitHub - grafana/k6: A modern load testing tool, using Go and JavaScript - https://k6.io
A modern load testing tool, using Go and JavaScript - https://k6.io - GitHub - grafana/k6: A modern load testing tool, using Go and JavaScript - https://k6.io
github.com
https://k6.io/docs/get-started/installation/#macos
Installation
k6 has packages for Linux, Mac, and Windows. As alternatives, you can also using a Docker container or a standalone binary.
k6.io
'devops bootcamp 4 > project log' 카테고리의 다른 글
| [SECTION3] <마이크로서비스> DAY 3 LOG (0) | 2023.05.26 |
|---|---|
| [SECTION3] <마이크로서비스> DAY 2 LOG (0) | 2023.05.25 |
| [SECTION3] <마이크로서비스> 프로젝트 개요 (0) | 2023.05.24 |
| [SECTION2] <AWS 배포 자동화> DAY 4 LOG (0) | 2023.05.02 |
| [SECTION2] <AWS 배포 자동화> DAY 3 LOG (0) | 2023.05.01 |
'devops bootcamp 4/project log' Related Articles
more
Comments