머신러닝 모델을 개발하는 것과 프로덕션 환경에서 안정적으로 운영하는 것은 완전히 다른 차원의 문제다. Jupyter Notebook에서 95% 정확도를 달성한 모델이 실제 서비스에서는 성능이 급격히 떨어지거나, 모델 업데이트 후 롤백이 어려워 서비스 장애가 발생하는 사례는 흔하다. MLOps(Machine Learning Operations)는 이러한 문제를 해결하기 위한 실무 방법론이다. 이 글은 MLOps의 핵심 개념부터 실전 구현까지, 프로덕션 환경에서 AI 모델을 안정적으로 운영하는 모든 것을 다룬다.
\nMLOps는 DevOps의 머신러닝 버전으로, ML 모델의 개발부터 배포, 모니터링, 재학습까지 전체 라이프사이클을 자동화하고 표준화하는 관행이다. 전통적인 소프트웨어 개발과 달리 ML은 다음과 같은 특수성을 가진다:
\n| 구분 | \nDevOps | \nMLOps | \n
|---|---|---|
| 버전 관리 | \n코드만 관리 | \n코드 + 데이터 + 모델 모두 관리 | \n
| 테스트 | \n단위/통합 테스트 | \n데이터 검증, 모델 성능 테스트 | \n
| 배포 | \n코드 배포 | \n모델 + 인퍼런스 서버 배포 | \n
| 모니터링 | \n애플리케이션 메트릭 | \n모델 성능, 데이터 드리프트, 예측 분포 | \n
| CI/CD | \n코드 변경 시 빌드/배포 | \n데이터/코드 변경 시 재학습/재배포 | \n
[데이터 수집] → [데이터 검증] → [특성 엔지니어링] → [모델 학습]\n ↓\n[모델 평가] ← [모델 검증] ← [모델 등록] ← [모델 패키징]\n ↓\n[모델 배포] → [A/B 테스트] → [프로덕션 서빙] → [모니터링]\n ↑ ↓\n[자동 재학습] ← [드리프트 감지] ← [성능 저하 알림] ← [로그 수집]MLflow는 Databricks에서 개발한 오픈소스 MLOps 플랫폼으로, 실험 추적, 모델 레지스트리, 모델 서빙을 통합 제공한다.
\n핵심 컴포넌트:
\nimport mlflow\nimport mlflow.sklearn\nfrom sklearn.ensemble import RandomForestClassifier\nfrom sklearn.model_selection import train_test_split\nfrom sklearn.metrics import accuracy_score, precision_score, recall_score\nimport pandas as pd\n\n# MLflow 실험 설정\nmlflow.set_experiment(\"customer_churn_prediction\")\n\n# 실험 시작\nwith mlflow.start_run(run_name=\"rf_baseline_v1\"):\n # 데이터 로드\n df = pd.read_csv(\"data/customer_churn.csv\")\n X_train, X_test, y_train, y_test = train_test_split(\n df.drop(\"churn\", axis=1), \n df[\"churn\"], \n test_size=0.2, \n random_state=42\n )\n \n # 하이퍼파라미터 정의\n params = {\n \"n_estimators\": 100,\n \"max_depth\": 10,\n \"min_samples_split\": 5,\n \"random_state\": 42\n }\n \n # 모델 학습\n model = RandomForestClassifier(**params)\n model.fit(X_train, y_train)\n \n # 예측 및 평가\n y_pred = model.predict(X_test)\n accuracy = accuracy_score(y_test, y_pred)\n precision = precision_score(y_test, y_pred)\n recall = recall_score(y_test, y_pred)\n \n # MLflow에 로깅\n mlflow.log_params(params)\n mlflow.log_metrics({\n \"accuracy\": accuracy,\n \"precision\": precision,\n \"recall\": recall\n })\n \n # 모델 저장\n mlflow.sklearn.log_model(\n model, \n \"model\",\n registered_model_name=\"ChurnPredictor\"\n )\n \n # 데이터셋 정보 로깅\n mlflow.log_artifact(\"data/customer_churn.csv\", \"dataset\")\n \n print(f\"Accuracy: {accuracy:.4f}\")\n print(f\"Run ID: {mlflow.active_run().info.run_id}\")from mlflow.tracking import MlflowClient\n\nclient = MlflowClient()\n\n# 모델 버전 조회\nmodel_versions = client.search_model_versions(\"name='ChurnPredictor'\")\nfor mv in model_versions:\n print(f\"Version: {mv.version}, Stage: {mv.current_stage}\")\n\n# 모델을 Staging으로 전환\nclient.transition_model_version_stage(\n name=\"ChurnPredictor\",\n version=1,\n stage=\"Staging\"\n)\n\n# Production으로 승격\nclient.transition_model_version_stage(\n name=\"ChurnPredictor\",\n version=1,\n stage=\"Production\"\n)\n\n# Production 모델 로드\nimport mlflow.pyfunc\nmodel = mlflow.pyfunc.load_model(\n model_uri=f\"models:/ChurnPredictor/Production\"\n)# MLflow 모델 서빙 서버 시작\nmlflow models serve -m models:/ChurnPredictor/Production -p 5000\n\n# 예측 요청\ncurl -X POST http://localhost:5000/invocations \\\n -H 'Content-Type: application/json' \\\n -d '{\n \"inputs\": [[25, 50000, 1, 0, 1]]\n }'KServe는 Kubernetes 네이티브 모델 서빙 플랫폼으로, 자동 스케일링, 카나리 배포, A/B 테스트를 지원한다.
\n# KServe 설치\nkubectl apply -f https://github.com/kserve/kserve/releases/download/v0.11.0/kserve.yaml\n\n# InferenceService 생성\nkubectl apply -f - <<EOF\napiVersion: \"serving.kserve.io/v1beta1\"\nkind: \"InferenceService\"\nmetadata:\n name: \"churn-predictor\"\nspec:\n predictor:\n sklearn:\n storageUri: \"s3://mlflow-bucket/models/ChurnPredictor/1\"\n resources:\n requests:\n cpu: \"100m\"\n memory: \"1Gi\"\n limits:\n cpu: \"1000m\"\n memory: \"2Gi\"\nEOFfrom kserve import KServeClient\nimport requests\nimport json\n\n# KServe 클라이언트\nkserve_client = KServeClient()\n\n# InferenceService 생성\nkserve_client.set_credentials(\n storage_type=\"S3\",\n s3_endpoint=\"s3.amazonaws.com\",\n s3_region=\"us-east-1\",\n s3_use_https=True,\n s3_verify_ssl=True\n)\n\n# 예측 요청\nservice_name = \"churn-predictor\"\nnamespace = \"default\"\nheaders = {\"Host\": f\"{service_name}.{namespace}.example.com\"}\n\ndata = {\n \"instances\": [\n [25, 50000, 1, 0, 1],\n [35, 75000, 0, 1, 0]\n ]\n}\n\nresponse = requests.post(\n f\"http://{service_name}.{namespace}/v1/models/churn-predictor:predict\",\n headers=headers,\n json=data\n)\n\npredictions = response.json()\nprint(predictions)Seldon Core는 더 세밀한 트래픽 라우팅과 A/B 테스트를 지원한다.
\napiVersion: machinelearning.seldon.io/v1\nkind: SeldonDeployment\nmetadata:\n name: churn-predictor\nspec:\n name: churn-ab-test\n predictors:\n - name: default\n replicas: 2\n graph:\n name: model-a\n type: MODEL\n modelUri: s3://models/model-a\n children:\n - name: model-b\n type: MODEL\n modelUri: s3://models/model-b\n children: []\n traffic: 50\n - name: canary\n replicas: 1\n graph:\n name: model-c\n type: MODEL\n modelUri: s3://models/model-c\n traffic: 50새로운 모델을 배포할 때 기존 모델과 성능을 비교하는 것이 중요하다.
\nimport numpy as np\nfrom scipy import stats\n\nclass ABTestEvaluator:\n def __init__(self, control_metrics, treatment_metrics):\n self.control = control_metrics\n self.treatment = treatment_metrics\n \n def statistical_significance(self, alpha=0.05):\n \"\"\"통계적 유의성 검정\"\"\"\n t_stat, p_value = stats.ttest_ind(\n self.control, \n self.treatment\n )\n \n is_significant = p_value < alpha\n improvement = np.mean(self.treatment) - np.mean(self.control)\n \n return {\n \"p_value\": p_value,\n \"is_significant\": is_significant,\n \"improvement\": improvement,\n \"improvement_pct\": (improvement / np.mean(self.control)) * 100\n }\n \n def confidence_interval(self, confidence=0.95):\n \"\"\"신뢰 구간 계산\"\"\"\n diff = np.array(self.treatment) - np.array(self.control)\n mean_diff = np.mean(diff)\n std_diff = np.std(diff, ddof=1)\n n = len(diff)\n \n t_critical = stats.t.ppf((1 + confidence) / 2, n - 1)\n margin = t_critical * (std_diff / np.sqrt(n))\n \n return {\n \"mean_difference\": mean_diff,\n \"lower_bound\": mean_diff - margin,\n \"upper_bound\": mean_diff + margin,\n \"confidence\": confidence\n }\n\n# 사용 예시\ncontrol_accuracy = [0.85, 0.86, 0.84, 0.87, 0.85]\ntreatment_accuracy = [0.88, 0.89, 0.87, 0.90, 0.88]\n\nevaluator = ABTestEvaluator(control_accuracy, treatment_accuracy)\nresult = evaluator.statistical_significance()\nci = evaluator.confidence_interval()\n\nprint(f\"P-value: {result['p_value']:.4f}\")\nprint(f\"Improvement: {result['improvement_pct']:.2f}%\")\nprint(f\"95% CI: [{ci['lower_bound']:.4f}, {ci['upper_bound']:.4f}]\")import random\nfrom typing import Dict, Any\n\nclass TrafficSplitter:\n def __init__(self, splits: Dict[str, float]):\n \"\"\"\n splits: {\"model-a\": 0.5, \"model-b\": 0.5}\n \"\"\"\n assert abs(sum(splits.values()) - 1.0) < 1e-6, \"Splits must sum to 1.0\"\n self.splits = splits\n self.cumulative = []\n cumulative = 0\n for model, weight in splits.items():\n cumulative += weight\n self.cumulative.append((cumulative, model))\n \n def route(self, user_id: str = None) -> str:\n \"\"\"사용자를 모델로 라우팅\"\"\"\n if user_id:\n # 일관된 라우팅을 위해 해시 사용\n hash_val = hash(user_id) % 10000\n threshold = (hash_val / 10000.0) * sum(self.splits.values())\n else:\n threshold = random.random()\n \n for cum, model in self.cumulative:\n if threshold <= cum:\n return model\n return self.cumulative[-1][1]\n\n# 사용 예시\nsplitter = TrafficSplitter({\n \"model-a\": 0.7, # 70% 트래픽\n \"model-b\": 0.3 # 30% 트래픽\n})\n\n# 사용자별 일관된 라우팅\nuser_model = splitter.route(user_id=\"user123\")\nprint(f\"User routed to: {user_model}\")데이터 분포가 시간에 따라 변화하는 현상을 감지하는 것이 중요하다.
\nimport pandas as pd\nimport numpy as np\nfrom scipy import stats\nfrom typing import Tuple\n\nclass DataDriftDetector:\n def __init__(self, reference_data: pd.DataFrame):\n self.reference = reference_data\n self.reference_stats = self._compute_stats(reference_data)\n \n def _compute_stats(self, data: pd.DataFrame) -> dict:\n \"\"\"기준 데이터의 통계량 계산\"\"\"\n stats = {}\n for col in data.columns:\n if data[col].dtype in ['int64', 'float64']:\n stats[col] = {\n \"mean\": data[col].mean(),\n \"std\": data[col].std(),\n \"min\": data[col].min(),\n \"max\": data[col].max()\n }\n else:\n stats[col] = {\n \"value_counts\": data[col].value_counts().to_dict()\n }\n return stats\n \n def detect_drift(self, current_data: pd.DataFrame, \n threshold: float = 0.05) -> dict:\n \"\"\"드리프트 감지\"\"\"\n drift_report = {\n \"drifted_features\": [],\n \"drift_scores\": {},\n \"overall_drift\": False\n }\n \n for col in self.reference.columns:\n if col not in current_data.columns:\n continue\n \n ref_col = self.reference[col]\n curr_col = current_data[col]\n \n if ref_col.dtype in ['int64', 'float64']:\n # Kolmogorov-Smirnov 테스트\n ks_stat, p_value = stats.ks_2samp(ref_col, curr_col)\n drift_score = 1 - p_value\n \n if drift_score > threshold:\n drift_report[\"drifted_features\"].append(col)\n drift_report[\"drift_scores\"][col] = {\n \"score\": drift_score,\n \"p_value\": p_value,\n \"test\": \"KS\"\n }\n else:\n # 카이제곱 검정\n ref_counts = ref_col.value_counts()\n curr_counts = curr_col.value_counts()\n \n # 공통 카테고리만 비교\n common_cats = set(ref_counts.index) & set(curr_counts.index)\n if len(common_cats) > 0:\n ref_vals = [ref_counts.get(cat, 0) for cat in common_cats]\n curr_vals = [curr_counts.get(cat, 0) for cat in common_cats]\n \n chi2, p_value = stats.chisquare(curr_vals, ref_vals)\n drift_score = 1 - p_value\n \n if drift_score > threshold:\n drift_report[\"drifted_features\"].append(col)\n drift_report[\"drift_scores\"][col] = {\n \"score\": drift_score,\n \"p_value\": p_value,\n \"test\": \"Chi-square\"\n }\n \n drift_report[\"overall_drift\"] = len(drift_report[\"drifted_features\"]) > 0\n return drift_report\n\n# 사용 예시\nreference_data = pd.DataFrame({\n \"age\": np.random.normal(35, 10, 1000),\n \"income\": np.random.normal(50000, 15000, 1000),\n \"category\": np.random.choice([\"A\", \"B\", \"C\"], 1000)\n})\n\n# 시간이 지나 데이터 분포가 변경됨\ncurrent_data = pd.DataFrame({\n \"age\": np.random.normal(40, 12, 1000), # 평균이 변경됨\n \"income\": np.random.normal(55000, 18000, 1000), # 분산이 증가\n \"category\": np.random.choice([\"A\", \"B\", \"C\", \"D\"], 1000, p=[0.3, 0.3, 0.3, 0.1]) # 새로운 카테고리 추가\n})\n\ndetector = DataDriftDetector(reference_data)\ndrift_report = detector.detect_drift(current_data, threshold=0.05)\n\nprint(f\"Drifted features: {drift_report['drifted_features']}\")\nprint(f\"Overall drift detected: {drift_report['overall_drift']}\")import time\nfrom datetime import datetime, timedelta\nfrom collections import defaultdict\nimport numpy as np\n\nclass ModelPerformanceMonitor:\n def __init__(self, window_size: int = 1000):\n self.window_size = window_size\n self.predictions = []\n self.actuals = []\n self.latencies = []\n self.timestamps = []\n self.metrics_history = []\n \n def log_prediction(self, prediction: float, actual: float, \n latency: float):\n \"\"\"예측 결과 로깅\"\"\"\n self.predictions.append(prediction)\n self.actuals.append(actual)\n self.latencies.append(latency)\n self.timestamps.append(datetime.now())\n \n # 윈도우 크기 제한\n if len(self.predictions) > self.window_size:\n self.predictions.pop(0)\n self.actuals.pop(0)\n self.latencies.pop(0)\n self.timestamps.pop(0)\n \n def compute_metrics(self) -> dict:\n \"\"\"현재 성능 메트릭 계산\"\"\"\n if len(self.predictions) == 0:\n return {}\n \n predictions = np.array(self.predictions)\n actuals = np.array(self.actuals)\n \n # 분류 문제 가정 (이진 분류)\n if len(np.unique(actuals)) == 2:\n from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score\n \n accuracy = accuracy_score(actuals, predictions)\n precision = precision_score(actuals, predictions, zero_division=0)\n recall = recall_score(actuals, predictions, zero_division=0)\n f1 = f1_score(actuals, predictions, zero_division=0)\n \n metrics = {\n \"accuracy\": accuracy,\n \"precision\": precision,\n \"recall\": recall,\n \"f1_score\": f1\n }\n else:\n # 회귀 문제\n from sklearn.metrics import mean_squared_error, mean_absolute_error, r2_score\n \n mse = mean_squared_error(actuals, predictions)\n mae = mean_absolute_error(actuals, predictions)\n rmse = np.sqrt(mse)\n r2 = r2_score(actuals, predictions)\n \n metrics = {\n \"mse\": mse,\n \"mae\": mae,\n \"rmse\": rmse,\n \"r2_score\": r2\n }\n \n # 지연 시간 통계\n metrics.update({\n \"avg_latency\": np.mean(self.latencies),\n \"p95_latency\": np.percentile(self.latencies, 95),\n \"p99_latency\": np.percentile(self.latencies, 99),\n \"throughput\": len(self.predictions) / (\n (self.timestamps[-1] - self.timestamps[0]).total_seconds() + 1e-6\n )\n })\n \n return metrics\n \n def check_performance_degradation(self, baseline_metrics: dict, \n threshold: float = 0.05) -> dict:\n \"\"\"성능 저하 감지\"\"\"\n current_metrics = self.compute_metrics()\n \n alerts = []\n for metric_name, baseline_value in baseline_metrics.items():\n if metric_name not in current_metrics:\n continue\n \n current_value = current_metrics[metric_name]\n \n # 정확도, F1 등은 높을수록 좋음\n if metric_name in [\"accuracy\", \"precision\", \"recall\", \"f1_score\", \"r2_score\"]:\n degradation = (baseline_value - current_value) / baseline_value\n if degradation > threshold:\n alerts.append({\n \"metric\": metric_name,\n \"baseline\": baseline_value,\n \"current\": current_value,\n \"degradation\": degradation,\n \"severity\": \"high\" if degradation > 0.1 else \"medium\"\n })\n # MSE, MAE, RMSE는 낮을수록 좋음\n elif metric_name in [\"mse\", \"mae\", \"rmse\"]:\n degradation = (current_value - baseline_value) / baseline_value\n if degradation > threshold:\n alerts.append({\n \"metric\": metric_name,\n \"baseline\": baseline_value,\n \"current\": current_value,\n \"degradation\": degradation,\n \"severity\": \"high\" if degradation > 0.1 else \"medium\"\n })\n \n return {\n \"has_degradation\": len(alerts) > 0,\n \"alerts\": alerts,\n \"current_metrics\": current_metrics\n }\n\n# 사용 예시\nmonitor = ModelPerformanceMonitor(window_size=1000)\n\n# 시뮬레이션: 예측 결과 로깅\nfor i in range(100):\n prediction = np.random.rand()\n actual = np.random.rand()\n latency = np.random.uniform(10, 50) # 10-50ms\n monitor.log_prediction(prediction, actual, latency)\n\n# 성능 메트릭 확인\nmetrics = monitor.compute_metrics()\nprint(\"Current metrics:\", metrics)\n\n# 성능 저하 감지\nbaseline = {\"r2_score\": 0.85, \"rmse\": 0.15}\ndegradation_check = monitor.check_performance_degradation(baseline, threshold=0.05)\nprint(\"Degradation check:\", degradation_check)# .github/workflows/ml-pipeline.yml\nname: ML Pipeline\n\non:\n push:\n branches: [ main, develop ]\n pull_request:\n branches: [ main ]\n schedule:\n - cron: '0 2 * * *' # 매일 새벽 2시 재학습\n\njobs:\n data-validation:\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v3\n \n - name: Set up Python\n uses: actions/setup-python@v4\n with:\n python-version: '3.9'\n \n - name: Install dependencies\n run: |\n pip install -r requirements.txt\n pip install great-expectations\n \n - name: Validate data\n run: |\n python scripts/validate_data.py\n env:\n DATA_PATH: ${{ secrets.DATA_PATH }}\n \n train-model:\n needs: data-validation\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v3\n \n - name: Set up Python\n uses: actions/setup-python@v4\n with:\n python-version: '3.9'\n \n - name: Install dependencies\n run: pip install -r requirements.txt\n \n - name: Train model\n run: python scripts/train.py\n env:\n MLFLOW_TRACKING_URI: ${{ secrets.MLFLOW_TRACKING_URI }}\n MLFLOW_S3_BUCKET: ${{ secrets.MLFLOW_S3_BUCKET }}\n \n - name: Evaluate model\n run: python scripts/evaluate.py\n \n - name: Check model performance\n run: |\n python scripts/check_performance.py\n if [ $? -ne 0 ]; then\n echo \"Model performance below threshold\"\n exit 1\n fi\n \n deploy-model:\n needs: train-model\n if: github.ref == 'refs/heads/main'\n runs-on: ubuntu-latest\n steps:\n - uses: actions/checkout@v3\n \n - name: Configure AWS credentials\n uses: aws-actions/configure-aws-credentials@v2\n with:\n aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}\n aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}\n aws-region: us-east-1\n \n - name: Deploy to Kubernetes\n run: |\n kubectl set image deployment/churn-predictor \\\n model=${{ secrets.ECR_REGISTRY }}/churn-predictor:${{ github.sha }}\n \n - name: Run smoke tests\n run: |\n python scripts/smoke_tests.py# scripts/check_performance.py\nimport mlflow\nimport sys\nfrom mlflow.tracking import MlflowClient\n\ndef check_model_performance():\n client = MlflowClient()\n \n # 최신 실험 실행 가져오기\n experiment = client.get_experiment_by_name(\"customer_churn_prediction\")\n runs = client.search_runs(\n experiment_ids=[experiment.experiment_id],\n order_by=[\"start_time DESC\"],\n max_results=1\n )\n \n if not runs:\n print(\"No runs found\")\n sys.exit(1)\n \n latest_run = runs[0]\n metrics = latest_run.data.metrics\n \n # 성능 임계값\n thresholds = {\n \"accuracy\": 0.85,\n \"precision\": 0.80,\n \"recall\": 0.75,\n \"f1_score\": 0.78\n }\n \n failed_checks = []\n for metric_name, threshold in thresholds.items():\n if metric_name in metrics:\n if metrics[metric_name] < threshold:\n failed_checks.append(\n f\"{metric_name}: {metrics[metric_name]:.4f} < {threshold}\"\n )\n \n if failed_checks:\n print(\"Model performance below thresholds:\")\n for check in failed_checks:\n print(f\" - {check}\")\n sys.exit(1)\n else:\n print(\"All performance checks passed!\")\n sys.exit(0)\n\nif __name__ == \"__main__\":\n check_model_performance()# 재학습 트리거 로직\ndef should_retrain(monitor: ModelPerformanceMonitor, \n baseline_metrics: dict) -> bool:\n degradation_check = monitor.check_performance_degradation(\n baseline_metrics, \n threshold=0.05\n )\n \n if degradation_check[\"has_degradation\"]:\n high_severity_alerts = [\n a for a in degradation_check[\"alerts\"] \n if a[\"severity\"] == \"high\"\n ]\n if len(high_severity_alerts) > 0:\n return True\n \n return Falseimport mlflow\nfrom mlflow.tracking import MlflowClient\n\ndef rollback_model(model_name: str, target_version: int):\n \"\"\"모델을 특정 버전으로 롤백\"\"\"\n client = MlflowClient()\n \n # 현재 Production 모델 버전 확인\n current_prod = client.get_latest_versions(\n model_name, \n stages=[\"Production\"]\n )\n \n if current_prod:\n # 현재 Production을 Archived로 이동\n client.transition_model_version_stage(\n name=model_name,\n version=current_prod[0].version,\n stage=\"Archived\"\n )\n \n # 타겟 버전을 Production으로 승격\n client.transition_model_version_stage(\n name=model_name,\n version=target_version,\n stage=\"Production\"\n )\n \n print(f\"Rolled back {model_name} to version {target_version}\")MLOps는 ML 모델을 프로덕션 환경에서 안정적으로 운영하기 위한 필수 관행이다. 이 글에서 다룬 핵심 내용을 요약하면:
\nMLOps를 제대로 구축하면 수백 개의 모델을 동시에 운영하면서도 안정성과 성능을 보장할 수 있다. 시작은 작게, 하지만 확장 가능한 구조로 설계하는 것이 중요하다.
\nMLOps 실전 가이드 - AI 모델 서빙과 운영 자동화 완벽 정리
\n