Integration

python-Integration CI/CD Pipeline Integration

1. Jenkins Pipeline

groovy

pipeline {
    agent any
    environment {
        PYTHON_VERSION = "3.11.11"
        VIRTUAL_ENV = "venv"
    }
    stages {
        stage('Setup Python') {
            steps {
                sh '''
                    python3.11 -m venv ${VIRTUAL_ENV}
                    source ${VIRTUAL_ENV}/bin/activate
                    pip install --upgrade pip setuptools wheel
                    pip install -r requirements.txt
                '''
            }
        }
        stage('Test') {
            steps {
                sh '''
                    source ${VIRTUAL_ENV}/bin/activate
                    python -m pytest tests/ --junitxml=test-results.xml
                    python -m coverage run -m pytest
                    python -m coverage xml
                '''
            }
        }
        stage('Build') {
            steps {
                sh '''
                    source ${VIRTUAL_ENV}/bin/activate
                    python setup.py sdist bdist_wheel
                '''
            }
        }
    }
    post {
        always {
            junit 'test-results.xml'
            publishCoverage adapters: [coberturaAdapter('coverage.xml')]
        }
    }
}

2. GitLab CI/CD

yaml

# .gitlab-ci.yml
image: python:3.11.11-slim
variables:
  PIP_CACHE_DIR: "$CI_PROJECT_DIR/.cache/pip"
cache:
  paths:
    - .cache/pip/
    - venv/
before_script:
  - python -m venv venv
  - source venv/bin/activate
  - pip install --upgrade pip
  - pip install -r requirements.txt
stages:
  - test
  - build
  - deploy
test:
  stage: test
  script:
    - source venv/bin/activate
    - python -m pytest --cov=src tests/
    - python -m flake8 src/
    - python -m mypy src/
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage.xml
build:
  stage: build
  script:
    - source venv/bin/activate
    - python setup.py sdist bdist_wheel
  artifacts:
    paths:
      - dist/

3. GitHub Actions

yaml

# .github/workflows/python-app.yml
name: Python Application
on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]
jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        python-version: [3.11.11]
    steps:
    - uses: actions/checkout@v3
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v4
      with:
        python-version: ${{ matrix.python-version }}
    - name: Cache pip packages
      uses: actions/cache@v3
      with:
        path: ~/.cache/pip
        key: ${{ runner.os }}-pip-${{ hashFiles('**/requirements.txt') }}
    - name: Install dependencies
      run: |
        python -m pip install --upgrade pip
        pip install -r requirements.txt
        pip install pytest pytest-cov flake8
    - name: Lint with flake8
      run: |
        flake8 . --count --select=E9,F63,F7,F82 --show-source --statistics
    - name: Test with pytest
      run: |
        pytest --cov=./ --cov-report=xml
    - name: Upload coverage to Codecov
      uses: codecov/codecov-action@v3

Database Integration

1. PostgreSQL Integration

python

# Database configuration
import psycopg2
from sqlalchemy import create_engine
# Connection string
DATABASE_URL = "postgresql://user:password@localhost:5432/dbname"
# SQLAlchemy engine
engine = create_engine(
    DATABASE_URL,
    pool_size=20,
    max_overflow=0,
    pool_pre_ping=True,
    pool_recycle=300
)

2. Redis Integration

python

import redis
from redis.connection import ConnectionPool
# Redis connection pool
redis_pool = ConnectionPool(
    host='localhost',
    port=6379,
    db=0,
    max_connections=20,
    socket_connect_timeout=5,
    socket_timeout=5
)
redis_client = redis.Redis(connection_pool=redis_pool)

Message Queue Integration

1. Celery Configuration

python

# celery_config.py
from celery import Celery
app = Celery('myapp')
app.config_from_object({
    'broker_url': 'redis://localhost:6379/0',
    'result_backend': 'redis://localhost:6379/0',
    'task_serializer': 'json',
    'accept_content': ['json'],
    'result_serializer': 'json',
    'timezone': 'UTC',
    'enable_utc': True,
    'worker_prefetch_multiplier': 1,
    'task_acks_late': True,
})