Deployment Guide

Overview

The LinkedIn Interactions Pipeline is deployed as a Google Cloud Run service with automatic deployments via GitHub Actions.

Repository: credinvest/social_media_scrapers

Deployment Workflow: .github/workflows/deploy-brand-interactions.yaml

Cloud Run Services

The pipeline consists of the following Cloud Run services:

API Services (HTTP)

• Dev: brand-interactions-pipeline-dev
• Staging: brand-interactions-pipeline-staging
• Production: brand-interactions-pipeline

Polling Jobs (Scheduled)

• Dev: brand-interactions-polling-job-dev
• Staging: brand-interactions-polling-job-staging
• Production: brand-interactions-polling-job

Deployment Architecture

Environments

Environment	API Service	Polling Job	Tracker Table	Branch	Auto-Deploy
Dev	brand-interactions-pipeline-dev	brand-interactions-polling-job-dev	ApifyInteractionsTrackerTableDev	develop	Yes
Staging	brand-interactions-pipeline-staging	brand-interactions-polling-job-staging	ApifyInteractionsTrackerTableStaging	staging	Yes
Production	brand-interactions-pipeline	brand-interactions-polling-job	ApifyInteractionsTrackerTable	main	Yes

Components per Environment

Each environment deploys two services:

1. API Service (Cloud Run - HTTP)
2. Handles API requests
3. Processes webhooks from Apify
4. Writes data to BigQuery

5. Publicly accessible endpoint

6. Polling Job (Cloud Run Jobs - Scheduled)

7. Runs hourly (0 * * * *)
8. Catches missed webhook events
9. Queries Apify and processes completed runs

Deployment Process

Automatic Deployment

Deployments happen automatically when code is pushed to the respective branches via GitHub Actions:

• Push to develop → Deploys to Dev environment
• Push to staging → Deploys to Staging environment
• Push to main → Deploys to Production environment

Environment Variables

Required for API Service

Variable	Description	Example
APIFY_TOKEN	Apify API authentication token	Secret (from Secret Manager)
GCP_PROJECT_ID	Google Cloud project ID	cred-1556636033881
ENVIRONMENT	Environment name	dev, staging, production
BRAND_INTERACTIONS_WEBHOOK_URL	Webhook URL for Apify callbacks	Auto-generated based on service URL

Required for Polling Job

Variable	Description	Example
APIFY_TOKEN	Apify API authentication token	Secret (from Secret Manager)
GCP_PROJECT_ID	Google Cloud project ID	cred-1556636033881
ENVIRONMENT	Environment name	dev, staging, production

Post-Deployment Verification

Health Check

# Dev
curl https://brand-interactions-pipeline-dev-691864719497.us-central1.run.app/health

# Staging
curl https://brand-interactions-pipeline-staging-691864719497.us-central1.run.app/health

# Production
curl https://brand-interactions-pipeline-691864719497.us-central1.run.app/health

Expected Response:

{
  "status": "healthy",
  "service": "brand-interactions-pipeline",
  "environment": "dev"
}

Test Endpoint

curl -X POST https://brand-interactions-pipeline-dev-691864719497.us-central1.run.app/apify/linkedin/interactions \
  -H "Content-Type: application/json" \
  -d '{
    "user_company_id": "test123",
    "targets": [{"linkedin_handle": "microsoft", "entity": "COMPANY"}],
    "max_posts": 2
  }'

Feature Development Workflow

Recommended workflow:

1. Create feature branch from develop
2. Merge to develop and test in Dev environment
3. Merge develop to staging and test in Staging environment
4. Open PR from staging to main for production deployment